Interactive method and system for model monitoring, communication device

Abstract

The application provides a model monitoring interaction method and system and a communication device. In the application, through negotiation interaction between a model management entity and a model monitoring collection entity or decision of the model management entity, a monitoring scheme of an AI / ML model can be flexibly determined, and the network side can comprehensively master the monitoring conditions of the AI / ML model and the allocable resources for monitoring the AI / ML model, which is beneficial to improving the monitoring effect of the AI / ML model.

Description

Technical Field

[0001] This application relates to the field of communication technology, and in particular to an interactive method and system for model monitoring, and a communication device. Background Technology

[0002] In communication networks, communication equipment, such as base stations, terminal equipment, or network-side equipment, can deploy and run artificial intelligence (AI) / machine learning (ML) models to support services such as positioning, beam management, and channel state information (CSI) feedback.

[0003] During AI / ML model runtime, it's necessary to monitor the model to ensure it's functioning correctly. If the AI / ML model's output error is within a preset range, it's considered running normally. Otherwise, it's considered malfunctioning. When an AI / ML model malfunctions, the communication device can be promptly notified to make monitoring decisions such as shutting down, switching, upgrading, or rolling back the model.

[0004] Therefore, how to monitor AI / ML models is a key area that urgently needs research. Summary of the Invention

[0005] This application provides an interactive method and system for model monitoring, as well as a communication device. By leveraging the negotiation and interaction between the model management entity and the model monitoring and data collection entity, or the decision-making of the model management entity, the monitoring scheme for AI / ML models can be flexibly determined. It also enables the network side to fully grasp the monitoring conditions of AI / ML models and the available resources for monitoring AI / ML models, which is conducive to improving the monitoring effect of AI / ML models and ensuring that corresponding services can be provided using AI / ML models.

[0006] Firstly, this application provides an interactive method for model monitoring, the method comprising:

[0007] The model management entity obtains the first information, which is used to indicate the monitoring conditions of the first model. The first model is deployed in the first model running entity and is used to provide the first service.

[0008] The model management entity sends a second message to the model monitoring and acquisition entity. The second message is used to instruct one or more monitoring schemes. The monitoring schemes are used to obtain the running status of the first model. The second message is related to the first message.

[0009] The model management entity receives third information sent by the model monitoring and acquisition entity. The third information is used to indicate a unique monitoring scheme among one or more monitoring schemes. The third information is related to the second information.

[0010] The model management entity sends a fourth message to the model monitoring decision entity. The fourth message is used to instruct a set of monitoring resources for the first model. The fourth message is related to the first message.

[0011] Using the method provided in the first aspect, the model management entity obtains first information, enabling it to instruct the monitoring conditions of the first model deployed on the model running entity. Based on the first information, the model management entity sends second information to the model monitoring and acquisition entity. Based on the second information, the model monitoring and acquisition entity can determine a unique monitoring scheme from one or more monitoring schemes. The model monitoring and acquisition entity sends third information to the model management entity, enabling it to instruct the unique monitoring scheme. Based on the first information, the model management entity sends fourth information to the model monitoring decision entity, enabling it to instruct a set of monitoring resources used by the first model, allowing the unique monitoring scheme to use this set of resources to monitor the operation of the first model. Thus, through the negotiation and interaction between the model management entity and the model monitoring and acquisition entity, the monitoring scheme for the AI / ML model can be flexibly determined. Furthermore, the network side can comprehensively grasp the monitoring conditions of the AI / ML model and the available resources for monitoring it, which is beneficial for monitoring the operation of the AI / ML model and achieving better monitoring results.

[0012] In one possible design, the model management entity receives initial information, including:

[0013] The model management entity sends a first request to one or more model running entities. The first request is used to request information corresponding to the monitoring conditions of the models deployed in the model running entities. The models deployed in the model running entities are used to provide a first service. The one or more model running entities include at least a first model running entity.

[0014] The model management entity receives the first information sent by the first model running entity.

[0015] In one possible design, the model management entity receives initial information, including:

[0016] The model management entity obtains the first information from a pre-stored second information set. The second information set includes information corresponding to the monitoring conditions of models deployed in one or more model running entities. The models deployed in the model running entities are used to provide the first service. The one or more model running entities include at least the first model running entity.

[0017] In one possible design, the model management entity receives initial information, including:

[0018] The model management entity sends a second request to other network-side entities or third-party network devices. The second request is used to request the first information, which is stored in the other network-side entities or third-party network devices.

[0019] The model management entity receives the first information sent by other network-side entities or third-party network devices.

[0020] In one possible design, the method also includes:

[0021] The model management entity determines the fourth information based on the first and second information;

[0022] Alternatively, the model management entity determines the fourth information based on the first and third information.

[0023] Secondly, this application provides an interactive method for model monitoring, the method comprising:

[0024] The model management entity receives a first request sent by the first model running entity. The first request is used to request monitoring of the first model. The first request carries first information, which is used to indicate the monitoring conditions of the first model. The first model is deployed in the first model running entity and is used to provide the first service.

[0025] The model management entity sends a second message to the model monitoring and acquisition entity. The second message is used to indicate a unique monitoring scheme, which is used to obtain the running status of the first model. The second message is related to the first message.

[0026] The model management entity sends a first response to the first model running entity, and the first response is used to respond to the first request;

[0027] The model management entity sends a fourth message to the model monitoring decision entity. The fourth message is used to instruct a set of monitoring resources used by the first model. The fourth message is related to the first and second messages.

[0028] Using the method provided in the second aspect, the model running entity sends a first request to the model management entity. This first request carries first information, which indicates the monitoring conditions of the first model and the need for monitoring. The model management entity then sends second information to the model monitoring and acquisition entity based on the first information. The model monitoring and acquisition entity determines a unique monitoring scheme based on the second information. Finally, the model management entity sends fourth information to the model monitoring decision entity based on the first and second information. This fourth information indicates a set of monitoring resources for the first model, enabling the unique monitoring scheme to utilize these resources to monitor the model's operation. Therefore, through the decision-making of the model management entity, the monitoring scheme for the AI / ML model can be flexibly determined. Furthermore, the network side can comprehensively grasp the monitoring conditions and available resources for monitoring the AI / ML model, facilitating better monitoring of its operation and achieving superior monitoring results.

[0029] In one possible design, the method also includes:

[0030] The model management entity sends the second message to the first model running entity.

[0031] In any of the first to second aspects described above, and in any possible design of that aspect, the method further includes:

[0032] The model management entity determines the second information based on the available resources of the model management entity and the first information.

[0033] In any of the first to second aspects described above, and in any possible design of that aspect, the method further includes:

[0034] The model management entity determines the second information based on the available resources of the model management entity, the available resources of the model monitoring decision entity, and the first information. The second information is also used to indicate a set of monitoring resources used by the first model.

[0035] In any of the first to second aspects described above, and in any possible design of that aspect, the method further includes:

[0036] The model management entity sends a third request to the model monitoring decision entity, which requests the model monitoring decision entity to configure a set of monitoring resources for the first model.

[0037] The model management entity receives a third response from the model monitoring decision entity, which is used to instruct the model monitoring decision entity to monitor one or more sets of resources used by the first model.

[0038] The model management entity determines a set of monitored resources for the first model based on the third response.

[0039] Thirdly, this application provides an interactive method for model monitoring, the method comprising:

[0040] The model monitoring and acquisition entity receives second information sent by the model management entity. The second information is used to indicate one or more monitoring schemes. The monitoring schemes are used to monitor the operation of the first model. The second information is related to the first information. The first information is obtained by the model management entity. The first information is used to indicate the monitoring conditions of the first model. The first model is deployed in the first model running entity. The first model is used to provide the first service.

[0041] The model monitoring and acquisition entity sends a third piece of information to the model management entity. This third piece of information is used to indicate a unique monitoring scheme among one or more monitoring schemes, and it is related to the second piece of information.

[0042] The beneficial effects of the methods provided in the third aspect and the various possible designs of the third aspect can be found in the beneficial effects of the first aspect and the various possible implementations of the first aspect, and will not be repeated here.

[0043] In one possible design, the method also includes:

[0044] After the unique monitoring scheme is started, the model monitoring and acquisition entity monitors the first model based on the third information and obtains the monitoring measurement data of the first model. The monitoring measurement data is used to represent the operating status of the first model.

[0045] The model monitoring and acquisition entity sends monitoring and measurement data to the model monitoring and decision-making entity. The monitoring and measurement data is used by the model monitoring and decision-making entity to generate the monitoring decision for the first model.

[0046] Fourthly, this application provides an interactive method for model monitoring, the method comprising:

[0047] The first model running entity sends a first request to the model management entity. The first request is used to request monitoring of the first model. The first request carries first information, which is used to indicate the monitoring conditions of the first model. The first model is deployed in the first model running entity and is used to provide the first service.

[0048] The first model running entity receives the first response sent by the model management entity, and the first response is used to respond to the first request.

[0049] In one possible design, the method also includes:

[0050] The first model running entity receives the second information sent by the model management entity. The second information is used to indicate a unique monitoring scheme, which is used to obtain the running status of the first model. The second information is related to the first information.

[0051] The beneficial effects of the methods provided in the fourth aspect and the various possible designs of the fourth aspect can be found in the beneficial effects of the second aspect and the various possible implementations of the second aspect, and will not be repeated here.

[0052] Fifthly, this application provides an interactive method for model monitoring, the method comprising:

[0053] The model monitoring and acquisition entity receives the second information sent by the model management entity. The second information is used to indicate a unique monitoring scheme. The monitoring scheme is used to monitor the operation of the first model. The second information is related to the first information. The first information is carried in the first request. The first request is sent by the first model running entity to the model management entity. The first request is used to request monitoring of the first model. The first information is used to indicate the monitoring conditions of the first model. The first model is deployed in the first model running entity. The first model is used to provide the first service.

[0054] The beneficial effects of the methods provided in the fifth aspect and the various possible designs of the fifth aspect can be found in the beneficial effects of the second aspect and the various possible implementations of the second aspect, and will not be repeated here.

[0055] In one possible design, the method also includes:

[0056] After the unique monitoring scheme is started, the model monitoring and acquisition entity monitors the first model according to the second information and obtains the monitoring measurement data of the first model. The monitoring measurement data is used to represent the operating status of the first model.

[0057] The model monitoring and acquisition entity sends monitoring and measurement data to the model monitoring and decision-making entity. The monitoring and measurement data is used by the model monitoring and decision-making entity to generate the monitoring decision for the first model.

[0058] Sixthly, this application provides an interactive method for model monitoring, the method comprising:

[0059] The model monitoring decision entity receives a fourth message sent by the model management entity. The fourth message is used to indicate a set of monitoring resources used by the first model. The fourth message is related to the first message, which is used to indicate the monitoring conditions of the first model. The first model is deployed in the first model running entity and is used to provide the first service.

[0060] The beneficial effects of the methods provided in the sixth aspect and its various possible designs can be found in the first aspect and its various possible implementations, and will not be repeated here.

[0061] In one possible design, the fourth information is determined by the model management entity based on the first and second information; or, the fourth information is determined by the model management entity based on the first and third information.

[0062] The second information is related to the first information. The second information is used to indicate one or more monitoring schemes. The monitoring schemes are used to obtain the operating status of the first model. The third information is related to the second information. The third information is used to indicate the unique monitoring scheme among one or more monitoring schemes.

[0063] In one possible design, the method also includes:

[0064] The model monitoring decision entity receives the monitoring measurement data sent by the model monitoring acquisition entity. The monitoring measurement data is obtained by the model monitoring acquisition entity after the unique monitoring scheme is started, based on the third information, to monitor the first model. The monitoring measurement data is used to represent the operating status of the first model.

[0065] The model monitoring decision entity generates monitoring decisions for the first model based on the monitoring measurement data.

[0066] Seventhly, this application provides an interactive method for model monitoring, the method comprising:

[0067] The model monitoring decision entity receives a fourth message from the model management entity. This fourth message indicates a set of monitoring resources used by the first model. The fourth message is related to the first and second messages. The first message indicates the monitoring conditions of the first model. The first model is deployed in the first model running entity and provides a first service. The second message is related to the first message and indicates a unique monitoring scheme. The monitoring scheme is used to obtain the operating status of the first model. The first message is carried in a first request. The first request is sent by the first model running entity to the model management entity and is used to request monitoring of the first model.

[0068] The beneficial effects of the methods provided in the seventh aspect and the various possible designs of the seventh aspect can be found in the beneficial effects of the second aspect and the various possible implementations of the second aspect, and will not be repeated here.

[0069] In one possible design, the method also includes:

[0070] The model monitoring decision entity receives the monitoring measurement data sent by the model monitoring acquisition entity. The monitoring measurement data is obtained by the model monitoring acquisition entity after the unique monitoring scheme is started, based on the second information, to monitor the first model. The monitoring measurement data is used to represent the operating status of the first model.

[0071] The model monitoring decision entity generates monitoring decisions for the first model based on the monitoring measurement data.

[0072] In any of the sixth to seventh aspects mentioned above, and in any possible design of that aspect, the method further includes:

[0073] The model monitoring decision entity receives a third request from the model management entity. The third request is used to request the model monitoring decision entity to configure a set of monitoring resources for the first model.

[0074] The model monitoring decision entity sends a third response to the model management entity. This third response indicates one or more sets of monitoring resources used by the first model configured by the model monitoring decision entity. The third response is used by the model management entity to determine a set of monitoring resources used by the first model. In any of the first to seventh aspects and any possible design of that aspect, the model operation entity or model monitoring and acquisition entity is a terminal device or a network device; the model management entity or model monitoring decision entity is a network-side entity within the core network equipment or a third-party network device.

[0075] In any of the first to seventh aspects described above, and in any possible design of that aspect, the first information includes at least one of the following:

[0076] The monitoring types supported by the first model include at least one of the following: the data format of the input data of the first model, the data format of the output data of the first model, or the parameters of the external environment of the first model.

[0077] Alternatively, monitor the type of decision entity in the model of the first model;

[0078] Alternatively, the performance metrics of the monitoring scheme supported by the first model.

[0079] In any of the first to seventh aspects mentioned above, and in any possible design of that aspect, the second information includes at least one of the following:

[0080] First information;

[0081] Alternatively, the identifier of the monitoring scheme supported by the first model;

[0082] Alternatively, monitor the address of the decision entity in the first model;

[0083] Alternatively, it could be auxiliary information for monitoring the decision-making entity of the first model; or, it could be the validity period information of the monitoring scheme supported by the first model.

[0084] Alternatively, the effective response time information for the second message;

[0085] Alternatively, the priority or recommendation level of the monitoring scheme supported by the first model.

[0086] In any of the first, third, and sixth aspects described above, and in any possible design of that aspect, the third information includes at least one of the following:

[0087] Second information;

[0088] Or, a unique identifier for the monitoring solution;

[0089] Alternatively, the actual response time information of the third party;

[0090] Alternatively, it can be used to monitor the startup time of the first model or to trigger the startup event of the first model.

[0091] Eighthly, this application provides an interactive device for model monitoring, comprising: a module for executing an interactive method for model monitoring in any of the above aspects and any possible design of such aspect.

[0092] Ninthly, this application provides an interactive system for model monitoring, comprising: one or more model running entities, a model monitoring acquisition entity for executing the interactive method for model monitoring in any possible design of the third aspect and the above, a model management entity for executing the interactive method for model monitoring in any possible design of the first aspect and the above, and a model monitoring decision entity for executing the interactive method for model monitoring in any possible design of the sixth aspect and the above.

[0093] And / or, a first model running entity for performing the interactive method of model monitoring in the fourth aspect and any possible design of that aspect, a model monitoring acquisition entity for performing the interactive method of model monitoring in the fifth aspect and any possible design of that aspect, a model management entity for performing the interactive method of model monitoring in the second aspect and any possible design of that aspect, and a model monitoring decision entity for performing the interactive method of model monitoring in the seventh aspect and any possible design of that aspect.

[0094] One or more model running entities include at least the first model running entity.

[0095] In a tenth aspect, this application provides a communication device including a processor. The processor is configured to invoke a stored computer program or computer instructions, causing the processor to implement any possible design method from any of the first to seventh aspects.

[0096] Optionally, the communication device may also include a transceiver, and the processor is used to control the transceiver to send and receive signals.

[0097] In one aspect, this application provides a communication device, including a processor for calling a computer program or instructions in the memory to cause the processor to execute any possible design method in any one of the first to seventh aspects.

[0098] Alternatively, the processor may be coupled to the memory via an interface.

[0099] In a twelfth aspect, this application provides a computer-readable storage medium having a computer-executable program or instructions stored thereon, wherein the computer-executable program or instructions, when executed by a processor, cause a communication device to implement any possible design method of any one of the first to seventh aspects.

[0100] In a thirteenth aspect, this application provides a chip, comprising: an interface circuit and a logic circuit, wherein the interface circuit is used to receive signals from other chips outside the chip and transmit them to the logic circuit, or to send signals from the logic circuit to other chips outside the chip, and the logic circuit is used to implement any possible design method in any of the first to seventh aspects.

[0101] In a fourteenth aspect, this application provides a computer program product comprising: execution instructions stored in a readable storage medium, at least one processor of a communication device being able to read the execution instructions from the readable storage medium, and the at least one processor executing the execution instructions causing the communication device to implement any possible design method of any one of the first to seventh aspects. Attached Figure Description

[0102] Figure 1 This is a schematic diagram of the structure of an interactive system for model monitoring provided in one embodiment of this application;

[0103] Figure 2 Signaling interaction diagram of an interactive method for model monitoring provided in an embodiment of this application;

[0104] Figure 3 Signaling interaction diagram of an interactive method for model monitoring provided in an embodiment of this application;

[0105] Figure 4A flowchart illustrating an interactive method for model monitoring provided in an embodiment of this application;

[0106] Figure 5 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application;

[0107] Figure 6 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application;

[0108] Figure 7 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application;

[0109] Figure 8 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application;

[0110] Figure 9 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application;

[0111] Figure 10 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application;

[0112] Figure 11 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application;

[0113] Figure 12 This is a schematic diagram of the hardware structure of a communication device provided in an embodiment of this application. Detailed Implementation

[0114] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c alone can mean: a alone, b alone, c alone, a combination of a and b, a combination of a and c, a combination of b and c, or a, b, and c, where a, b, and c can be single or multiple. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0115] Please see Figure 1 , Figure 1 A schematic diagram of the structure of an interactive system for model monitoring according to an embodiment of this application is shown. Figure 1As shown, the interactive system for model monitoring in this application may include: one or more model running entities 11, model monitoring and acquisition entities 12, model management entities 13, and model monitoring decision entities 14.

[0116] The model running entity 11 may include at least one first model running entity. The model monitoring decision entity 14 may include one or more. This application does not limit the number of model running entities 11 and model monitoring decision entities 14 included in the interactive system for model monitoring.

[0117] The model running entity 11 and the model monitoring and acquisition entity 12 can be the same entity. Alternatively, the model running entity 11 and the model monitoring and acquisition entity 12 can be different entities.

[0118] The model management entity 13 and the model monitoring and decision-making entity 14 can be the same entity. Alternatively, the model management entity 13 and the model monitoring and decision-making entity 14 can be different entities.

[0119] Among them, the model operation entity 11 or the model monitoring and data collection entity 12 can be a terminal device or a network device. Correspondingly, the model management entity 13 or the model monitoring and decision-making entity 14 can be a network-side entity in the core network device or a third-party network device.

[0120] Additionally, the model management entity 13 or the model monitoring and decision-making entity 14 can be a network-side entity within the core network equipment or a third-party network device. Correspondingly, the model management entity 13 or the model monitoring and decision-making entity 14 can be a terminal device or a network device.

[0121] When the structure of the interactive system for model monitoring is relatively simple, the model management entity 13 and the model monitoring decision entity 14 can be the same entity. For example, the model management entity 13 and the model monitoring decision entity 14 can be deployed in the same entity.

[0122] When the interactive system for model monitoring has a large and complex structure, such as when monitoring a massive number of AI / ML models, the model management entity 13 and the model monitoring decision entity 14 can be different entities. The model monitoring decision entity 14 can adopt a distributed design, with multiple entities configured. For example, multiple model monitoring decision entities 14 can be deployed on network-side entities and cloud servers. Alternatively, multiple model monitoring decision entities 14 can be deployed on terminal devices and network devices.

[0123] Terminal equipment can refer to user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, wireless communication equipment, user agent, or user device. Terminal devices can also be satellite phones, cellular phones, smartphones, wireless data cards, wireless modems, machine-type communication devices, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices or wearable devices, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical care, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, terminal devices in 5G networks, terminal devices in 6G networks, or terminal devices in future communication networks, etc. In addition, terminal device 13 can also be a terminal device in an Internet of Things (IoT) system.

[0124] Network equipment: This can be a base station, access point, or access network device, or it can refer to a device in an access network that communicates with a wireless terminal via one or more sectors on an air interface. Network equipment can be used to convert received air frames to and from IP packets, and acts as a router between the wireless terminal and the rest of the access network, which may include an Internet Protocol (IP) network. Network equipment can also coordinate the attribute management of the air interface. For example, network equipment can be a satellite, a drone, a base transceiver station (BTS) in Global System for Mobile Communication (GSM) or Code Division Multiple Access (CDMA), a base station (NodeB, NB) in Wideband Code Division Multiple Access (WCDMA), an evolved Node B (eNB or eNodeB) in Long Term Evolution (LTE), or a terminal, relay station, or access point that performs base station functions in V2X (vehicular to everything), Device-to-Device (D2D), and Machine-to-Machine (M2M) communications, or a base station in a 5G network, such as a gNB, or a base station in a future 6G network; there are no limitations on this.

[0125] In a network architecture, network devices may include centralized unit (CU) nodes, distributed unit (DU) nodes, RAN devices including CU nodes and DU nodes, or RAN devices including control plane CU nodes (CU-CP nodes), user plane CU nodes (CU-UP nodes), and DU nodes.

[0126] Core network equipment: This can be operator core network equipment or a functional entity within the core network used to provide the first service. The first service includes, but is not limited to, any one of the following: positioning, beam management, or channel state information feedback. Furthermore, in 6G mobile communication systems or future mobile communication systems, the functional entity can still be a network element, or have other names, such as network entity or entity, etc., and this application does not impose any limitations.

[0127] Core network equipment: including but not limited to operator core network equipment, which can detect the communication services subscribed by the calling and called parties. This application can use numbers, codes, or identifiers to distinguish different communication services, facilitating accurate detection of communication services by the core network equipment. Typically, the core network equipment establishes communication connections with both the first and second equipment.

[0128] Third-party network equipment: This can be servers, such as those from chip, mobile phone, and base station manufacturers. Third-party network equipment can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0129] Any two entities among the model operation entity 11, model monitoring and acquisition entity 12, model management entity 13, and model monitoring and decision-making entity 14 can be directly or indirectly connected via wired or wireless communication methods. Wired communication methods can include coaxial cable, fiber optic cable, digital subscriber line (DSL), etc. Wireless communication methods can include Bluetooth, infrared, Wi-Fi, microwave, etc.

[0130] Model running entity 11 is used to deploy and run AI / ML models.

[0131] The model monitoring and acquisition entity 12 is used to obtain monitoring and measurement data of the AI / ML model from the model execution entity 11 during the monitoring process after the monitoring scheme is started. For example, the model execution entity 11 can send the initial measurement data of the AI / ML model to the model monitoring and acquisition entity 12. The model monitoring and acquisition entity 12 processes the initial measurement data of the AI / ML model to obtain the monitoring and measurement data of the AI / ML model, which is used to indicate the running status of the AI / ML model. The monitoring and measurement data can also be called monitoring and measurement parameters, and the initial measurement data can also be called initial measurement parameters. Alternatively, the monitoring and measurement data can also be the initial measurement data of the AI / ML model.

[0132] Model management entity 13 is used to coordinate all stages of the AI / ML model lifecycle, such as data acquisition, model training, model inference, and model monitoring.

[0133] The model monitoring and decision-making entity 14 is used, after the monitoring scheme is initiated, to generate monitoring decisions for the AI / ML model based on the monitoring and measurement data of the AI / ML model, using algorithms to produce indicators that reflect the operational effect, performance, or metrics of the AI / ML model. These monitoring decisions are then used to output the operational status of the AI / ML model in ways such as text, voice, or images.

[0134] When the AI / ML model's operation status indicates an abnormal operation, the monitoring decisions for the AI / ML model may include, but are not limited to: model shutdown, model switching, model activation, and model rollback.

[0135] Taking the use of AI / ML models to provide location services as an example, assuming that the location accuracy degrades in the location scenario, there are multiple ways to implement the monitoring and decision-making of AI / ML models when they malfunction.

[0136] For example, monitoring decisions for AI / ML models can include model switching, model upgrading, and model rollback. Model rollback here refers to ceasing to use the AI / ML model for positioning and instead employing traditional methods such as GPS, WLAN, or Bluetooth for positioning.

[0137] For example, monitoring decisions for AI / ML models can also include model activation and model shutdown, that is, first shutting down the AI / ML model and then activating other AI / ML models.

[0138] It should be noted that the interactive system for model monitoring described above may include, but is not limited to, the structure described above. Furthermore, the monitoring scheme may also be referred to as a monitoring method, monitoring approach, etc., and this application does not limit the terminology used.

[0139] Based on the foregoing description, the following embodiments of this application will be based on having Figure 1 Taking the interactive system for model monitoring with the structure shown as an example, and combining scenario one and scenario two, the interactive method for model monitoring provided in this application is described in detail.

[0140] The technical solution described in Scenario 1 involves determining the monitoring scheme for the AI / ML model through network-side negotiation. The technical solution described in Scenario 2 involves determining the monitoring scheme for the AI / ML model through network-side decision-making.

[0141] Scene 1

[0142] Please see Figure 2 , Figure 2 The diagram illustrates the signaling interaction of an interactive method for model monitoring according to an embodiment of this application. Figure 2 As shown, the interactive method for model monitoring in this application may include: S101-S107.

[0143] S101-S104 are executed before the monitoring plan is started, and S105-S107 are executed after the monitoring plan is started. S101-S104 are used to determine the monitoring plan, and S105-S107 are used to determine the monitoring decision. S105-S107 are optional steps.

[0144] S101, The model management entity obtains the first information, which is used to indicate the monitoring conditions of the first model. The first model is deployed in the first model running entity and is used to provide the first service.

[0145] Different AI / ML models may provide different services, or they may provide the same services. At least one of the input data, output data, or environmental data may differ between different AI / ML models. Therefore, the parameters monitored may differ between different AI / ML models. Furthermore, the entities monitored may also differ between different AI / ML models.

[0146] The first model, which is an AI / ML model, is deployed within the first model running entity and can provide a first service. This first service may include features such as localization, beam management, or CSI feedback.

[0147] Based on this, the first information can be used to indicate the monitoring conditions of the first model. The monitoring conditions of the first model can indicate which parameter(s) of the first model can be monitored. Additionally, the monitoring conditions of the first model can also indicate which entity(s) can monitor the first model.

[0148] In summary, the model management entity can obtain the first information before the first model is needed to provide the first service.

[0149] S102, The model management entity sends the second information to the model monitoring and acquisition entity.

[0150] Correspondingly, the model monitoring and acquisition entity receives the second information sent by the model management entity.

[0151] The second information is used to indicate one or more monitoring schemes, which are used to obtain the operating status of the first model. The second information is related to the first information.

[0152] The model management entity can obtain the monitoring conditions of the first model based on the first information. The model management entity can then select one or more monitoring schemes based on these conditions, ensuring that each monitoring scheme matches the monitoring conditions of the first model and that the available resources of the model management entity can support each monitoring scheme, thereby achieving monitoring of the first model's operational status.

[0153] Each of the one or more monitoring schemes is a feasible monitoring scheme. The available resources of the model management entity may include at least one of the following: idle central processing unit (CPU) capacity, idle graphics processing unit (GPU) capacity, or idle memory capacity.

[0154] Therefore, the model management entity can determine the second information based on one or more monitoring schemes and send the second information to the model monitoring and acquisition entity. It can use the second information to query the model monitoring and acquisition entity which monitoring scheme is monitoring the operation of the first model.

[0155] The operational status of the first model can include: its operational state, such as whether it is running normally, abnormally, or some of its sub-functions are running normally; which sub-functions are running normally or abnormally. Additionally, the operational status of the first model can also include its performance parameters, such as its runtime and resource utilization.

[0156] S103, The model monitoring and acquisition entity sends third information to the model management entity.

[0157] Correspondingly, the model management entity receives the third information sent by the model monitoring and data collection entity.

[0158] The third information is used to indicate the unique monitoring scheme among one or more monitoring schemes, and the third information is related to the second information.

[0159] Based on the second information, the model monitoring and acquisition entity can select a unique monitoring scheme from one or more monitoring schemes, ensuring that the available resources of the model monitoring and acquisition entity can support this unique monitoring scheme, thereby enabling the monitoring of the first model's operational status. In this way, the available resources of both the model monitoring and acquisition entity and the model management entity can support a unique monitoring scheme.

[0160] The available resources for the model monitoring and acquisition entity may include at least one of the following: idle CPU, idle GPU, or idle memory capacity.

[0161] Therefore, the model monitoring and acquisition entity can determine the third information based on the unique monitoring scheme and send the third information to the model management entity, so that the model management entity can know the unique monitoring scheme with the help of the third information.

[0162] S104. The model management entity sends the fourth message to the model monitoring and decision-making entity.

[0163] Correspondingly, the model monitoring and decision-making entity receives the fourth piece of information sent by the model management entity.

[0164] The fourth piece of information is used to indicate a set of monitoring resources used by the first model, and the fourth piece of information is related to the first piece of information.

[0165] In some examples, the model management entity can determine a set of monitoring resources for the first model based on first information, or first information and second information. This ensures that the set of monitoring resources for the first model can support each of one or more monitoring schemes in monitoring the first model, avoiding the problem of monitoring interruption or inability to monitor the first model due to insufficient resources. This facilitates resource configuration based on monitoring requirements.

[0166] In other examples, the model management entity can determine a set of monitoring resources for the first model based on the first and third information, so that the set of monitoring resources for the first model can support the use of a unique monitoring scheme to monitor the first model, avoiding the problem of monitoring interruption or inability to monitor the first model due to insufficient resources, and facilitating the configuration of resources based on resource conditions and monitoring requirements.

[0167] Among them, the set of monitoring resources used in the first model refers to the resources used to monitor the first model.

[0168] Therefore, the model management entity can determine the fourth information based on a set of monitored resources of the first model, and send the fourth information to the model monitoring decision entity, so that the model monitoring decision entity can know the resources used by the first model.

[0169] In summary, the model management entity and the model monitoring and acquisition entity can determine a unique monitoring scheme and a set of monitoring resources for the first model through network-side negotiation. Therefore, during the operation of the first model by the model running entity, the model monitoring and acquisition entity and the model monitoring decision-making entity can monitor the operation of the first model using the unique monitoring scheme based on the set of monitoring resources used by the first model.

[0170] S105. After the unique monitoring scheme is started, the model monitoring and acquisition entity monitors the first model according to the third information and obtains the monitoring measurement data of the first model. The monitoring measurement data is used to represent the operating status of the first model.

[0171] The unique monitoring scheme can be started before the first model runs, or when the first model runs, or when the first model malfunctions, or when other AI / ML models used to provide the first service malfunction; this application does not limit this. The unique monitoring scheme can be started based on startup time or startup event.

[0172] Once the sole monitoring scheme is activated, the model monitoring and acquisition entity can monitor the first model based on the third information and obtain the monitoring measurement data of the first model.

[0173] The monitoring and measurement data of the first model can be the initial measurement data of the first model or the data after processing of the initial measurement data of the first model. This application does not limit this.

[0174] S106. The model monitoring acquisition entity sends monitoring measurement data to the model monitoring decision entity.

[0175] Correspondingly, the model monitoring decision entity receives the monitoring and measurement data sent by the model monitoring acquisition entity.

[0176] S107. The model monitoring decision entity generates the monitoring decision of the first model based on the monitoring measurement data.

[0177] The model monitoring and acquisition entity can then send the monitoring and measurement data of the first model to the model monitoring and decision-making entity. Based on this data, the model monitoring and decision-making entity determines the operational status of the first model. Therefore, based on the monitoring and measurement data, the model monitoring and decision-making entity can generate monitoring decisions for the first model.

[0178] The specific implementation of the monitoring decision of the first model can be found in the description of the monitoring decision of the AI / ML model above, and will not be repeated here.

[0179] Taking model replacement as an example of monitoring decisions for the first model, the model monitoring decision entity can inform the model management entity that the first model needs to be replaced. Then, the model management entity, the model monitoring and data acquisition entity, and the model monitoring decision entity can continue to execute S101-S104 to determine the unique monitoring scheme and the corresponding set of monitoring resources for the replaced model. Then, the model monitoring and data acquisition entity and the model monitoring decision entity can continue to execute S105-S107 to determine the operational status of the replaced model and make monitoring decisions.

[0180] The model monitoring interaction method provided in this application obtains first information through a model management entity, which instructs the monitoring conditions of a first model deployed on a model running entity. The model management entity sends second information to a model monitoring and acquisition entity based on the first information. The model monitoring and acquisition entity determines a unique monitoring scheme from one or more monitoring schemes based on the second information. The model monitoring and acquisition entity sends third information to the model management entity, instructing the unique monitoring scheme. The model management entity sends fourth information to a model monitoring decision entity based on the first information, instructing a set of monitoring resources for the first model, enabling the unique monitoring scheme to use these resources to monitor the first model's operation. Therefore, through the negotiation and interaction between the model management entity and the model monitoring and acquisition entity, the monitoring scheme for the AI / ML model can be flexibly determined. This also allows the network side to fully grasp the monitoring conditions of the AI / ML model and the available resources for monitoring it, which is beneficial for monitoring the AI / ML model's operation and achieving better monitoring results.

[0181] Furthermore, after the sole monitoring scheme is activated, the model monitoring and acquisition entity and the model monitoring and decision-making entity begin monitoring the first model. The model monitoring and acquisition entity collects monitoring measurement data of the first model. The model monitoring and acquisition entity sends the monitoring measurement data of the first model to the model monitoring and decision-making entity. The model monitoring and decision-making entity generates a monitoring decision for the first model based on the monitoring measurement data. Thus, the monitoring process of the first model is realized.

[0182] Based on the description of the above embodiments, in S101, when the first service needs to be used, the model running entity or the model management entity needs to monitor the running status of the first model.

[0183] When the model running entity needs to monitor the running status of the first model, the model running entity can send the first information to the model management entity, so that the model management entity can obtain the first information.

[0184] When the model management entity needs to monitor the operation of the first model, it can obtain the first information in various ways.

[0185] As a feasible implementation, the model capable of providing the first service may be deployed in one model running entity or multiple model running entities. Therefore, the model management entity can send the first request to one or more model running entities.

[0186] The first request requests information regarding the monitoring conditions of the models deployed in each model running entity. Each model running entity deploys a model to provide the first service, and at least one of the one or more model running entities includes the first model running entity. Additionally, the first request indicates a current need to monitor the first model. The deployed model can be an AI / ML model.

[0187] Therefore, after receiving the first request, each model running entity can send information corresponding to the monitoring conditions of the models deployed in each model running entity to the model management entity.

[0188] In some examples, the model management entity can receive the first message sent by the first model running entity.

[0189] In other examples, the model management entity can receive information corresponding to the monitoring conditions of all models deployed in all model management entities. Therefore, the model management entity can select the information corresponding to the monitoring conditions of the model deployed in the first model management entity from the information corresponding to the monitoring conditions of all models deployed in all model management entities; this information is called the "first information."

[0190] In other examples, the model management entity can determine whether a model deployed in a model runtime entity can provide the first service after receiving information corresponding to the monitoring conditions of that model. Upon first determining that a model deployed in that model runtime entity can provide the first service, the model management entity can stop receiving information corresponding to the monitoring conditions of models deployed in other model runtime entities. Here, the model runtime entity is the first model runtime entity, the model deployed in that model runtime entity is the first model, and the information corresponding to the monitoring conditions of the model deployed in that model runtime entity is the first information.

[0191] As another feasible implementation, the model management entity can pre-store second information. This second information set includes information corresponding to the monitoring conditions of models deployed in one or more model running entities. Each model running entity deploys a model to provide the first service, and the one or more model running entities include at least one first model running entity. The deployed model can be an AI / ML model. Therefore, the model management entity can obtain the first information from the pre-stored second information set.

[0192] As another feasible implementation, other network-side entities or third-party network devices may pre-store information corresponding to the monitoring conditions of models deployed in one or more model running entities. The models deployed in each model running entity are used to provide the first service. The one or more model running entities include at least the first model running entity, and the other network-side entities or third-party network devices store the first information.

[0193] Other network-side entities can be entities in the core network equipment other than the model management entity, model monitoring and data collection entity, and model monitoring and decision-making entity, such as the model registration and management entity. Alternatively, other network-side entities can be third-party network devices. The deployed model can be an AI / ML model.

[0194] Therefore, the model management entity can send a second request to other network-side entities or third-party network devices. This second request is used to request the first information. Upon receiving the second request, the other network-side entities or third-party network devices can send the first information to the model management entity. Thus, the model management entity can obtain the first information.

[0195] In summary, the model-managed entity can obtain the first information.

[0196] Scene 2

[0197] Please see Figure 3 , Figure 3 The diagram illustrates the signaling interaction of an interactive method for model monitoring according to an embodiment of this application. Figure 3 As shown, the interactive method for model monitoring in this application may include: S201-S208.

[0198] Steps S201-S205 are executed before the monitoring plan is started, while steps S206-S208 are executed after the monitoring plan is started. S201-S205 are used to determine the monitoring plan, and S206-S208 are used to determine the monitoring decision. Step S204 is optional, and steps S205-S208 are also optional.

[0199] S201. The first model running entity sends a first request to the model management entity.

[0200] Correspondingly, the model management entity receives the first request sent by the first model running entity.

[0201] The first request is used to request monitoring of the first model. The first request carries first information, which is used to indicate the monitoring conditions of the first model. The first model is deployed in the first model running entity and is used to provide the first service.

[0202] The same model running entity may deploy one or more AI / ML models. Multiple AI / ML models may provide the same service or they may provide different services.

[0203] Based on this, the first model is deployed in the first model running entity. The first model is an AI / ML model, and it can provide a first service. The first service may include, for example, localization, beam management, or CSI feedback.

[0204] Before the first model is required to provide the first service, the first model running entity can send a first request to the model management entity, so that the model management entity knows that a monitoring scheme for the first model needs to be determined.

[0205] The specific implementation method of the monitoring conditions of the first model indicated by the first information can be found in [reference needed]. Figure 2 The description of S101 in the embodiment will not be repeated here.

[0206] It should be noted that the first request in Scenario 2 is different from the first request in Scenario 1. Furthermore, the first request may not include the first information. The first model running entity can send both the first request and the first information to the model management entity. There is no temporal order between sending the first request and the first information, and they can be sent simultaneously or sequentially.

[0207] S202, The model management entity sends a second message to the model monitoring and data collection entity.

[0208] Correspondingly, the model monitoring and acquisition entity receives the second information sent by the model management entity.

[0209] The second information is used to indicate a unique monitoring scheme, which is used to obtain the operating status of the first model. The second information is related to the first information.

[0210] The model management entity can obtain the monitoring conditions of the first model based on the first information. The model management entity can then determine a unique monitoring scheme based on these conditions, ensuring that the unique monitoring scheme matches the monitoring conditions of the first model and that the available resources of the model management entity can support the unique monitoring scheme, thereby achieving monitoring of the first model's operational status.

[0211] Therefore, the model management entity can determine the second information based on the unique monitoring scheme and send the second information to the model monitoring and acquisition entity. It can use the second information to inform the model monitoring and acquisition entity that the unique monitoring scheme is used to monitor the operation of the first model.

[0212] For details on the implementation of the first model, please refer to [link / reference needed]. Figure 2 The description of S101 in the embodiment will not be repeated here.

[0213] S203, The model management entity sends the first response to the first model running entity.

[0214] Correspondingly, the first model running entity receives the first response sent by the model management entity.

[0215] The first response is used to answer the first request.

[0216] Upon receiving the first request, the model management entity can send a first response to the first model running entity, thereby informing the first model running entity that the model management entity has received the first request.

[0217] It should be noted that there is no temporal order between S202 and S203, and S202 and S203 can be executed simultaneously or sequentially.

[0218] S204. The model management entity sends the second information to the first model running entity.

[0219] Correspondingly, the first model running entity receives the second information sent by the model management entity.

[0220] S204 is an optional step. The model management entity may choose not to inform the first model running entity of the unique monitoring scheme. Alternatively, the model management entity may send a second message to the first model running entity, using this second message to inform it of the unique monitoring scheme.

[0221] It should be noted that there is no temporal order between S203 and S204, and S203 and S204 can be executed simultaneously or sequentially.

[0222] Additionally, the model management entity can send a first response and a second message to the first model running entity, respectively. Alternatively, the first response may contain the second message.

[0223] S205. The model management entity sends the fourth message to the model monitoring and decision-making entity.

[0224] Correspondingly, the model monitoring and decision-making entity receives the fourth piece of information sent by the model management entity.

[0225] The fourth piece of information is used to indicate a set of monitoring resources used by the first model, and the fourth piece of information is related to the first and second pieces of information.

[0226] The model management entity can determine a set of monitoring resources for the first model based on the first information and the second information, so that the set of monitoring resources for the first model can support a unique monitoring scheme to monitor the first model, avoiding the problem of monitoring interruption or inability to monitor the first model due to insufficient resources.

[0227] Therefore, the model management entity can determine the fourth information based on a set of monitored resources of the first model, and send the fourth information to the model monitoring decision entity, so that the model monitoring decision entity can know the resources used by the first model.

[0228] In summary, the model management entity can determine a unique monitoring scheme and a set of monitoring resources for the first model. This is communicated to the model monitoring and data acquisition entity. The model management entity, through network-side decision-making, notifies the model monitoring and data acquisition entity of the unique monitoring scheme, and the model monitoring and decision-making entity of the set of monitoring resources used by the first model. Therefore, during the operation of the first model by the model running entity, the model monitoring and data acquisition entity and the model monitoring and decision-making entity can monitor the operation of the first model using the unique monitoring scheme based on the set of monitoring resources used by the first model.

[0229] S206. After the unique monitoring scheme is started, the model monitoring and acquisition entity monitors the first model according to the second information and obtains the monitoring measurement data of the first model. The monitoring measurement data is used to represent the operating status of the first model.

[0230] S207. The model monitoring acquisition entity sends monitoring measurement data to the model monitoring decision entity.

[0231] Correspondingly, the model monitoring decision entity receives the monitoring and measurement data sent by the model monitoring acquisition entity.

[0232] S208. The model monitoring decision entity generates the monitoring decision of the first model based on the monitoring measurement data.

[0233] Among them, S206-S208 are respectively with Figure 2 The implementation methods of S105-S107 in the embodiments are similar, and will not be described again here.

[0234] The interactive method for model monitoring provided in this application involves a model running entity sending a first request to a model management entity. This first request carries first information, which indicates the monitoring conditions for the first model and the need for monitoring. The model management entity then sends second information to a model monitoring and data collection entity based on the first information. The model monitoring and data collection entity determines a unique monitoring scheme based on the second information. Finally, the model management entity sends fourth information to a model monitoring decision entity based on the first and second information. This fourth information indicates a set of monitoring resources for the first model, enabling the unique monitoring scheme to utilize these resources to monitor the model's operation. Therefore, through the decision-making of the model management entity, a flexible monitoring scheme for the AI / ML model can be determined. Furthermore, the network side can comprehensively understand the monitoring conditions and available resources for monitoring the AI / ML model, facilitating better monitoring of its operation and achieving superior monitoring results.

[0235] Furthermore, after the sole monitoring scheme is activated, the model monitoring and acquisition entity and the model monitoring and decision-making entity begin monitoring the first model. The model monitoring and acquisition entity collects monitoring measurement data of the first model. The model monitoring and acquisition entity sends the monitoring measurement data of the first model to the model monitoring and decision-making entity. The model monitoring and decision-making entity generates a monitoring decision for the first model based on the monitoring measurement data. Thus, the monitoring process of the first model is realized.

[0236] In both Scenario 1 and Scenario 2, the model management entity can use various methods to determine the second information.

[0237] As a feasible implementation method, the model management entity can select one or more feasible monitoring schemes based on the available resources of the model management entity and the first information to determine the second information.

[0238] As another feasible implementation, the model management entity can determine the second information by selecting one or more feasible monitoring schemes based on its available resources, the available resources of the model monitoring decision entity, and the first information. Correspondingly, the second information is also used to indicate the resources used by a set of monitoring for the first model.

[0239] Below, in conjunction with Figure 4 The document describes in detail the specific implementation process by which the model management entity determines a set of monitoring resources used by the first model.

[0240] Please see Figure 4 , Figure 4 The diagram shows a flowchart of an interactive method for model monitoring provided in an embodiment of this application.

[0241] like Figure 4 As shown, the interactive method for model monitoring in this application may include:

[0242] S301, The model management entity sends a third request to the model monitoring decision entity.

[0243] Correspondingly, the model monitoring decision entity receives the third request sent by the model management entity.

[0244] The third request is used to request the model monitoring decision entity to configure a set of monitoring resources for the first model.

[0245] In some examples, the model monitoring decision entity may be idle, currently busy but idle after a preset time, or continuously busy. Based on this, the third request can be used to query the model monitoring decision entity whether its available resources can monitor the first model, that is, to query whether the model monitoring decision entity can configure available resources that meet the monitoring conditions of the first model. Here, the model monitoring decision entity can be one or more. The available resources can include at least one of idle CPU, idle GPU, or idle memory capacity.

[0246] In other examples, the model monitoring decision entity may have no resources, some resources, or all resources. Based on this, a third request can be used to query the model monitoring decision entity about the available resources for monitoring the first model, i.e., to query the model monitoring decision entity about its current available resources.

[0247] The allocatable resources here can include resources available before the first model is started. For example, currently idle available resources, and / or currently non-idle but idle available resources before the first model is started. These allocatable resources can include resources such as CPU, GPU, and memory, or predefined resources. These allocatable resources can be represented by resource type, name, or identifier (ID).

[0248] Therefore, the model management entity can send a third request to the model monitoring decision entity, requesting the model monitoring decision entity to configure a set of monitoring resources for the first model. In other words, the model management entity can query one or more model monitoring decision entities for a set of monitoring resources for the first model.

[0249] S302, The model monitoring decision entity sends a third response to the model management entity.

[0250] Correspondingly, the model management entity receives the third response sent by the model monitoring and decision-making entity.

[0251] The third response is used to instruct the model monitoring decision entity to monitor one or more sets of resources used by the first model. These sets of resources may include CPU, GPU, memory, or predefined resource types, names, or identifiers (IDs).

[0252] The model monitoring decision entity can determine one or more sets of monitoring resources for the first model based on its own allocable resources. The model monitoring decision entity can then determine the third response based on these sets of monitoring resources for the first model. In other words, the model monitoring decision entity can configure one or more resource usage schemes, each scheme indicating a set of monitoring resources for the first model.

[0253] Thus, the model monitoring decision entity can send a third response to the model management entity, which can inform the model management entity of one or more sets of monitoring resources configured by the model monitoring decision entity for monitoring the first model.

[0254] Additionally, the third response can also be used to indicate the model monitoring decision entities that use resources to configure the first model, such as at least one of name, identifier (ID), IP address, or port address. The third response can also indicate the validity period information of the monitoring scheme corresponding to the set of monitoring resources used by the first model, i.e., how long the monitoring scheme is valid and can fully utilize the available resources of the model monitoring decision entities.

[0255] S303. The model management entity determines a set of monitoring resources used by the first model based on the third response.

[0256] Based on the third response, the model management entity can select one set of monitoring resources from one or more sets of monitoring resources of the first model.

[0257] In the first model, a set of monitored resources may include CPU, GPU, memory, or predefined resources. These resources can be represented by their type, name, or identifier (ID).

[0258] In addition to the set of monitoring resources used by the first model, the model management entity may also identify the model monitoring decision entity that configures the set of monitoring resources used by the first model, such as at least one of name, identifier (ID), IP address, or port address.

[0259] In addition to the set of monitoring resources used by the first model, the model management entity can also determine whether the current time is outside the validity period information of the monitoring scheme corresponding to the set of monitoring resources used by the first model.

[0260] In summary, the model management entity can configure the monitoring resources for the first model through the model monitoring decision entity, which can ensure that the monitoring process of the first model starts normally.

[0261] In both Scenario 1 and Scenario 2, the first information can include various implementation methods. For example, the first information can be referred to as monitoring auxiliary information for the first model. In some examples, the first information may include at least one of the following: the monitoring types supported by the first model, the type of model monitoring decision entity monitoring the first model, or the performance metrics of the monitoring scheme supported by the first model.

[0262] The monitoring type may include at least one of the following: the data format of the input data of the first model, the data format of the output data of the first model, or parameters of the external environment of the first model. The data formats mentioned in this application include data type and data length.

[0263] Taking the first model for providing positioning services as an example, the data types of the input data for the first model may include: channel impulse response (CIR), power delay profile (PDP), delay profile (DP), reference signal received power (RSRP), and reference signal received path power (RSRPP). The data length of CIRD can be a complex number of 64, 128, or 256 units.

[0264] Taking the first model used to provide positioning services as an example, the data type of the output data of the first model can include: location information. For example, a three-dimensional relative position (x, y, z) with an accuracy of m, or latitude and longitude coordinates, or coordinate position in the WGS84 coordinate system, etc. The data format of the output data of the first model can include: coordinate format, latitude and longitude format, or coordinate format in the WGS84 coordinate system, etc. In addition, the data type of the output data of the first model can also include: intermediate value information related to the location information, such as quality parameters, time-related information, etc. For example, time-related information can be the time of arrival (ToA), or the reference signal time difference (RSTD), etc.

[0265] The parameters of the external environment may include, for example, the signal-to-noise ratio (SNR) or signal-to-interference-plus-noise ratio (SINR) of the external environment, as well as the synchronization status of the system time.

[0266] The types of entities that can be monitored and decided by the model include core network equipment, third-party network-side equipment, terminal equipment, and network equipment.

[0267] The performance metrics of a monitoring scheme are used to indicate the performance, effects, and impacts resulting from its use. These performance metrics may include, but are not limited to: computational complexity, latency characteristics caused by computation, system overhead, and information about the second model when monitoring is performed using a second model (such as an AI or ML model). Information about the second model may include its structure, parameters, complexity, and requirements for system software and / or hardware.

[0268] For example, computational complexity can be represented as high, medium, or low. The latency characteristics caused by computation can be represented by time granularity, such as milliseconds (ms), 10ms, 100ms, or simply by a numerical value. System overhead can be represented by metrics such as the amount of CPU, GPU, or memory used.

[0269] In both Scenario 1 and Scenario 2, the second information can be implemented in various ways. In some examples, the second information can include at least one of the following: first information, the identification (ID) of the monitoring scheme supported by the first model, the address of the model monitoring decision entity monitoring the first model, auxiliary information of the model monitoring decision entity monitoring the first model, the validity period information of the monitoring scheme supported by the first model, the valid response time information of the second information, or the priority or recommendation level of the monitoring scheme supported by the first model.

[0270] The address of the model monitoring decision entity can be an Internet Protocol (IP) address or a port address, etc.

[0271] Among them, the auxiliary information for the model monitoring decision entity is the metadata of the model monitoring decision entity, such as name or identifier (ID).

[0272] In this context, considering that after the monitoring scheme is determined, a set of monitoring resources for the first model will be allocated to the monitoring scheme for monitoring the first model, thus occupying a set of monitoring resources for the first model. Therefore, the second information includes the validity period information of the monitoring scheme. The validity period information indicates the timeframe during which the monitoring scheme can be activated, i.e., how long the monitoring scheme is effective and can fully utilize the available resources of the model monitoring decision entity. For example, the validity period information of the monitoring scheme can be represented by the latest activation time or the activation period.

[0273] The effective response time information for the second message is used to indicate the time limit within which a response to the second message can be made. For example, the effective response time information for the second message can be represented by the latest response time or a response period.

[0274] The priority or recommendation level of a monitoring scheme indicates the order in which it is recommended for adoption among all monitoring schemes, and can serve as a reference for the model monitoring entity to determine a unique monitoring scheme. The priority or recommendation level of a monitoring scheme can be set based on the model monitoring decision entity's resources, usage strategies (such as the frequency of use of the monitoring scheme), and actual conditions. Furthermore, the priority or recommendation level of a monitoring scheme can be included in the second information within the second scenario.

[0275] It should be noted that in Scenario 1, the monitoring scheme in the second information refers to each of one or more monitoring schemes. In Scenario 2, the monitoring scheme in the second information refers to a unique monitoring scheme.

[0276] In Scenario 1, the third information can be implemented in various ways. In some examples, the third information can include: the second information, a unique identifier (ID) of the monitoring scheme, the actual response time information of the third information, or the start time of the first monitoring model or at least one of the following: the start time of the first monitoring model.

[0277] Among them, the actual response time information of the third information is used to indicate the time when the third information is sent or received, such as by using timestamp information.

[0278] If the third information response is valid within the time limit indicated by the second information's valid response time information, it indicates that the third information response is valid. Otherwise, it indicates that the third information response is invalid. And / or, if the third information response is valid within the time limit indicated by the monitoring plan's validity period information, it indicates that the third information response is valid. Otherwise, it indicates that the third information response is invalid. Therefore, the model management entity can resend the second information to the model monitoring and acquisition entity to redetermine the monitoring plan.

[0279] The start time of the first monitoring model indicates how long after the third message the unique monitoring scheme will be initiated to monitor the first model. The start time of the first monitoring model can be the start moment or a start period. For example, it can start immediately, start after 500ms, start after an entity sends information to initiate the unique monitoring scheme, or start after receiving information from an entity to initiate the unique monitoring scheme.

[0280] The start event used to trigger the monitoring of the first model indicates which start event(s) trigger a unique monitoring model to monitor the first model. For example, startup begins upon receiving a start event from an entity.

[0281] Among them, one of the aforementioned entities can be any one of the model management entity, model monitoring and data collection entity, or model monitoring and decision-making entity.

[0282] In Scenario 2, the second information may also include: the startup time of the first monitoring model and / or the information used to trigger the startup event of the first monitoring model. For details on its implementation, please refer to the description in Scenario 1. It will not be elaborated here.

[0283] By way of example, this application also provides a communication device.

[0284] Figure 5 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application.

[0285] like Figure 5 As shown, the communication device 100 can exist independently or be integrated into other devices. It can communicate with one or more model running entities, monitoring and acquisition entities, and model monitoring and decision-making entities mentioned above to implement the operations corresponding to the model management entity in any of the above method embodiments.

[0286] The communication device 100 may include a processing unit 101 and a transceiver unit 102. The processing unit 101 is used for data processing, and the transceiver unit 102 can implement corresponding communication functions. The transceiver unit 102 may also be referred to as a communication interface or a communication unit.

[0287] Optionally, the communication device 100 may further include a storage unit, which may be used to store instructions and / or data, and the processing unit 101 may read the instructions and / or data in the storage unit so that the communication device 100 implements the aforementioned method embodiments.

[0288] The communication device 100 can be used to perform the actions performed by the model management entity in the preceding method embodiments. The communication device 100 can be the model management entity or a component configurable on the model management entity. The transceiver unit 102 is used to perform receiving-related operations of the model management entity in the preceding method embodiments.

[0289] Optionally, the transceiver unit 102 may include a sending unit and a receiving unit. The sending unit is used to perform the sending operation in the foregoing method embodiments. The receiving unit is used to perform the receiving operation in the foregoing method embodiments.

[0290] It should be noted that the communication device 100 may include a transmitting unit but not a receiving unit. Alternatively, the communication device 100 may include a receiving unit but not a transmitting unit. Specifically, it depends on whether the above-described scheme executed by the communication device 100 includes both transmitting and receiving actions.

[0291] As an example, the communication device 100 is used to perform the foregoing. Figure 2 and Figure 4 The actions performed by the model management entity in the illustrated embodiment.

[0292] The communication device 100 may include a processing unit 101 and a transceiver unit 102.

[0293] The processing unit 101 is used to obtain first information, which is used to indicate the monitoring conditions of the first model. The first model is deployed in the first model running entity and is used to provide a first service.

[0294] The transceiver unit 102 is used to send second information to the model monitoring and acquisition entity. The second information is used to instruct one or more monitoring schemes. The monitoring schemes are used to obtain the running status of the first model. The second information is related to the first information.

[0295] The transceiver unit 102 is also used to receive third information sent by the model monitoring and acquisition entity. The third information is used to indicate a unique monitoring scheme among one or more monitoring schemes. The third information is related to the second information.

[0296] The transceiver unit 102 is also used to send a fourth message to the model monitoring decision entity. The fourth message is used to instruct a set of monitoring resources used by the first model. The fourth message is related to the first message.

[0297] It should be understood that the corresponding processes performed by each unit have been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0298] The processing unit in the preceding embodiments can be implemented by at least one processor or processor-related circuitry. The transceiver unit 102 can be implemented by a transceiver or transceiver-related circuitry. The transceiver unit 102 can also be referred to as a communication unit or communication interface. The storage unit can be implemented by at least one memory.

[0299] In some examples, the processing unit 101 is specifically used to send a first request to one or more model running entities, the first request being used to request information corresponding to the monitoring conditions of the models deployed in the model running entities, the models deployed in the model running entities being used to provide a first service, and the one or more model running entities including at least a first model running entity; and to receive the first information sent by the first model running entity.

[0300] In some examples, the processing unit 101 is specifically used to obtain first information from a pre-stored second information set, the second information set including information corresponding to the monitoring conditions of models deployed in one or more model running entities, the models deployed in the model running entities being used to provide the first service, and the one or more model running entities including at least the first model running entity.

[0301] In some examples, the processing unit 101 is specifically used to send a second request to other network-side entities or third-party network devices, the second request being used to request first information, which is stored in the other network-side entities or third-party network devices; and to receive the first information sent by other network-side entities or third-party network devices.

[0302] In some examples, the processing unit 101 is also configured to determine fourth information based on the first information and the second information; or, the model management entity determines the fourth information based on the first information and the third information.

[0303] By way of example, this application also provides a communication device.

[0304] Figure 6 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application.

[0305] like Figure 6 As shown, the communication device 200 can exist independently or be integrated into other devices. It can communicate with the first model running entity, the monitoring and acquisition entity, and the model monitoring and decision-making entity mentioned above to implement the operations corresponding to the model management entity in any of the above method embodiments.

[0306] The communication device 200 may include a transceiver unit 201. The transceiver unit 201 may also be referred to as a communication interface or a communication unit.

[0307] Optionally, the communication device 200 may further include a storage unit and a processing unit. The storage unit can be used to store instructions and / or data, and the processing unit can read the instructions and / or data from the storage unit to enable the communication device 200 to implement the aforementioned method embodiments. The processing unit is used for data processing, and the transceiver unit 201 can implement corresponding communication functions.

[0308] The communication device 200 can be used to execute the actions performed by the model management entity in the preceding method embodiments. The communication device 200 can be the model management entity or a component configurable on the model management entity. The transceiver unit 201 is used to execute the receiving-related operations of the model management entity in the preceding method embodiments, and the processing unit is used to execute the processing-related operations of the model management entity in the preceding method embodiments.

[0309] Optionally, the transceiver unit 201 may include a sending unit and a receiving unit. The sending unit is used to perform the sending operation in the above method embodiments. The receiving unit is used to perform the receiving operation in the above method embodiments.

[0310] It should be noted that the communication device 200 may include a transmitting unit but not a receiving unit. Alternatively, the communication device 200 may include a receiving unit but not a transmitting unit. Specifically, it depends on whether the above-described scheme executed by the communication device 200 includes both transmitting and receiving actions.

[0311] As an example, the communication device 200 is used to perform the foregoing Figure 3 and Figure 4 The actions performed by the model management entity in the illustrated embodiment.

[0312] The communication device 200 may include a transceiver unit 201.

[0313] The transceiver unit 201 is used to receive a first request sent by the first model running entity. The first request is used to request monitoring of the first model. The first request carries first information, which is used to indicate the monitoring conditions of the first model. The first model is deployed in the first model running entity and is used to provide a first service.

[0314] The transceiver unit 201 is also used to send second information to the model monitoring and acquisition entity. The second information is used to indicate a unique monitoring scheme. The monitoring scheme is used to obtain the running status of the first model. The second information is related to the first information.

[0315] The transceiver unit 201 is also used to send a first response to the first model running entity, the first response being used to respond to the first request;

[0316] The transceiver unit 201 is also used to send a fourth message to the model monitoring decision entity. The fourth message is used to instruct a set of monitoring resources used by the first model. The fourth message is related to the first message and the second message.

[0317] It should be understood that the corresponding processes performed by each unit have been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0318] The processing unit in the preceding embodiments can be implemented by at least one processor or processor-related circuitry. The transceiver unit 201 can be implemented by a transceiver or transceiver-related circuitry. The transceiver unit 201 can also be referred to as a communication unit or communication interface. The storage unit can be implemented by at least one memory.

[0319] In some examples, transceiver unit 201 is also used to send second information to the first model running entity.

[0320] By way of example, this application also provides a communication device.

[0321] Figure 7 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application.

[0322] like Figure 7 As shown, the communication device 300 can exist independently or be integrated into other devices. It can communicate with one or more model running entities, model management entities, and model monitoring decision entities mentioned above to implement the operations corresponding to the model monitoring and acquisition entity in any of the above method embodiments.

[0323] The communication device 300 may include a transceiver unit 301. The transceiver unit 301 can implement corresponding communication functions. The transceiver unit 301 may also be referred to as a communication interface or a communication unit.

[0324] Optionally, the communication device 300 may further include a storage unit and a processing unit. The processing unit is used to perform data processing, and the storage unit can be used to store instructions and / or data. The processing unit can read the instructions and / or data in the storage unit so that the communication device 100 can implement the aforementioned method embodiments.

[0325] The communication device 300 can be used to perform the actions performed by the model monitoring and acquisition entity in the preceding method embodiments. The communication device 300 can be the model monitoring and acquisition entity or a component configurable on the model monitoring and acquisition entity. The transceiver unit 301 is used to perform reception-related operations of the model monitoring and acquisition entity in the preceding method embodiments.

[0326] Optionally, the transceiver unit 301 may include a sending unit and a receiving unit. The sending unit is used to perform the sending operation in the foregoing method embodiments. The receiving unit is used to perform the receiving operation in the foregoing method embodiments.

[0327] It should be noted that the communication device 300 may include a transmitting unit but not a receiving unit. Alternatively, the communication device 300 may include a receiving unit but not a transmitting unit. Specifically, it depends on whether the above-described scheme executed by the communication device 300 includes both transmitting and receiving actions.

[0328] As an example, the communication device 300 is used to perform the foregoing Figure 2 and Figure 4 The model in the illustrated embodiment monitors and collects the actions performed by the entity.

[0329] The communication device 300 may include a transceiver unit 301.

[0330] The transceiver unit 301 is used to receive second information sent by the receiving model management entity. The second information is used to indicate one or more monitoring schemes. The monitoring schemes are used to monitor the operation of the first model. The second information is related to the first information. The first information is obtained by the model management entity. The first information is used to indicate the monitoring conditions of the first model. The first model is deployed in the first model running entity. The first model is used to provide the first service.

[0331] The transceiver unit 301 is also used to send third information to the model management entity. The third information is used to indicate a unique monitoring scheme among one or more monitoring schemes. The third information is related to the second information.

[0332] It should be understood that the corresponding processes performed by each unit have been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0333] The processing unit in the preceding embodiments can be implemented by at least one processor or processor-related circuitry. The transceiver unit 301 can be implemented by a transceiver or transceiver-related circuitry. The transceiver unit 301 can also be referred to as a communication unit or communication interface. The storage unit can be implemented by at least one memory.

[0334] In some examples, the processing unit is used to monitor the first model based on third information after the unique monitoring scheme is started, and to obtain the monitoring measurement data of the first model, which is used to represent the operating status of the first model.

[0335] The transceiver unit 301 is also used to send monitoring measurement data to the model monitoring decision entity, and the monitoring measurement data is used by the model monitoring decision entity to generate monitoring decisions for the first model.

[0336] By way of example, this application also provides a communication device.

[0337] Figure 8 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application.

[0338] like Figure 8 As shown, the communication device 400 can exist independently or be integrated into other devices. It can communicate with the model management entity, model monitoring and acquisition entity and model monitoring and decision-making entity mentioned above to implement the operation corresponding to the first model running entity in any of the above method embodiments.

[0339] The communication device 400 may include a transceiver unit 401. The transceiver unit 401 can implement corresponding communication functions. The transceiver unit 401 may also be referred to as a communication interface or a communication unit.

[0340] Optionally, the communication device 400 may further include a storage unit and a processing unit. The processing unit is used to perform data processing. The storage unit can be used to store instructions and / or data. The processing unit can read the instructions and / or data in the storage unit so that the communication device 400 can implement the aforementioned method embodiments.

[0341] The communication device 400 can be used to execute the actions performed by the first model running entity in the aforementioned method embodiments. The communication device 400 can be the first model running entity or a component configurable on the first model running entity. The transceiver unit 401 is used to execute receiving-related operations of the first model running entity in the aforementioned method embodiments, and the processing unit is used to execute processing-related operations of the first model running entity in the aforementioned method embodiments.

[0342] Optionally, the transceiver unit 401 may include a sending unit and a receiving unit. The sending unit is used to perform the sending operation in the above method embodiments. The receiving unit is used to perform the receiving operation in the above method embodiments.

[0343] It should be noted that the communication device 400 may include a transmitting unit but not a receiving unit. Alternatively, the communication device 400 may include a receiving unit but not a transmitting unit. Specifically, it depends on whether the above-described scheme executed by the communication device 400 includes both transmitting and receiving actions.

[0344] As an example, the communication device 400 is used to perform the aforementioned Figure 3 and Figure 4 The actions performed by the first model running entity in the illustrated embodiment.

[0345] The communication device 400 may include a transceiver unit 401.

[0346] The transceiver unit 401 is used to send a first request to the model management entity. The first request is used to request monitoring of the first model. The first request carries first information, which is used to indicate the monitoring conditions of the first model. The first model is deployed in the first model running entity and is used to provide a first service.

[0347] The transceiver unit 401 is also used to receive a first response sent by the model management entity, the first response being used to respond to the first request.

[0348] It should be understood that the corresponding processes performed by each unit have been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0349] The processing unit in the preceding embodiments can be implemented by at least one processor or processor-related circuitry. The transceiver unit 401 can be implemented by a transceiver or transceiver-related circuitry. The transceiver unit 404 can also be referred to as a communication unit or communication interface. The storage unit can be implemented by at least one memory.

[0350] In some examples, the transceiver unit 401 is also used to receive second information sent by the model management entity. The second information is used to indicate a unique monitoring scheme, which is used to obtain the operating status of the first model. The second information is related to the first information.

[0351] By way of example, this application also provides a communication device.

[0352] Figure 9 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application.

[0353] like Figure 9 As shown, the communication device 500 can exist independently or be integrated into other devices. It can communicate with one or more model running entities, model management entities, and model monitoring decision entities mentioned above to implement the operations corresponding to the model monitoring and acquisition entity in any of the above method embodiments.

[0354] The communication device 500 may include a transceiver unit 501. The transceiver unit 501 can implement corresponding communication functions. The transceiver unit 501 may also be referred to as a communication interface or a communication unit.

[0355] Optionally, the communication device 500 may further include a storage unit and a processing unit. The processing unit is used to perform data processing, and the storage unit can be used to store instructions and / or data. The processing unit can read the instructions and / or data in the storage unit so that the communication device 500 can implement the aforementioned method embodiments.

[0356] The communication device 500 can be used to perform the actions performed by the model monitoring and acquisition entity in the preceding method embodiments. The communication device 500 can be the model monitoring and acquisition entity or a component configurable on the model monitoring and acquisition entity. The transceiver unit 501 is used to perform reception-related operations of the model monitoring and acquisition entity in the preceding method embodiments.

[0357] Optionally, the transceiver unit 501 may include a sending unit and a receiving unit. The sending unit is used to perform the sending operation in the foregoing method embodiments. The receiving unit is used to perform the receiving operation in the foregoing method embodiments.

[0358] It should be noted that the communication device 500 may include a transmitting unit but not a receiving unit. Alternatively, the communication device 500 may include a receiving unit but not a transmitting unit. Specifically, it depends on whether the above-described scheme executed by the communication device 500 includes both transmitting and receiving actions.

[0359] As an example, the communication device 500 is used to perform the aforementioned... Figure 3 and Figure 4 The model in the illustrated embodiment monitors and collects the actions performed by the entity.

[0360] The communication device 500 may include a transceiver unit 501.

[0361] The transceiver unit 501 is used to receive second information sent by the model management entity. The second information is used to indicate a unique monitoring scheme. The monitoring scheme is used to monitor the operation of the first model. The second information is related to the first information. The first information is carried in a first request. The first request is sent by the first model running entity to the model management entity. The first request is used to request monitoring of the first model. The first information is used to indicate the monitoring conditions of the first model. The first model is deployed in the first model running entity. The first model is used to provide the first service.

[0362] It should be understood that the corresponding processes performed by each unit have been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0363] The processing unit in the preceding embodiments can be implemented by at least one processor or processor-related circuitry. The transceiver unit 501 can be implemented by a transceiver or transceiver-related circuitry. The transceiver unit 501 can also be referred to as a communication unit or communication interface. The storage unit can be implemented by at least one memory.

[0364] In some examples, the processing unit is also used to monitor the first model according to the second information after the unique monitoring scheme is started, and to obtain the monitoring measurement data of the first model, which is used to represent the operating status of the first model.

[0365] The transceiver unit 501 is also used to send monitoring measurement data to the model monitoring decision entity, which is used by the model monitoring decision entity to generate monitoring decisions for the first model.

[0366] By way of example, this application also provides a communication device.

[0367] Figure 10 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application.

[0368] like Figure 10 As shown, the communication device 600 can exist independently or be integrated into other devices. It can communicate with one or more model running entities, model management entities, and model monitoring and acquisition entities mentioned above to implement the operations corresponding to the model monitoring and decision-making entity in any of the above method embodiments.

[0369] The communication device 600 may include a transceiver unit 601. The transceiver unit 601 can implement corresponding communication functions. The transceiver unit 601 may also be referred to as a communication interface or a communication unit.

[0370] Optionally, the communication device 600 may further include a storage unit and a processing unit. The processing unit is used to perform data processing. The storage unit can be used to store instructions and / or data. The processing unit can read the instructions and / or data in the storage unit so that the communication device 600 can implement the aforementioned method embodiments.

[0371] The communication device 600 can be used to execute the actions performed by the model monitoring and decision-making entity in the preceding method embodiments. The communication device 600 can be the model monitoring and decision-making entity or a component configurable on the model monitoring and decision-making entity. The transceiver unit 601 is used to perform receiving-related operations of the model monitoring and decision-making entity in the preceding method embodiments, and the processing unit is used to perform processing-related operations of the model monitoring and decision-making entity in the preceding method embodiments.

[0372] Optionally, the transceiver unit 601 may include a sending unit and a receiving unit. The sending unit is used to perform the sending operation in the above method embodiments. The receiving unit is used to perform the receiving operation in the above method embodiments.

[0373] It should be noted that the communication device 600 may include a transmitting unit but not a receiving unit. Alternatively, the communication device 600 may include a receiving unit but not a transmitting unit. Specifically, it depends on whether the above-described scheme executed by the communication device 600 includes both transmitting and receiving actions.

[0374] As an example, the communication device 600 is used to perform the aforementioned... Figure 2 and Figure 4 In the illustrated embodiment, the model monitors the actions performed by the decision-making entity.

[0375] The communication device 600 may include a transceiver unit 601.

[0376] The transceiver unit 601 is used to receive fourth information sent by the model management entity. The fourth information is used to indicate a set of monitoring resources used by the first model. The fourth information is related to the first information. The first information is used to indicate the monitoring conditions of the first model. The first model is deployed in the first model running entity and is used to provide the first service.

[0377] It should be understood that the corresponding processes performed by each unit have been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0378] The processing unit in the preceding embodiments can be implemented by at least one processor or processor-related circuitry. The transceiver unit 601 can be implemented by a transceiver or transceiver-related circuitry. The transceiver unit 601 can also be referred to as a communication unit or communication interface. The storage unit can be implemented by at least one memory.

[0379] In some examples, the fourth information is determined by the model management entity based on the first and second information; or, the fourth information is determined by the model management entity based on the first and third information.

[0380] The second information is related to the first information. The second information is used to indicate one or more monitoring schemes. The monitoring schemes are used to obtain the operating status of the first model. The third information is related to the second information. The third information is used to indicate the unique monitoring scheme among one or more monitoring schemes.

[0381] In some examples, the transceiver unit 601 is also used to receive monitoring measurement data sent by the model monitoring and acquisition entity. The monitoring measurement data is obtained by the model monitoring and acquisition entity after the unique monitoring scheme is started, based on third information, to monitor the first model. The monitoring measurement data is used to represent the operating status of the first model.

[0382] The processing unit is used to generate monitoring decisions based on the monitoring measurement data of the first model.

[0383] By way of example, this application also provides a communication device.

[0384] Figure 11 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application.

[0385] like Figure 11 As shown, the communication device 700 can exist independently or be integrated into other devices. It can communicate with one or more model running entities, model management entities, and model monitoring and acquisition entities mentioned above to implement the operations corresponding to the model monitoring and decision-making entity in any of the above method embodiments.

[0386] The communication device 700 may include a transceiver unit 701. The transceiver unit 701 can implement corresponding communication functions. The transceiver unit 701 may also be referred to as a communication interface or a communication unit.

[0387] Optionally, the communication device 700 may further include a storage unit and a processing unit. The processing unit is used to perform data processing, and the storage unit can be used to store instructions and / or data. The processing unit can read the instructions and / or data in the storage unit so that the communication device 700 can implement the aforementioned method embodiments.

[0388] The communication device 700 can be used to execute the actions performed by the model monitoring and decision-making entity in the preceding method embodiments. The communication device 700 can be the model monitoring and decision-making entity or a component configurable within it. The transceiver unit 701 is used to perform reception-related operations of the model monitoring and decision-making entity in the preceding method embodiments.

[0389] Optionally, the transceiver unit 701 may include a sending unit and a receiving unit. The sending unit is used to perform the sending operation in the foregoing method embodiments. The receiving unit is used to perform the receiving operation in the foregoing method embodiments.

[0390] It should be noted that the communication device 700 may include a transmitting unit but not a receiving unit. Alternatively, the communication device 700 may include a receiving unit but not a transmitting unit. Specifically, it depends on whether the above-described scheme executed by the communication device 700 includes both transmitting and receiving actions.

[0391] As an example, the communication device 700 is used to perform the aforementioned... Figure 3 and Figure 4 The model in the illustrated embodiment monitors the actions performed by the decision-making entity.

[0392] The communication device 700 may include a transceiver unit 701.

[0393] The transceiver unit 701 is used to receive fourth information sent by the model management entity. The fourth information is used to indicate a set of monitoring resources used by the first model. The fourth information is related to the first information and the second information. The first information is used to indicate the monitoring conditions of the first model. The first model is deployed in the first model running entity and is used to provide the first service. The second information is related to the first information and is used to indicate a unique monitoring scheme. The monitoring scheme is used to obtain the running status of the first model. The first information is carried in a first request. The first request is sent by the first model running entity to the model management entity and is used to request monitoring of the first model.

[0394] It should be understood that the corresponding processes performed by each unit have been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0395] The processing unit in the preceding embodiments can be implemented by at least one processor or processor-related circuitry. The transceiver unit 701 can be implemented by a transceiver or transceiver-related circuitry. The transceiver unit 701 can also be referred to as a communication unit or communication interface. The storage unit can be implemented by at least one memory.

[0396] In some examples, the transceiver unit 701 is also used to receive monitoring measurement data sent by the model monitoring and acquisition entity. The monitoring measurement data is obtained by the model monitoring and acquisition entity after the unique monitoring scheme is started, based on the second information, to monitor the first model. The monitoring measurement data is used to represent the operating status of the first model.

[0397] The processing unit is used to generate monitoring decisions based on the monitoring measurement data of the first model.

[0398] By way of example, this application also provides a communication device.

[0399] Figure 12 This is a schematic diagram of the hardware structure of a communication device provided in an embodiment of this application.

[0400] The communication device 800 includes a processor 801 coupled to a memory 802. The memory 802 is used to store computer programs or instructions and / or data. The processor 801 is used to execute the computer programs or instructions and / or data stored in the memory 802, so that the methods in the preceding method embodiments are executed.

[0401] Optionally, the communication device 800 may include one or more processors 801.

[0402] Optionally, such as Figure 12 As shown, the communication device 800 may also include a memory 802.

[0403] Optionally, the communication device 800 may include one or more memories 802.

[0404] Alternatively, the memory 802 may be integrated with the processor 801, or it may be set separately.

[0405] like Figure 12 As shown, the communication device 800 may further include a transceiver 803 for receiving and / or transmitting signals. For example, the processor 801 is used to control the transceiver 803 to receive and / or transmit signals.

[0406] As one approach, the communication device 800 is used to implement the operations performed by any one of the following entities in the method embodiments described above: the model running entity, the model management entity, the model monitoring and acquisition entity, and the model monitoring and decision-making entity.

[0407] For example, processor 801 is used to implement processing-related operations performed by any one of the model running entity, model management entity, model monitoring and acquisition entity, and model monitoring and decision-making entity in the method embodiment described above, and transceiver 803 is used to implement sending and receiving-related operations performed by any one of the model running entity, model management entity, model monitoring and acquisition entity, and model monitoring and decision-making entity in the method embodiment described above.

[0408] The above Figure 12In the communication device shown, the device in transceiver 803 used for receiving power can be considered a receiving unit, and the device in transceiver 803 used for transmitting can be considered a transmitting unit. That is, transceiver 803 can include a receiver and a transmitter. Transceiver 803 can also be called a transceiver unit, transceiver circuit, etc. Receiver can also be called a receiver, receiving unit, receiver, or receiving circuit, etc. Transmitter can also be called a transmitter, transmitter, transmitting unit, or transmitting circuit, etc. Processor 801 has processing functions and can be called a processing unit. Memory 802 is used to store computer program code and data; memory 802 can also be called a storage unit.

[0409] When the communication device is a chip, the chip includes a transceiver, a memory, and a processor. The transceiver can be an input / output circuit or a communication interface; the processor can be a processor integrated on the chip, a microprocessor, or an integrated circuit. In the above method embodiments, the sending operation of any one of the following entities—the model running entity, the model management entity, the model monitoring and acquisition entity, and the model monitoring and decision-making entity—can be understood as the chip's output, and the receiving operation of any one of these entities can be understood as the chip's input.

[0410] For example, this application also provides a computer-readable storage medium having stored thereon computer instructions for implementing the method executed by any one of the model running entity, model management entity, model monitoring and acquisition entity, and model monitoring and decision-making entity in the above method embodiments.

[0411] For example, when the computer program is executed by a computer, the computer can implement the method executed by any one of the following entities in the above method embodiments: the model running entity, the model management entity, the model monitoring and acquisition entity, and the model monitoring and decision-making entity.

[0412] For example, this application also provides a computer program product containing instructions that, when executed by a computer, cause the computer to implement the method performed by any one of the entities in the above method embodiments: the model running entity, the model management entity, the model monitoring and acquisition entity, and the model monitoring and decision-making entity.

[0413] For example, this application also provides an interactive system for model monitoring, which includes a model running entity, a model management entity, a model monitoring and data acquisition entity, and a model monitoring and decision-making entity for performing the above embodiments.

[0414] For example, this application also provides a communication device including a processor for calling computer programs or computer instructions stored in the memory to cause the processor to execute the methods of the above embodiments.

[0415] In one possible implementation, the input of the communication device corresponds to the above. Figures 2-4 The receiving operation in the illustrated embodiment corresponds to the output of the communication device described above. Figures 2-4 The sending operation in the illustrated embodiment.

[0416] Optionally, the processor is coupled to the memory via an interface.

[0417] Optionally, the communication device may also include a memory storing computer programs or computer instructions.

[0418] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, a baseband processor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of a program for controlling the methods of the preceding embodiments. The memory mentioned above can be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0419] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the explanations and beneficial effects of the relevant content in any of the communication devices provided above can be referred to the corresponding method embodiments provided above, and will not be repeated here.

[0420] In this application, any one of the following entities—the model execution entity, the model management entity, the model monitoring and acquisition entity, and the model monitoring and decision-making entity—can include a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. The hardware layer can include hardware such as a central processing unit (CPU), a memory management unit (MMU), and main memory. The operating system layer can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer can include applications such as browsers, address books, word processing software, and instant messaging software.

[0421] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0422] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0423] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0424] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0425] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the essential contribution of the technical solution of this application, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the processes of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0426] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An interactive method for model monitoring, characterized in that, The method includes: The model management entity obtains first information, which is used to indicate the monitoring conditions of the first model. The first information includes at least one of the monitoring types supported by the first model, the type of model monitoring decision entity that monitors the first model, or the performance indicators of the monitoring scheme supported by the first model. The monitoring type includes at least one of the data format of the input data of the first model, the data format of the output data of the first model, or the parameters of the external environment of the first model. The first model is deployed in the first model running entity and is used to provide a first service. The model management entity sends second information to the model monitoring and acquisition entity. The second information is used to instruct one or more monitoring schemes. The monitoring schemes are used to obtain the running status of the first model. The running status of the first model includes the running state of the first model and the running performance parameters of the first model. The second information is related to the first information. The model management entity receives third information sent by the model monitoring and acquisition entity. The third information is used to indicate a unique monitoring scheme among the one or more monitoring schemes, so that the available resources of the model management entity and the model monitoring and acquisition entity support the unique monitoring scheme. The third information is related to the second information. The model management entity sends a fourth message to the model monitoring decision entity. The fourth message is used to indicate a set of monitoring resources used by the first model. The fourth message is related to the first message. The set of monitoring resources used by the first model and the unique monitoring scheme are used to monitor the first model. The method for determining the second information includes: The model management entity determines the second information based on the available resources of the model management entity, the available resources of the model monitoring decision entity, and the first information. The second information is also used to indicate a set of monitoring resources used by the first model. The method for determining a set of monitoring resources used in the first model includes: The model management entity sends a third request to the model monitoring decision entity, the third request being used to request the model monitoring decision entity to configure a set of monitoring resources used by the first model; The model management entity receives a third response sent by the model monitoring decision entity, the third response being used to indicate one or more sets of monitoring resources configured by the model monitoring decision entity for the first model. The model management entity determines a set of monitoring resources used by the first model based on the third response; Wherein, the first model running entity or the model monitoring and acquisition entity is a terminal device or a network device; the model management entity or the model monitoring and decision-making entity is a network-side entity in the core network device or a third-party network device.

2. The method according to claim 1, characterized in that, The model management entity obtains the first information, including: The model management entity sends a first request to one or more model running entities. The first request is used to request information corresponding to the monitoring conditions of the models deployed in the model running entities. The models deployed in the model running entities are used to provide the first service. The one or more model running entities include at least the first model running entity. The model management entity receives the first information sent by the first model running entity.

3. The method according to claim 1, characterized in that, The model management entity obtains the first information, including: The model management entity obtains the first information from a pre-stored second information set, the second information set including information corresponding to the monitoring conditions of models deployed in one or more model running entities, the models deployed in the model running entities being used to provide the first service, and the one or more model running entities including at least the first model running entity.

4. The method according to claim 1, characterized in that, The model management entity obtains the first information, including: The model management entity sends a second request to other network-side entities or third-party network devices. The second request is used to request the first information, which is stored in the other network-side entities or the third-party network devices. The model management entity receives the first information sent by the other network-side entity or the third-party network device.

5. The method according to claim 1, characterized in that, The method further includes: The model management entity determines the fourth information based on the first information and the second information; Alternatively, the model management entity may determine the fourth information based on the first information and the third information.

6. An interactive method for model monitoring, characterized in that, The method includes: The model management entity receives a first request sent by the first model running entity. The first request is used to request monitoring of the first model. The first request carries first information, which is used to indicate the monitoring conditions of the first model. The first information includes at least one of the monitoring types supported by the first model, the type of model monitoring decision entity monitoring the first model, or the performance indicators of the monitoring scheme supported by the first model. The monitoring type includes at least one of the data format of the input data of the first model, the data format of the output data of the first model, or the parameters of the external environment of the first model. The first model is deployed in the first model running entity and is used to provide a first service. The model management entity sends a second piece of information to the model monitoring and acquisition entity. The second piece of information is used to indicate a unique monitoring scheme so that the available resources of the model management entity and the model monitoring and acquisition entity support the unique monitoring scheme. The monitoring scheme is used to obtain the running status of the first model, which includes the running status of the first model and the running performance parameters of the first model. The second piece of information is related to the first piece of information. The model management entity sends a first response to the first model running entity, and the first response is used to respond to the first request; The model management entity sends a fourth message to the model monitoring decision entity. The fourth message is used to indicate a set of monitoring resources used by the first model. The fourth message is related to the first message and the second message. The set of monitoring resources used by the first model and the unique monitoring scheme are used to monitor the first model. The method for determining the second information includes: The model management entity determines the second information based on the available resources of the model management entity, the available resources of the model monitoring decision entity, and the first information. The second information is also used to indicate a set of monitoring resources used by the first model. The method for determining a set of monitoring resources used in the first model includes: The model management entity sends a third request to the model monitoring decision entity, the third request being used to request the model monitoring decision entity to configure a set of monitoring resources used by the first model; The model management entity receives a third response from the model monitoring decision entity, the third response being used to indicate one or more sets of monitoring resources configured by the model monitoring decision entity for the first model. The model management entity determines a set of monitoring resources used by the first model based on the third response; Wherein, the first model running entity or the model monitoring and acquisition entity is a terminal device or a network device; the model management entity or the model monitoring and decision-making entity is a network-side entity in the core network device or a third-party network device.

7. The method according to claim 6, characterized in that, The method further includes: The model management entity sends the second information to the first model running entity.

8. An interactive method for model monitoring, characterized in that, The method includes: The model monitoring decision entity receives a fourth piece of information sent by the model management entity. The fourth piece of information is used to indicate a set of monitoring resources used by the first model. The fourth piece of information is related to the first information. The first information is used to indicate the monitoring conditions of the first model. The first information includes at least one of the monitoring types supported by the first model, the type of the model monitoring decision entity monitoring the first model, or the performance indicators of the monitoring scheme supported by the first model. The monitoring type includes at least one of the data format of the input data of the first model, the data format of the output data of the first model, or the parameters of the external environment of the first model. The first model is deployed in the first model running entity and is used to provide a first service. Wherein, the fourth information is determined by the model management entity based on the first information and the second information; or, the fourth information is determined by the model management entity based on the first information and the third information; Wherein, the second information is related to the first information, and the second information is used to indicate one or more monitoring schemes, the monitoring schemes being used to obtain the operating status of the first model, the operating status of the first model including the operating state of the first model and the operating performance parameters of the first model; the third information is related to the second information, and the third information is used to indicate a unique monitoring scheme among the one or more monitoring schemes, so that the available resources of the model management entity and the model monitoring and acquisition entity support the unique monitoring scheme; a set of monitoring resources for the first model and the unique monitoring scheme are used to monitor the first model; The method for determining a set of monitoring resources used in the first model includes: The model monitoring decision entity receives a third request sent by the model management entity, the third request being used to request the model monitoring decision entity to configure a set of monitoring resources for the first model; The model monitoring decision entity sends a third response to the model management entity. The third response is used to indicate one or more sets of monitoring resources configured by the model monitoring decision entity for the first model. The third response is used by the model management entity to determine a set of monitoring resources for the first model. Wherein, the first model running entity or the model monitoring and acquisition entity is a terminal device or a network device; the model management entity or the model monitoring and decision-making entity is a network-side entity in the core network device or a third-party network device.

9. The method according to claim 8, characterized in that, The method further includes: The model monitoring decision entity receives monitoring measurement data sent by the model monitoring acquisition entity. The monitoring measurement data is obtained by the model monitoring acquisition entity after the unique monitoring scheme is started, based on the third information, and is used to represent the operating status of the first model. The model monitoring decision entity generates the monitoring decision for the first model based on the monitoring measurement data.

10. An interactive method for model monitoring, characterized in that, The method includes: The model monitoring decision entity receives a fourth piece of information sent by the model management entity. This fourth piece of information indicates a set of monitoring resources used by the first model. The fourth piece of information is related to the first and second pieces of information. The first piece of information indicates the monitoring conditions of the first model. This information includes at least one of the following: the monitoring type supported by the first model, the type of model monitoring decision entity monitoring the first model, or the performance indicators of the monitoring scheme supported by the first model. The monitoring type includes at least one of the data format of the input data of the first model, the data format of the output data of the first model, or the parameters of the external environment of the first model. The first model is deployed in the first model running entity and provides a first service. The second piece of information is related to the first information and indicates a unique monitoring scheme so that the available resources of the model management entity and the model monitoring acquisition entity support the unique monitoring scheme. The monitoring scheme is used to obtain the operating status of the first model, which includes the operating state and the operating performance parameters of the first model. The first information is carried in a first request sent by the first model running entity to the model management entity. The first request requests monitoring of the first model. A set of monitoring resources used by the first model and the unique monitoring scheme are used to monitor the first model. The method for determining a set of monitoring resources used in the first model includes: The model monitoring decision entity receives a third request sent by the model management entity, the third request being used to request the model monitoring decision entity to configure a set of monitoring resources for the first model; The model monitoring decision entity sends a third response to the model management entity. The third response is used to indicate one or more sets of monitoring resources configured by the model monitoring decision entity for the first model. The third response is used by the model management entity to determine a set of monitoring resources for the first model. Wherein, the first model running entity or the model monitoring and acquisition entity is a terminal device or a network device; the model management entity or the model monitoring and decision-making entity is a network-side entity in the core network device or a third-party network device.

11. The method according to claim 10, characterized in that, The method further includes: The model monitoring decision entity receives monitoring measurement data sent by the model monitoring acquisition entity. The monitoring measurement data is obtained by the model monitoring acquisition entity after the unique monitoring scheme is started, based on the second information, and is used to represent the operating status of the first model. The model monitoring decision entity generates the monitoring decision for the first model based on the monitoring measurement data.

12. The method according to any one of claims 1-11, characterized in that, The first information includes at least one of the following: The monitoring types supported by the first model include at least one of the following: the data format of the input data of the first model, the data format of the output data of the first model, or the parameters of the external environment of the first model. Alternatively, monitor the type of model monitoring decision entity in the first model; Alternatively, the performance metrics of the monitoring scheme supported by the first model.

13. The method according to any one of claims 1-11, characterized in that, The second information includes at least one of the following: The first information; Alternatively, the identifier of the monitoring scheme supported by the first model; Alternatively, monitor the address of the decision entity in the first model; Alternatively, auxiliary information for monitoring the decision-making entity of the first model; or, validity information of the monitoring scheme supported by the first model. Alternatively, the effective response time information of the second information; Alternatively, the priority or recommendation level of the monitoring schemes supported by the first model.

14. The method according to any one of claims 1-5, 8-9, and 12-13, characterized in that, The third information includes at least one of the following: The second information; Or, a unique identifier for the monitoring solution; Alternatively, the actual response time information of the third information; Alternatively, it can monitor the startup time of the first model or trigger a startup event to monitor the first model.

15. An interactive system for model monitoring, characterized in that, The interactive system for model monitoring includes: one or more model running entities, a model monitoring and acquisition entity, a model management entity for performing the method according to any one of claims 1-5, 12-14, and a model monitoring decision entity for performing the method according to any one of claims 8-9, 12-14; And / or, a first model running entity, a model monitoring and acquisition entity, a model management entity for performing the method according to any one of claims 6-7, 12-13, and a model monitoring and decision-making entity for performing the method according to any one of claims 10-13; The one or more model running entities include at least a first model running entity.

16. A communication device, characterized in that, include: processor; The processor is configured to execute a computer-executable program or instructions in the memory, causing the communication device to perform the method according to any one of claims 1-7, 12-14; or causing the communication device to perform the method according to any one of claims 8-14.

17. A communication device, characterized in that, The communication device includes: Memory is used to store computer programs or instructions; A processor is configured to execute a computer program or instructions stored in the memory, causing the communication device to perform the method as described in any one of claims 1-7, 12-14; or causing the communication device to perform the method as described in any one of claims 8-14.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer-executable program or instructions configured to perform the method according to any one of claims 1-7, 12-14; or, the computer-executable program or instructions are configured to perform the method according to any one of claims 8-14.

19. A chip, characterized in that, include: An interface circuit and a logic circuit, wherein the interface circuit is used to receive signals from other chips outside the chip and transmit them to the logic circuit, or to send signals from the logic circuit to other chips outside the chip, and the logic circuit is used to implement the method as described in any one of claims 1-7, 12-14; or, the logic circuit is used to implement the method as described in any one of claims 8-14.

Images