Network performance supervision method, apparatus and device

Abstract

The application discloses a network performance supervision method, device and equipment, and belongs to the technical field of communication. The network performance supervision method of the application comprises the following steps: a first device receives first data from a first data network element, wherein the first data is performance supervision related data; and the first device determines first performance supervision feedback information of a first target based on the first data, wherein the first target is a model or an artificial intelligence (AI) task.

Description

Technical Field

[0001] This application belongs to the field of communication technology, specifically relating to a network performance monitoring method, apparatus, and device. Background Technology

[0002] The data plane was proposed in the discussion of relevant network architectures. The data plane consists of core network data plane functions, radio access network data plane functions, and terminal data plane functions, possessing end-to-end connectivity. The data plane is responsible for data control, including data collection coordination, data collection configuration, and data transmission configuration. The data plane is also responsible for functions such as data acquisition, data transmission, data preprocessing, data privacy and security, data analysis, data storage, and data services.

[0003] In related technologies, if a unified data plane framework (also known as a data acquisition framework) is deployed to connect data acquisition from various domains (including terminals, wireless access networks, core networks, access functions, etc.), then how to deploy and implement network-related performance monitoring based on this data plane framework is a problem that needs to be solved. Summary of the Invention

[0004] This application provides a network performance monitoring method, apparatus, and device that can solve the problem of network-related performance monitoring based on a data plane framework.

[0005] Firstly, a network performance monitoring method is provided, executed by a first device, the method comprising:

[0006] The first device receives first data from the first data network element, and the first data is performance monitoring related data.

[0007] Based on the first data, the first device determines the first performance supervision feedback information of the first target, which is a model or an artificial intelligence (AI) task.

[0008] Secondly, a network performance monitoring method is provided, executed by a fifth device, the method comprising:

[0009] The fifth device receives first data from the first data network element, and the first data is performance monitoring related data;

[0010] And / or, the fifth device receives second data from the first device or the fourth device, the second data being first performance supervision feedback information for the first target, the first target being a model or an artificial intelligence (AI) task;

[0011] The fifth device determines the second performance monitoring feedback information of the first target based on the first data and / or the second data.

[0012] Thirdly, a network performance monitoring method is provided, executed by a fourth device, the method comprising:

[0013] The fourth device receives a performance supervision request message from the fifth device. The performance supervision request message is used to request performance supervision of a first target, which is a model or an artificial intelligence (AI) task.

[0014] The fourth device determines at least one first device based on the identification information of the first target;

[0015] The fourth device sends a performance supervision request message to the at least one first device, the performance supervision request message being used to request the at least one first device to perform performance supervision on the first target;

[0016] or,

[0017] The fourth device sends a performance monitoring request message to the first device or the fifth device, the performance monitoring request message being used to request the first device or the fifth device to perform performance monitoring on the first target.

[0018] Fourthly, a network performance monitoring method is provided, executed by a second data network element, the method comprising:

[0019] The second data network element receives a data request message from the first device or the fifth device. The data request message is used to request the acquisition of first data, which is performance monitoring related data.

[0020] The second data network element sends a data provision message to the first data network element, the data provision message being used to notify the first data network element to send the first data.

[0021] Fifthly, a network performance monitoring method is provided, executed by a first data network element, the method comprising:

[0022] The first data network element receives a data provision message from the second data network element. The data provision message is used to notify the first data network element to send first data, which is performance monitoring related data.

[0023] The first data network element sends a data provision response message to the second data network element. The data provision response message includes at least one of the following: an indication to accept the data provision message; an indication to reject the data provision message; and information about the first data network element.

[0024] In a sixth aspect, a network performance monitoring device is provided, the device being configured to perform the steps of the method described in the first aspect, or implement the steps of the method described in the second aspect, or implement the steps of the method described in the third aspect, or implement the steps of the method described in the fourth aspect, or implement the steps of the method described in the fifth aspect.

[0025] In a seventh aspect, a network-side device is provided, the network-side device including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method as described in the first aspect, or the steps of the method as described in the second aspect, or the steps of the method as described in the third aspect, or the steps of the method as described in the fourth aspect, or the steps of the method as described in the fifth aspect.

[0026] Eighthly, a network-side device is provided, including a processor and a communication interface, wherein the processor and the communication interface are configured to implement the steps of the method described in the first aspect, or the steps of the method described in the second aspect, or the steps of the method described in the third aspect, or the steps of the method described in the fourth aspect, or the steps of the method described in the fifth aspect.

[0027] A ninth aspect provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the method described in the first aspect, or the steps of the method described in the second aspect, or the steps of the method described in the third aspect, or the steps of the method described in the fourth aspect, or the steps of the method described in the fifth aspect.

[0028] In a tenth aspect, a wireless communication system is provided, comprising: a network-side device, the network-side device being configured to perform the steps of the method described in the first aspect, or implement the steps of the method described in the second aspect, or implement the steps of the method described in the third aspect, or implement the steps of the method described in the fourth aspect, or implement the steps of the method described in the fifth aspect.

[0029] Eleventhly, a chip is provided, the chip including a processor and a communication interface coupled to the processor, the processor being configured to run a program or instructions to implement the steps of the method described in the first aspect, or the steps of the method described in the second aspect, or the steps of the method described in the third aspect, or the steps of the method described in the fourth aspect, or the steps of the method described in the fifth aspect.

[0030] In a twelfth aspect, a computer program / program product is provided, which is stored in a storage medium and is executed by at least one processor to implement the steps of the method as described in the first aspect, or the steps of the method as described in the second aspect, or the steps of the method as described in the third aspect, or the steps of the method as described in the fourth aspect, or the steps of the method as described in the fifth aspect.

[0031] In this embodiment, the first device receives performance supervision-related data from the first data network element and determines the performance information of the model or AI task based on the performance supervision-related data, thereby realizing performance supervision-related data interaction based on the data plane and network-related performance supervision. Attached Figure Description

[0032] Figure 1 This is a block diagram of a wireless communication system applicable to embodiments of this application;

[0033] Figure 2 One of the flowcharts of the network performance monitoring method provided in the embodiments of this application;

[0034] Figure 3 A second schematic flowchart illustrating the network performance monitoring method provided in this application embodiment;

[0035] Figure 4 The third flowchart illustrating the network performance monitoring method provided in this application embodiment;

[0036] Figure 5 The fourth flowchart illustrating the network performance monitoring method provided in this application embodiment;

[0037] Figure 6 Fifth flowchart illustrating the network performance monitoring method provided in this application embodiment;

[0038] Figure 7 A flowchart illustrating the network performance monitoring method provided in this application embodiment is shown in Figure 6.

[0039] Figure 8 The seventh flowchart illustrating the network performance monitoring method provided in this application embodiment;

[0040] Figure 9 Eighth flowchart illustrating the network performance monitoring method provided in this application embodiment;

[0041] Figure 10 This is one of the structural schematic diagrams of the network performance monitoring device provided in the embodiments of this application;

[0042] Figure 11A second schematic diagram of the network performance monitoring device provided in the embodiments of this application;

[0043] Figure 12 The third schematic diagram of the network performance monitoring device provided in the embodiments of this application;

[0044] Figure 13 Fourth schematic diagram of the network performance monitoring device provided in the embodiments of this application;

[0045] Figure 14 Fifth schematic diagram of the network performance monitoring device provided in the embodiments of this application;

[0046] Figure 15 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application;

[0047] Figure 16 This is a schematic diagram of the network-side device provided in an embodiment of this application. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0049] The terms "first," "second," etc., used in this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, the first object can be one or more. Furthermore, "or" in this application indicates at least one of the connected objects. For example, the scope of protection for "A or B" covers at least three scenarios: Scenario 1: including A but not B; Scenario 2: including B but not A; Scenario 3: including both A and B. In addition, the terms "A and / or B," "at least one of A and B," and "at least one of A or B" also cover at least the above three scenarios. The character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0050] The term "instruction" in this application can be either a direct instruction (or explicit instruction) or an indirect instruction (or implicit instruction). A direct instruction can be understood as the sender explicitly informing the receiver of specific information, the required operation, or the requested result in the instruction sent. An indirect instruction can be understood as the receiver determining the corresponding information based on the instruction sent by the sender, or making a judgment and determining the required operation or requested result based on the judgment result.

[0051] It is worth noting that the technologies described in this application are not limited to Long Term Evolution (LTE) / LTE-Advanced (LTE-A) systems, but can also be used in other wireless communication systems, such as Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency-Division Multiple Access (SC-FDMA), or other systems. The terms "system" and "network" in this application are often used interchangeably, and the described technologies can be used with the systems and radio technologies mentioned above, as well as with other systems and radio technologies. The following description describes New Radio (NR) systems for illustrative purposes, and NR terminology is used in most of the following description; however, these technologies can also be applied to systems other than NR systems, such as 6th generation (6G) radio systems. th Generation 6G communication system.

[0052] Figure 1This diagram illustrates a block diagram of a wireless communication system applicable to embodiments of this application. The wireless communication system includes a terminal 11 and a network-side device 12. The terminal 11 can also be referred to as User Equipment (UE), and can be a mobile phone, tablet computer, laptop computer, notebook computer, personal digital assistant (PDA), handheld computer, netbook, ultra-mobile personal computer (UMPC), mobile internet device (MID), augmented reality (AR), virtual reality (VR) device, robot, wearable device, flight vehicle, vehicle user equipment (VUE), shipboard equipment, pedestrian user equipment (PUE), smart home devices (home appliances with wireless communication capabilities, such as refrigerators, televisions, washing machines, or furniture), game consoles, personal computers (PCs), ATMs, or self-service machines, etc. Wearable devices include: smartwatches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart chains, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. Among these, in-vehicle devices can also be referred to as in-vehicle terminals, in-vehicle controllers, in-vehicle modules, in-vehicle components, in-vehicle chips, or in-vehicle units, etc. It should be noted that the specific type of terminal 11 is not limited in this application embodiment. Network-side equipment 12 may include access network equipment or core network equipment, wherein access network equipment may also be referred to as Radio Access Network (RAN) equipment, radio access network function, or radio access network unit. Access network equipment may include base stations, Wireless Local Area Network (WLAN) access points (APs), or Wireless Fidelity (WiFi) nodes, etc.Among them, base stations can be referred to as Node B (NB), Evolved Node B (eNB), Next Generation Node B (gNB), New Radio Node B (NRNode B), Access Point, Relay Base Station (RBS), Serving Base Station (SBS), Base Transceiver Station (BTS), Radio Base Station, Radio Transceiver, Basic Service Set (BSS), Extended Service Set (ESS), Home Node B (HNB), Home Evolved Node B, Transmit / Receive Point (TRP), and Non-Terrestrial Network (NTN) equipment (such as satellite or high-altitude platform). The term "base station" can be any suitable term in the field, such as "station" or any other appropriate term in the relevant field, as long as the same technical effect is achieved. The term "base station" is not limited to specific technical terms. It should be noted that the embodiments of this application only use the base station in the NR system as an example for introduction, and do not limit the specific type of base station.

[0053] Core network equipment, also known as core network nodes, core network functions, or core network elements, includes, but is not limited to, at least one of the following: Mobility Management Entity (MME), Access and Mobility Management Function (AMF), Session Management Function (SMF), User Plane Function (UPF), Policy Control Function (PCF), Policy and Charging Rules Function (PCRF), Edge Application Server Discovery Function (EASDF), Unified Data Management (UDM), Unified Data Repository (UDR), Home Subscriber Server (HSS), Centralized Network Configuration (CNC), Network Repository Function (NRF), Network Exposure Function (NEF), and Local NEF. The core network equipment includes NEF (L-NEF), Binding Support Function (BSF), Application Function (AF), Location Management Function (LMF), Gateway Mobile Location Centre (GMLC), Network Data Analytics Function (NWDAF), and Non-Terrestrial Network (NTN) equipment (such as satellites or high-altitude platform stations). It should be noted that this application embodiment only uses core network equipment in the NR system as an example and does not limit the specific type of core network equipment. If the name of the core network equipment mentioned in this application embodiment changes in subsequent protocol versions (e.g., 6G), it will still be within the scope of protection of this application.

[0054] Optionally, the core network equipment can be implemented by one or more functional modules in a single device, or by multiple devices working together; this application does not specifically limit this. It is understood that the aforementioned functional modules can be network elements in hardware devices, software functional modules running on dedicated hardware, or virtualized functional modules instantiated on a platform (e.g., a cloud platform).

[0055] The data plane was proposed in the discussion of relevant network architectures. The data plane consists of core network data plane functions, radio access network data plane functions, and UE data plane functions, possessing end-to-end connectivity. The data plane is responsible for data control, including data collection coordination, data collection configuration, and data transmission configuration. The data plane is also responsible for functions such as data acquisition, data transmission, data preprocessing, data privacy and security, data analysis, data storage, and data services.

[0056] 5G networks introduce an in-network AI intelligent framework, deploying NWDAF (Network Data Analyzer Framework). As an intelligent network element, NWDAF performs data analysis, generates results, and provides these results to other network elements for network optimization. NWDAF can be divided into two functional parts: model training and model inference. Model training generates a model, which is then provided to the inference function for model reasoning, thereby producing data analysis results (or inference results). However, these functions are only applicable within the core network (CN domain). Both model training and inference are deployed by the operator and do not include other participants or other domains.

[0057] In the 5G era, the core network proposes a unified data acquisition framework for AI services. Data sources and data consumers are connected via DCCF and MFAF, achieving isolation between them and enabling data to be collected once and reused multiple times. However, this is limited to the core network; the data source, data consumer, and DCCF are all core network functional elements, not involving other functions and elements such as UE and RAN. Furthermore, data interaction is based on the core network SBI interface, using the service-oriented architecture protocol HTTP, which has low data transmission efficiency and may not be suitable for the massive data transmission of multiple services in 6G networks.

[0058] The relevant technologies do not take into account the impact of introducing a data plane and / or AI model management platform, as well as the changes in architecture and processes. If a unified data acquisition framework is deployed in a 6G network to connect data acquisition from various domains (including UE, RAN, CN, AF, etc.), then how to deploy and implement network-related (but not limited to all deployed by operators) performance supervision based on this data plane framework is a problem that needs to be solved.

[0059] The network performance monitoring method provided in this application will be described in detail below with reference to the accompanying drawings, through some embodiments and application scenarios.

[0060] Figure 2 This is one of the flowcharts illustrating the network performance monitoring method provided in the embodiments of this application, such as... Figure 2 As shown, the network performance monitoring method includes steps 200 and 201.

[0061] Step 200: The first device receives first data from the first data network element, wherein the first data is performance monitoring related data;

[0062] Optionally, the first device is a device with model inference capabilities, such as a model inference device, which can also be called a model inference function.

[0063] Optionally, the first device or model inference device has at least one of the following functions: performing inference; acquiring analytical information (such as acquiring statistical data and / or predictions according to the analysis user's request) and providing corresponding services (such as data analysis services or inference services).

[0064] The first data network element can be a data module (DPF-DP) in a Data Plane Function (DPF), or it can be other devices with data plane functions or data plane-related capabilities, such as devices with the ability to process and forward data packets, manage and schedule traffic, and / or process network protocols.

[0065] It is understandable that the first data network element provides the first data to the first device via a data plane. It should be noted that "the first data network element provides the first data to the first device via a data plane" can mean that the first data network element sends the first data to the first device through data plane-related protocols, protocol stacks, or data buses.

[0066] Performance monitoring data refers to data used for performance monitoring or to determine performance information.

[0067] In some embodiments, the first data includes at least one of the following: inference input data; inference output data; and truth data.

[0068] Among them, inference input data refers to the input data required for inference based on AI models.

[0069] Inference output data, also known as inference result data, can be understood as the output results, data analysis results, or prediction results of the AI ​​model.

[0070] Ground truth data is the data that corresponds to inference data (inference input data or inference output data), such as network data, observable data, etc., and can also be called labeled data.

[0071] Step 201: Based on the first data, the first device determines the first performance supervision feedback information of the first target, where the first target is a model or an artificial intelligence (AI) task.

[0072] Understandably, the first device determines the performance information of the model or AI task based on performance supervision-related data, or determines the first performance supervision feedback information of the first target. This first performance supervision feedback information includes performance information, and may also include performance degradation indicators, calculated performance values, etc. Here, "AI task" refers to AI-related tasks, which can be tasks that require model or data analysis to generate results, such as AI localization. It should be noted that "task" can also refer to an event, analytics, data analysis, or data analysis task, etc., all conveying the same meaning.

[0073] It should be noted that the models described in the various embodiments of this application can be machine learning (ML) models, AI models, AI / ML models, analytical models, or mathematical models, etc., and the specific form is not limited.

[0074] It should be noted that performance can also refer to things like appearance.

[0075] It should be noted that performance information may include performance metrics and / or performance values, etc.

[0076] Optionally, performance information can be the performance value corresponding to the calculated performance index. Performance information can also be referred to as performance monitoring results, performance value, performance result, or performance status, etc.

[0077] The model inference device determines performance information based on the inference output data and the acquired label data. It can determine the model performance or task performance (performance information of one or more models for a specific task), which can also be referred to as the inference performance.

[0078] Optionally, the inference device compares the inference output data with the acquired ground truth to determine model performance, etc.

[0079] Optionally, the number of correct results in the inference output data can be determined by comparing the inference output data with the corresponding label data. The accuracy is then calculated by dividing the number of correct results by the total number of results. Accuracy can be understood as a performance metric.

[0080] The formula is expressed as follows:

[0081] Accuracy = Number of correct results ÷ Total number of results

[0082] The correct result can mean that the inference output data is consistent with the label data, or that the difference between the inference output data and the label result data is within an allowable range.

[0083] The accuracy can be expressed in various ways, not limited to a specific percentage value, such as 90%, or it can be expressed in a categorical form, such as high, medium, low, etc., or normalized data, such as 0.9.

[0084] It should be noted that performance supervision can also be understood as performance monitoring, performance determination, determining performance information or accuracy, etc.

[0085] The network performance monitoring method provided in this application embodiment involves a first device receiving performance monitoring-related data from a first data network element and determining the performance information of a model or AI task based on the performance monitoring-related data, thereby realizing performance monitoring-related data interaction based on the data plane and network-related performance monitoring.

[0086] In some embodiments, before the first device receives first data from the first data network element, the method further includes:

[0087] The first device sends a data request message to the second data network element, the data request message being used to request the acquisition of the first data.

[0088] It should be noted that the second data network element can be the control module (DPF-CP) in the data plane function, or other devices with data plane capability or data plane control capability.

[0089] Optionally, the second data network element is configured separately from the first data network element. After the first device sends a data request message to the second data network element, the second data network element interacts with the first data network element. The second data network element sends a data provision message to the first data network element, which notifies the first data network element to send the first data to the first device. The first data network element sends the first data to the first device based on the data plane, thereby the first device receives the first data from the first data network element based on the data plane.

[0090] Optionally, the second data network element and the first data network element are the same network element. The second data network element and the first data network element can be referred to as a second device, meaning that both functions are implemented through the same network element or device. Alternatively, the functions of the first and second data network elements can be combined, or the first and second data network elements can be considered two logical functional units within the second device. In the case of combined functions, the interaction information between the two will change from explicit external interaction to implicit internal interaction.

[0091] The network performance monitoring method provided in this application embodiment involves a first device sending a data request message to a second data network element and receiving performance monitoring-related data from the first data network element, thereby realizing data interaction based on the data plane. It should be noted that the data request message is not limited to a specific message name; any message used to request the first data is within the scope of protection of this application.

[0092] Optionally, the data request message carries at least one of the following:

[0093] 1) Description information of the first data;

[0094] Optionally, the description information includes at least one of the following:

[0095] Data type;

[0096] Data identification;

[0097] Data function description;

[0098] Data limitation information.

[0099] It is understandable that the data request message carries descriptive information about the first data, including the data type, data identifier, and / or data function description of the first data.

[0100] It should be noted that data types can also be indicated through event identifiers or other means.

[0101] A data identifier, or dataset identifier, is an identifier that can indicate one or more data items.

[0102] Data function description refers to the ability to indicate one or more data through the data function of one or more data, such as location-related data, data corresponding to a certain task, etc.

[0103] Optionally, the description information may also include data constraint information, or data filtering information, to further restrict the data. Examples include data generation time, data collection time, data generation region, and the corresponding data slice information.

[0104] 2) Information about the first device;

[0105] In one embodiment, the information of the first device may be information used to identify the first device.

[0106] Optionally, the information of the first device includes at least one of the following:

[0107] The identifier of the first device;

[0108] The IP address of the first device;

[0109] The port number of the first device;

[0110] The data plane tunnel endpoint identifier corresponding to the first device.

[0111] 3) Supported data transmission protocols and their corresponding protocol types;

[0112] That is, the type of protocol used for data transmission;

[0113] 4) The protocol type corresponding to the expected data transmission protocol;

[0114] Understandably, the data request message sent by the first device carries the protocol type corresponding to the data transmission protocol supported or expected by the first device, such as HTTP2, HTTP3, TCP or Quic, indicating which protocol type the first device negotiates with the second data network element to use for data transmission. The data transmission here includes performance supervision-related data transmission and AI model transmission.

[0115] 5) Data transmission time, used to indicate the transmission time of the first data.

[0116] It is understandable that the data request message sent by the first device carries the data transmission time, which is used to instruct the first data network element to send the first data at that time.

[0117] Optionally, the method further includes: the first device receiving a data request response message from the second data network element, the data request response message including at least one of the following: an indication to accept the data request message; an indication to reject the data request message; and a data transmission method.

[0118] It is understood that the data request response message is a response message sent by the second data network element to the data request message, used to inform the first device whether the data request message was successfully received. It should be noted that the data request response message is not a limitation on the message name.

[0119] The data transmission method is determined by the second data network element and can be a protocol type indicating the data transmission protocol to be used, such as a combination of QUIC, HTTP3, and protobuf protocols. Optionally, the data transmission method can also be negotiated between the second and first data network elements.

[0120] Optionally, the data request response message includes: indication information for accepting the data request message, and / or, the data transmission method.

[0121] Understandably, in one embodiment, the data request response message includes indication information for accepting the data request message, that is, explicitly informing the first device that the data request message has been successfully received. In another embodiment, the data request response message includes a data transmission method, that is, implicitly informing the first device that the data transmission protocol to be used corresponds to the protocol type, thereby informing the first device that the data request message has been successfully received. In yet another embodiment, the data request response message includes: indication information for accepting the data request message and a data transmission method.

[0122] Optionally, the data request response message includes: indication information to reject the data request message. In specific implementations, the rejection of the data request message can be explicitly indicated, or it can be implicitly indicated.

[0123] The network performance monitoring method provided in this application embodiment, after the first device sends a data request message, receives a data request response message from the second data network element, thereby realizing data interaction related to performance monitoring based on the data plane.

[0124] In some embodiments, the first device sends a data request message to the second data network element, including:

[0125] The first device sends a data network element lookup message to the third device, the data network element lookup message being used to request the lookup of a data network element;

[0126] The first device receives data network element information from the third device;

[0127] The first device sends the data request message to the second data network element based on the data network element information, wherein the data network information includes the information of the second data network element.

[0128] It is understandable that the first device searches for and discovers the data network element from the third device before sending a data request message to the second data network element.

[0129] Optionally, the third device is a network element with network storage capabilities, such as a Network Repository Function (NRF), which stores a network information database.

[0130] The first device sends a data network element lookup message to the third device. The data network element lookup message is used to request the lookup of a data network element. The third device returns data network element information to the first device. Further, the first device determines a second data network element based on the data network element information returned by the third device. Then, the first device sends a data request message to the second data network element to request the acquisition of the first data.

[0131] In some embodiments, the data network element information includes at least one of the following:

[0132] Information about the target data network element;

[0133] Indication information indicating that the second data network element and the first data network element are set separately;

[0134] The network supports instruction information for data interaction based on the data plane;

[0135] The data plane protocol information supported by the target data network element.

[0136] It is understandable that the data network element information returned by the third device is used to indicate the data network elements that can provide relevant data for performance monitoring.

[0137] The data network element information includes: information about the target data network element, indicating that the target data network element can provide performance monitoring-related data; indication information that the second data network element and the first data network element are separately configured, indicating that the DPF contains an independent DP module in addition to the CP module; indication information that the network supports data interaction based on the data plane, indicating that the network supports data interaction using a data plane mechanism (a unified data acquisition framework); and information about the data plane protocols supported by the target data network element, i.e., the protocol types corresponding to the supported data transmission protocols. The supported data plane protocol information can be a combination of protocol types, such as using QUCI, HTTP3, and protobuf.

[0138] The unified data acquisition framework has at least one of the following functions: all data exchange is based on the data plane; data and / or model storage; and data preprocessing.

[0139] In some embodiments, before the first device receives first data from the first data network element, the method further includes:

[0140] The first device receives a performance monitoring request message from a fourth, fifth, or sixth device, the performance monitoring request message being used to request the first device to perform performance monitoring on the first target.

[0141] Optionally, the fourth device is a device with model management capabilities, such as an AI platform (AIPlatform), which is a device in the network responsible for AI / ML management. An AI platform can also be called an AI model management platform.

[0142] AI platforms typically have at least one of the following functions: model registration / publishing; model security verification; model usage authorization; PC interoperability verification (allowing interoperable models between producers and consumers, based on vendor ID or format); model storage; and distributed AI management.

[0143] Optionally, the AI ​​platform can be co-located with the DPF. This means the functions of the AI ​​platform are implemented by the DPF (or DPF-CP). With this co-location, the interaction between the two will change from explicit external interaction to implicit internal interaction.

[0144] The fifth device is a device with model training capabilities, such as a model training device, which can be responsible for training AI models and can also be called a model training function.

[0145] The sixth device is a consumer device (Consumer NF, or NF service consumer, or NFconsumer), such as a device that sends a task request to the first device (e.g., requests the first device to obtain a certain prediction result, etc.), and can be any device in the network.

[0146] Before performing model supervision, the model inference device may receive performance supervision request messages from consumer devices, AI platforms, or model training devices. These messages may request supervision of the model or AI task's performance, or request the computation of performance information to determine performance details. It's important to note that the performance supervision request message does not restrict the message name.

[0147] In one implementation, the consumer device may send a performance supervision request message to the model inference device when requesting an AI task or afterward to request performance information for supervising the relevant AI task.

[0148] In one implementation, the AI ​​platform or model training device may also send a performance supervision request message to the model inference device when sending the model, or after sending the AI ​​model to the model inference device, to request supervision of the AI ​​model's performance information, such as the performance information of the model in actual / specific tasks.

[0149] Understandably, in one implementation, after receiving a performance monitoring request message from a fourth, fifth, or sixth device, the first device sends a data network element lookup message to the third device. The first device receives data network element information from the third device, determines a second data network element based on the data network element information, sends a data request message to the second data network element, and receives first data from the first data network element.

[0150] Optionally, the first device can also perform performance supervision on the model or AI task based on its own implementation logic.

[0151] The network performance supervision method provided in this application embodiment involves a first device receiving a performance supervision request message from an AI platform, a model training device, or a consumer device, and performing performance supervision based on the performance supervision request message, thereby realizing network performance supervision.

[0152] In some embodiments, the performance monitoring request message carries at least one of the following:

[0153] 1) Identification information of the first target;

[0154] The primary target identification information, such as task identifier and model identifier, is used to indicate the model or task that the performance monitoring is targeting.

[0155] 2) The type of the first target;

[0156] 3) Performance indicators, used to indicate the calculation method of performance values;

[0157] For example, for classification problems, performance metrics could be accuracy, while for regression problems, performance metrics could be average error.

[0158] 4) Threshold value, used to indicate the threshold for triggering the reporting of performance monitoring feedback information;

[0159] When the calculated performance value is lower than the threshold, a performance monitoring feedback message is triggered and reported.

[0160] 5) Conditions for reporting performance monitoring feedback information.

[0161] Performance monitoring feedback can be reported periodically or triggered by events, such as when the performance value corresponding to a performance indicator reaches a threshold value, triggering the reporting of performance information.

[0162] In some embodiments, the first device determines first performance supervision feedback information for the first target based on the first data, including:

[0163] The first device determines the first performance monitoring feedback information of the first target based on the first data and the performance monitoring request message.

[0164] Optionally, based on the performance monitoring request message, the first device can determine, according to the acquired first data, to calculate the performance index of the first target, thereby obtaining the performance value corresponding to the performance index, and based on the performance value, determine whether the performance of the first target has decreased, and determine the first performance monitoring feedback information of the first target.

[0165] In some embodiments, the method further includes:

[0166] The first device sends the first performance monitoring feedback information to the fourth, fifth, or sixth device.

[0167] Optionally, after the first device calculates the performance value corresponding to the performance index, it reports the performance supervision feedback information according to the reporting conditions of the performance supervision feedback information, that is, sends the first performance supervision feedback information to the AI ​​platform, model training device or consumer device.

[0168] Optionally, the first device sends a performance result notification message to the fourth, fifth, or sixth device, the performance result notification message carrying the first performance monitoring feedback information.

[0169] Optionally, the first device may also perform performance supervision on the model or AI task according to its own implementation logic, and send first performance supervision feedback information to the AI ​​platform, model training device or consumer device.

[0170] In some embodiments, the first performance supervision feedback information includes at least one of the following:

[0171] 1) Performance values;

[0172] The performance value is calculated based on the first set of data.

[0173] 2) Identification information of the first target, such as the identifier of the AI ​​model or the identifier of the AI ​​task;

[0174] 3) Performance indicators;

[0175] Performance metrics are used to indicate how performance values ​​are calculated.

[0176] 4) Monitoring range, used to indicate the monitoring range corresponding to the performance value;

[0177] The scope of supervision includes a time range or a geographical range, which is the range of information corresponding to the data used to determine performance values.

[0178] It should be noted that supervision can also refer to determining the meaning of performance, etc.

[0179] 5) The first data;

[0180] The first data refers to the data used to determine performance values, such as inference input data, inference output data, and label data.

[0181] 6) Identification information of the first data;

[0182] 7) Storage information of the first data;

[0183] It should be noted that the first data can be sent directly through the first performance monitoring feedback information, or it can be sent by sending the storage identifier corresponding to the first data and / or the stored device information (i.e., the storage information of the first data) to instruct the peer device to obtain the first data.

[0184] 8) Performance degradation indication information, used to indicate a performance degradation of the first target;

[0185] Based on the calculated performance value, determine whether the performance of the first target has declined. If it is determined that the performance of the first target has declined, then provide feedback on the performance decline indication.

[0186] 9) First instruction information, used to instruct the replacement of the first target or the model corresponding to the first target;

[0187] The first instruction is an instruction from the model inference device requesting a model replacement. For example, the model inference device requests the AI ​​platform to replace it with a model that is suitable for the same task (such as a model that can solve the same task).

[0188] 10) Second indication information, used to indicate updating the first target or the model corresponding to the first target;

[0189] The second instruction information is used to instruct the model to be updated. For example, the model inference device requests the AI ​​platform or model training device to retrain the model in order to update the model.

[0190] Optionally, the model corresponding to the first target can be updated by combining the first data and the second instruction information, or the model corresponding to the first target can be updated based on the first data.

[0191] 11) Third instruction information, used to indicate fine-tuning of the first target or the model corresponding to the first target.

[0192] For example, the model inference device requests the AI ​​platform or model training device to fine-tune the model via a third instruction.

[0193] Optionally, by combining the first data and the third instruction information, the model corresponding to the first target can be fine-tuned, or the model corresponding to the first target can be fine-tuned based on the first data.

[0194] The network performance monitoring method provided in this application embodiment involves a first device reporting performance information of a model or AI task to an AI platform or model training device, thereby achieving network performance monitoring.

[0195] Optionally, the first device receives a new model, or an updated model, from an AI platform or model training device.

[0196] Figure 3 This is a second flowchart illustrating the network performance monitoring method provided in this application embodiment. Figure 3 As shown, network performance monitoring methods include:

[0197] Step 300a: The consumer device sends a performance supervision request message to the model inference device;

[0198] Step 300b: The model training device or AI platform sends a performance supervision request message to the model inference device;

[0199] Optionally, the performance monitoring request message is used to request the first device to perform performance monitoring on the first target. The performance monitoring request message carries at least one of the following: identification information of the first target; type of the first target; performance index, used to indicate the calculation method of the performance value; threshold value, used to indicate the threshold for triggering the reporting of performance monitoring feedback information; and reporting conditions for performance monitoring feedback information.

[0200] Step 301: The model inference device sends a data request message to the control module (DPF-CP) in the data plane function. The data request message carries at least one of the following: description information of the first data; information of the first device; protocol type corresponding to the supported data transmission protocol; protocol type corresponding to the desired data transmission protocol; and data transmission time, used to indicate the transmission time of the first data.

[0201] Step 302: The DPF-CP sends a data request response message to the model inference device. The data request response message includes at least one of the following: an indication to accept the data request message; an indication to reject the data request message; and a data transmission method.

[0202] Step 303: The DPF-CP interacts with the data module (DPF-DP) in the data plane function, including: the DPF-CP sending a data provision message to the DPF-DP, the data provision message being used to notify the first data network element to send the first data; the DPF-DP sending a data provision response message to the DPF-CP, the data provision response message including at least one of the following: an indication to accept the data provision message; an indication to reject the data provision message; and information about the DPF-DP.

[0203] Step 304a: DPF-DP sends inference data to the model inference device based on the data, including inference input data and / or inference output data;

[0204] Step 304b: DPF-DP sends truth data to the model inference device based on data;

[0205] Step 305: The model inference device determines the performance value of the first target;

[0206] Step 306a: The model inference device sends a performance result notification message to the AI ​​platform or model training device. The performance result notification message carries the performance value, as well as indications that the performance has deteriorated, the model needs to be replaced, or the model needs to be retrained / fine-tuned.

[0207] Step 306b: The model inference device sends a performance result notification message to the consumer device. The performance result notification message carries the performance value, as well as indications that the performance has deteriorated, the model needs to be replaced, or the model needs to be retrained / fine-tuned.

[0208] Step 307: The AI ​​platform sends the updated model or the reselected model to the model inference device.

[0209] Some of the steps in steps 300a to 307 above are optional, and can be referred to the description in the foregoing embodiments.

[0210] The network performance monitoring method provided in this application embodiment involves a first device responding to a performance monitoring request message or obtaining performance monitoring-related data from the data plane according to its own logic, and determining the performance information of the model or AI task based on the performance monitoring-related data, thereby realizing performance monitoring-related data interaction based on the data plane and network-related performance monitoring.

[0211] Figure 4 The third flowchart illustrating the network performance monitoring method provided in this application embodiment is as follows: Figure 4 As shown, the network performance monitoring method includes steps 400 and 401.

[0212] Step 400: The fifth device receives first data from the first data network element, wherein the first data is performance monitoring related data;

[0213] And / or, the fifth device receives second data from the first device or the fourth device, the second data being first performance supervision feedback information for the first target, the first target being a model or an artificial intelligence (AI) task;

[0214] It should be noted that the fifth device is a device with model training capabilities, such as a model training device, which can be responsible for model training and can also be referred to as a model training function.

[0215] In this embodiment of the application, the fifth device collects first data based on the data plane to determine the performance information of the model or AI task, and / or determines the performance information of the model or AI task based on the performance supervision results of other devices, where the performance supervision results of other devices are the second data sent by the first device or the fourth device.

[0216] In one embodiment, the fifth device receives second data from the first device, the second data being the performance monitoring result obtained by the first device from performing performance monitoring on the first target.

[0217] In one embodiment, the fifth device receives second data from the fourth device, which is obtained by the fourth device processing the performance monitoring results of at least one first device.

[0218] Step 401: The fifth device determines the second performance monitoring feedback information of the first target based on the first data and / or the second data.

[0219] The model training device calculates the final performance supervision result of the model or AI task based on the first data it collects and / or the performance supervision result (i.e., the second data) obtained by the AIPlatform or model inference device to perform performance supervision on the model or AI task. For example, it can be calculated by averaging, weighted averaging, etc., and then generates the second performance supervision feedback information of the first objective.

[0220] Optionally, the model training device can determine the performance supervision results of the model or AI task based on the first data, such as inference output data and corresponding label data, and then generate second performance supervision feedback information. Performance supervision results can also be referred to as performance values, performance results, performance information, performance status, etc.

[0221] Optionally, the model training device determines the second performance supervision feedback information of the first objective based on the second data, namely the performance supervision results fed back by the model inference device or AI platform.

[0222] Optionally, the model training device performs operations such as averaging or weighted averaging based on the first data and the second data to obtain the final performance supervision result of the first objective and generate the second performance supervision feedback information.

[0223] Optionally, the second performance monitoring feedback information may include performance monitoring results, and may also include performance degradation indicators, calculated performance values, etc.

[0224] The network performance supervision method provided in this application embodiment enables the fifth device to obtain performance supervision-related data from the data plane and / or determine the performance information of the model or AI task based on the performance supervision results of other devices, thereby realizing performance supervision-related data interaction based on the data plane and network-related performance supervision.

[0225] In some embodiments, before the fifth device receives the first data from the first data network element, the method further includes:

[0226] The fifth device sends a data request message to the second data network element, the data request message being used to request the acquisition of the first data.

[0227] It should be noted that the second data network element can be the control module (DPF-CP) in the data plane function, or other devices with data plane capability or data plane control capability.

[0228] Optionally, the second data network element is configured separately from the first data network element. After the fifth device sends a data request message to the second data network element, the second data network element interacts with the first data network element. The second data network element sends a data provision message to the first data network element, which notifies the first data network element to send the first data to the fifth device. The first data network element sends the first data to the fifth device based on the data plane, thereby the fifth device receives the first data from the first data network element based on the data plane.

[0229] Optionally, the second data network element and the first data network element are the same network element. The second data network element and the first data network element can be referred to as a second device, meaning that both functions are implemented through the same network element or device. Alternatively, the functions of the first and second data network elements can be combined, or the first and second data network elements can be considered two logical functional units within the second device. In the case of combined functions, the interaction information between the two will change from explicit external interaction to implicit internal interaction.

[0230] The network performance monitoring method provided in this application embodiment enables a fifth device to receive performance monitoring-related data from a second data network element by sending a data request message to the second data network element, thereby realizing data interaction related to performance monitoring based on the data plane. It should be noted that the data request message is not limited to a specific message name; any message used to request the first data is within the scope of protection of this application.

[0231] In some embodiments, the data request message carries at least one of the following:

[0232] 1) Description information of the first data;

[0233] The descriptive information includes at least one of the following:

[0234] Data type;

[0235] Data identification;

[0236] Data function description;

[0237] Data limitation information.

[0238] For an understanding of the description information of the first data, please refer to the description in the foregoing embodiments, which will not be repeated here.

[0239] 2) Information about the fifth device;

[0240] In one embodiment, the information of the fifth device may be information used to identify the first device.

[0241] Optionally, the information of the fifth device includes at least one of the following:

[0242] The identification information of the fifth device;

[0243] The IP address of the fifth device;

[0244] The port number of the fifth device;

[0245] The data plane tunnel endpoint identifier corresponding to the fifth device.

[0246] 3) Supported data transmission protocols and their corresponding protocol types;

[0247] 4) The protocol type corresponding to the expected data transmission protocol;

[0248] Understandably, the data request message sent by the fifth device carries the protocol type corresponding to the data transmission protocol supported or expected by the fifth device, such as HTTP2, HTTP3, TCP or Quic, indicating which protocol type the fifth device negotiates with the second data network element to use for data transmission. The data transmission here includes performance supervision-related data transmission and AI model transmission.

[0249] 5) Data transmission time, used to indicate the time when the first data was transmitted.

[0250] It is understandable that the data request message sent by the fifth device carries the data transmission time, which is used to instruct the first data network element to send the first data at that time.

[0251] Optionally, the method further includes:

[0252] The fifth device receives a data request response message from the second data network element. The data request response message includes at least one of the following: an indication to accept the data request message; an indication to reject the data request message; and a data transmission method.

[0253] It is understood that the data request response message is a response message sent by the second data network element to the data request message, used to inform the fifth device whether the data request message was successfully received. It should be noted that the data request response message is not a limitation on the message name.

[0254] The data transmission method is determined by the second data network element and can be a protocol type indicating the data transmission protocol to be used, such as a combination of QUIC, HTTP3, and protobuf protocols. Optionally, the data transmission method can also be negotiated between the second and first data network elements.

[0255] Optionally, the data request response message includes: indication information for accepting the data request message, and / or, the data transmission method.

[0256] Understandably, in one implementation, the data request response message includes indication information for accepting the data request message, that is, explicitly informing the fifth device that the data request message has been successfully received. In another implementation, the data request response message includes a data transmission method, that is, by informing the fifth device of the protocol type corresponding to the data transmission protocol to be used, implicitly informing the first device that the data request message has been successfully received. In yet another implementation, the data request response message includes: indication information for accepting the data request message and a data transmission method.

[0257] Optionally, the data request response message includes: indication information to reject the data request message. In specific implementations, the rejection of the data request message can be explicitly indicated, or it can be implicitly indicated.

[0258] The network performance monitoring method provided in this application embodiment, after the fifth device sends a data request message, receives a data request response message from the second data network element, thereby realizing data interaction related to performance monitoring based on the data plane.

[0259] In some embodiments, before the fifth device sends a data request message to the second data network element, the method further includes:

[0260] The fifth device sends a data network element lookup message to the third device, the data network element lookup message being used to request the lookup of a data network element;

[0261] The fifth device receives data network element information from the third device;

[0262] The fifth device sends the data request message to the second data network element based on the data network element information, wherein the data network information includes the information of the second data network element.

[0263] Optionally, the third device is a network element with network storage capabilities, such as a Network Repository Function (NRF), which stores a network information database.

[0264] The fifth device sends a data network element lookup message to the third device. The data network element lookup message is used to request the lookup of a data network element. The third device returns data network element information to the fifth device. Further, the fifth device determines the second data network element based on the data network element information returned by the third device. Then, the fifth device sends a data request message to the second data network element to request the acquisition of the first data.

[0265] In some embodiments, the data network element information includes at least one of the following:

[0266] Information about the target data network element;

[0267] Indication information indicating that the second data network element and the first data network element are set separately;

[0268] The network supports instruction information for data interaction based on the data plane;

[0269] The data plane protocol information supported by the target data network element.

[0270] It is understandable that the data network element information returned by the third device is used to indicate the data network elements that can provide relevant data for performance monitoring.

[0271] The data network element information includes: information about the target data network element, indicating that the target data network element can provide performance monitoring-related data; indication information that the second data network element and the first data network element are separately configured, indicating that the DPF contains an independent DP module in addition to the CP module; indication information that the network supports data interaction based on the data plane, indicating that the network supports data interaction using a data plane mechanism (a unified data acquisition framework); and information about the data plane protocols supported by the target data network element, i.e., the protocol types corresponding to the supported data transmission protocols. The supported data plane protocol information can be a combination of protocol types, such as using QUCI, HTTP3, and protobuf.

[0272] The unified data acquisition framework has at least one of the following functions: all data exchange is based on the data plane; data and model storage; and data preprocessing.

[0273] In some embodiments, before the fifth device receives the first data from the first data network element, the method further includes:

[0274] The fifth device receives a performance monitoring request message from the first device or the fourth device, the performance monitoring request message being used to request the fifth device to perform performance monitoring on the first target.

[0275] In some embodiments, the fourth device is a device with model management capabilities, such as an AI platform (AIPlatform), which is a device in the network responsible for AI / ML management. An AI platform can also be called an AI model management platform.

[0276] AI platforms typically have at least one of the following functions: model registration / publishing; model security verification; model usage authorization; PC interoperability verification (allowing interoperable models between producers and consumers, based on vendor ID or format); model storage; and distributed AI management.

[0277] Optionally, the AI ​​platform can be co-located with the DPF. This means the functionality of the AI ​​platform is implemented by the DPF (or DPF-CP). With this co-location, the interaction between the two will change from explicit external interaction to implicit internal interaction.

[0278] The first device is a device with model inference capability, such as a model inference device, or a model inference function.

[0279] Before model training, the device may receive a performance supervision request message from the AI ​​platform or model inference device to request supervision of the performance of the model or AI task.

[0280] Understandably, in one implementation, after receiving a performance monitoring request message from the first or fourth device, the fifth device sends a data network element lookup message to the third device. The fifth device receives data network element information from the third device, determines the second data network element based on the data network element information, sends a data request message to the second data network element, and receives the first data from the first data network element.

[0281] Optionally, the fifth device can also perform performance supervision on the model or AI task based on its own implementation logic.

[0282] The network performance supervision method provided in this application embodiment involves a fifth device receiving a performance supervision request message from an AI platform or a model inference device, and performing performance supervision based on the performance supervision request message, thereby realizing network performance supervision.

[0283] In some embodiments, before the fifth device receives second data from the first device or the fourth device, the method further includes:

[0284] The fifth device sends a performance monitoring request message to the first device or the fourth device, the performance monitoring request message being used to request the first device or the fourth device to perform performance monitoring on the first target.

[0285] Understandably, the fifth device can also send a performance monitoring request message to the first or fourth device to obtain the performance monitoring results from other devices.

[0286] In some embodiments, the fifth device sends a performance monitoring request message to the first device, including:

[0287] The fifth device sends a performance monitoring request message to the first device through a service-based architecture or through a service.

[0288] or,

[0289] The fifth device sends a performance monitoring request message to the first device through the fourth device.

[0290] Understandably, model training devices can directly send performance supervision request messages to model inference devices, for example, through a service-based architecture (SBA) or through services.

[0291] Optionally, the model training device sends performance supervision request messages to the model inference device via a forwarding mechanism from the AI ​​platform.

[0292] Optionally, the model training device can also send a performance supervision request message to the AI ​​platform. The AI ​​platform can identify devices that can perform performance supervision (e.g., one or more inference devices that use (have requested) the model) based on information such as model identification, and send a performance supervision request message to the devices that can perform performance supervision.

[0293] The network performance supervision method provided in this application embodiment allows the fifth device to send a performance supervision request message to an AI platform or model inference device to obtain the performance supervision results of other devices, thereby realizing network performance supervision.

[0294] In some embodiments, the performance monitoring request message carries at least one of the following:

[0295] 1) Identification information of the first target;

[0296] The primary target identification information, such as task identifier and model identifier, is used to indicate the model or AI task that the performance supervision is targeting.

[0297] 2) The type of the first target;

[0298] 3) Performance indicators, used to indicate the calculation method of performance values;

[0299] For example, for classification problems, performance metrics could be accuracy, while for regression problems, performance metrics could be average error.

[0300] 4) Threshold value, used to indicate the threshold for triggering the reporting of performance monitoring feedback information;

[0301] For example, when the calculated performance result is lower than the threshold, a performance monitoring feedback message is triggered and reported.

[0302] 5) Conditions for reporting performance monitoring feedback information.

[0303] Performance monitoring feedback information can be reported periodically or triggered by events, such as when the performance value corresponding to a performance indicator reaches a threshold value, triggering the reporting of performance monitoring feedback information.

[0304] In some embodiments, the second data is the first performance monitoring feedback information of the first target determined by the first device, or the second data is determined by the fourth device based on the performance monitoring feedback information of the first target fed back by at least one first device.

[0305] It is understood that if the fifth device sends a performance monitoring request message to the first device through a service-based architecture or via a service; or if the fifth device sends a performance monitoring request message to the first device through the fourth device, and the fifth device then receives the performance monitoring feedback information returned by the first device, i.e., the first performance monitoring feedback information of the first target. For an understanding of the first performance monitoring feedback information, please refer to the description in the foregoing embodiments.

[0306] Optionally, the fifth device sends a performance supervision request message to the fourth device (i.e., the AI ​​platform). The AI ​​platform can determine the devices that can perform performance supervision (e.g., one or more inference devices that use (have requested) the model) based on information such as model identification, and send a performance supervision request message to the devices that can perform performance supervision. The AI ​​platform receives and processes the performance supervision feedback information returned by one or more inference devices, such as averaging or weighted averaging the performance values ​​in the performance supervision feedback information or using other aggregation methods to obtain the second data.

[0307] In some embodiments, the method further includes:

[0308] The fifth device performs a first operation, the first operation including at least one of the following:

[0309] The fifth device performs a first operation, the first operation including at least one of the following:

[0310] Send the second performance monitoring feedback information to the first device or the fourth device;

[0311] Retrain the first target or the model corresponding to the first target;

[0312] Reselect the first target or the model corresponding to the first target;

[0313] Fine-tune the first target or the model corresponding to the first target;

[0314] Send the model, which has been retrained, reselected, or fine-tuned, to the first or fourth device.

[0315] It is understandable that after the fifth device determines the second performance supervision feedback information of the first target based on the first data and / or the second data, it sends the second performance supervision feedback information to the first device or the fourth device.

[0316] The fifth device can also be used to retrain, reselect, and fine-tune the model.

[0317] Optionally, if it is determined from the performance supervision results of the first target that the first target or the model corresponding to the first target needs to be retrained, then the fifth device retrains the first target or the model corresponding to the first target.

[0318] Optionally, if it is determined from the performance supervision results of the first target that the first target or the model corresponding to the first target needs to be reselected, then the fifth device reselects the first target or the model corresponding to the first target.

[0319] Optionally, if it is determined from the performance monitoring results of the first target that the first target or the model corresponding to the first target needs to be fine-tuned, then the fifth device fine-tunes the first target or the model corresponding to the first target.

[0320] Further, optionally, the model obtained after retraining, reselection, or fine-tuning can be sent to the first device or the fourth device.

[0321] The network performance supervision method provided in this application embodiment involves a fifth device determining the performance result of a model or AI task based on first data and / or second data, then sending performance supervision feedback information to achieve network performance supervision. Furthermore, the method allows for retraining, reselecting, or fine-tuning the model as needed, and sending the retrained, reselected, or fine-tuned model to optimize network performance.

[0322] In some embodiments, the second performance supervision feedback information includes at least one of the following:

[0323] 1) Performance values;

[0324] 2) Identification information of the first target;

[0325] 3) Performance indicators;

[0326] Performance metrics are used to indicate how performance values ​​are calculated.

[0327] 4) Monitoring range, used to indicate the monitoring range corresponding to the performance value;

[0328] The scope of supervision includes either a time frame or a geographical area.

[0329] 5) Performance monitoring data used to calculate the performance values;

[0330] It should be noted that the performance supervision related data here includes the first data mentioned above, as well as the performance supervision related data used to calculate the performance value corresponding to the first performance supervision feedback information.

[0331] 6) Identification information or storage information corresponding to the performance monitoring-related data used to calculate the performance value;

[0332] It should be noted that the performance monitoring data used to calculate the performance value can be sent directly through this message, or the storage identifier and / or the stored device information (i.e., storage information) corresponding to the performance monitoring data can be sent to instruct the peer device to obtain the performance monitoring data.

[0333] 7) Performance degradation indication information, used to indicate a performance degradation of the first target;

[0334] Based on the calculated performance value, determine whether the performance of the first target has declined. If it is determined that the performance of the first target has declined, then provide feedback on the performance decline indication.

[0335] 8) Fourth instruction information, used to instruct the replacement of the first target or the model corresponding to the first target;

[0336] The fourth instruction is an instruction from the model training device requesting a model replacement. For example, the model training device may request the AI ​​platform to replace the model with one that is designed for the same task (such as a model that can solve the same task).

[0337] 9) Fifth instruction information, used to instruct the updating of the first target or the model corresponding to the first target;

[0338] The fifth instruction information is used to instruct or request a model update. For example, the model training device may use the fifth instruction information to request the AI ​​platform to retrain the model in order to update the model.

[0339] Optionally, the model corresponding to the first target can be updated by combining the performance supervision-related data used to calculate the performance value and the fifth indication information, or the model corresponding to the first target can be updated based on the performance supervision-related data used to calculate the performance value.

[0340] 10) The sixth instruction information is used to indicate fine-tuning of the first target or the model corresponding to the first target.

[0341] For example, the model training device requests the AI ​​platform to fine-tune the model via a sixth instruction message.

[0342] Optionally, the model corresponding to the first target can be fine-tuned by combining the performance supervision-related data used to calculate the performance value and the sixth indication information, or the model corresponding to the first target can be fine-tuned based on the performance supervision-related data used to calculate the performance value.

[0343] In some embodiments, the performance supervision related data includes at least one of the following: inference input data, inference output data, and truth data.

[0344] The network performance supervision method provided in this application embodiment involves a fifth device determining the performance result of a model or AI task based on first data and / or second data, then sending performance information to achieve network performance supervision. Furthermore, it allows for the retraining or reselection of AI models as needed, thereby optimizing network performance.

[0345] Figure 5 The fourth flowchart illustrating the network performance monitoring method provided in this application embodiment is as follows: Figure 5 As shown, network performance monitoring methods include:

[0346] Step 500a: The model inference device sends a performance supervision request message to the model training device;

[0347] Step 500b: The AI ​​platform sends a performance supervision request message to the model training device;

[0348] Step 501a: The model training device sends a performance supervision request message to the AI ​​platform;

[0349] Step 502: The AI ​​platform distributes a performance supervision request message to at least one model inference device;

[0350] Understandably, an AI platform can identify devices capable of performance supervision (e.g., one or more inference devices that use (or have requested) the model) based on information such as model identification, and distribute performance supervision request messages to at least one model inference device capable of performance supervision. The AI ​​platform receives and processes the performance supervision results returned by one or more inference devices, such as by averaging, weighted averaging, or other aggregation methods.

[0351] Step 501b: The model training device sends a performance supervision request message to the model inference device;

[0352] Optionally, the performance monitoring request message is used to request the first device to perform performance monitoring on the first target. The performance monitoring request message carries at least one of the following: identification information of the first target; type of the first target; performance index, used to indicate the calculation method of the performance value; threshold value, used to indicate the threshold for triggering the reporting of performance monitoring feedback information; and reporting conditions for performance monitoring feedback information.

[0353] Step 503: The model training device sends a data request message to the control module (DPF-CP) in the data plane function. The data request message carries at least one of the following: description information of the first data; information of the first device; protocol type corresponding to the supported data transmission protocol; protocol type corresponding to the desired data transmission protocol; and data transmission time, used to indicate the transmission time of the first data.

[0354] Step 504: The DPF-CP sends a data request response message to the model training device. The data request response message includes at least one of the following: an indication to accept the data request message; an indication to reject the data request message; and a data transmission method.

[0355] Step 505: The DPF-CP interacts with the data module (DPF-DP) in the data plane function, including: the DPF-CP sending a data provision message to the DPF-DP, the data provision message being used to notify the first data network element to send the first data; the DPF-DP sending a data provision response message to the DPF-CP, the data provision response message including at least one of the following: an indication to accept the data provision message; an indication to reject the data provision message; and information about the DPF-DP.

[0356] Step 505a: DPF-DP sends inference data, including inference input data and / or inference output data, to the model training device based on the data.

[0357] Step 505b: DPF-DP sends ground truth data to the model training device based on the data;

[0358] Inference data and truth data constitute the first data.

[0359] Step 506: The model inference device determines the performance information of the first target;

[0360] Step 507a: The model inference device sends a performance result notification message to the model training device. The performance result notification message carries the performance information of the first target and also includes indications of performance degradation, need to replace the model, or need to retrain the model.

[0361] Step 507b: The model inference device sends a performance result notification message to the AI ​​platform. The performance result notification message carries the performance information of the first objective, and also includes indications of performance degradation, need to replace the model, or need to retrain the model.

[0362] Step 508: The AI ​​platform sends a performance result notification message to the model training device. The performance result notification message carries the second data and also includes indications that the performance has deteriorated, the model needs to be replaced, or the model needs to be retrained.

[0363] Step 509: The model training device determines the second performance supervision feedback information of the first target based on the first data and / or the second data;

[0364] Step 510: The model training device sends the second performance supervision feedback information to the AI ​​platform;

[0365] Step 511: The model training device sends the second performance supervision feedback information to the model inference device;

[0366] Step 512: The model training equipment performs model retraining and other operations.

[0367] The network performance supervision method provided in this application embodiment involves a model training device acquiring performance supervision-related data and / or performance supervision results from other devices through a data plane. Based on the performance supervision-related data and / or performance supervision results from other devices, the final performance result of the model or AI task is determined, and second performance supervision feedback information is generated. This achieves network performance supervision, and the AI ​​model can be retrained or reselected as needed to optimize network performance.

[0368] Figure 6 The fifth flowchart illustrating the network performance monitoring method provided in this application embodiment is as follows: Figure 6 As shown, the network performance monitoring method includes steps 600, 601, and 602.

[0369] Step 600: The fourth device receives a performance supervision request message from the fifth device. The performance supervision request message is used to request performance supervision of a first target, which is a model or an artificial intelligence (AI) task.

[0370] Optionally, the fourth device is a device with model management capabilities, such as an AI platform (AIPlatform) device, which is responsible for model management in the network. The model here can be a machine learning (ML) model, an AI model, an AI / ML model, an analytical model, or a mathematical model, etc., and its specific form is not limited.

[0371] Optionally, the AI ​​platform can be co-located with the DPF. This means the functions of the AI ​​platform are implemented by the DPF (or DPF-CP). With this co-location, the interaction between the two will change from explicit external interaction to implicit internal interaction.

[0372] Optionally, the fifth device is a device with model training capabilities, such as a model training device, which can be responsible for model training.

[0373] In some embodiments, the performance monitoring request message carries at least one of the following:

[0374] The identification information of the first target;

[0375] The type of the first target;

[0376] Performance metrics are used to indicate the methods for calculating performance values;

[0377] Threshold value, used to indicate the threshold for triggering the reporting of performance monitoring feedback information;

[0378] Conditions for reporting performance monitoring feedback information.

[0379] Step 601: The fourth device determines at least one first device based on the identification information of the first target;

[0380] The fourth device receives a performance monitoring request message from the fifth device and identifies at least one first device based on the identification information of the first target carried in the performance monitoring request message.

[0381] Optionally, the first device is a device with model inference capabilities, such as a model inference device, which can also be called a model inference function.

[0382] Step 602: The fourth device sends a performance supervision request message to the at least one first device, the performance supervision request message being used to request the at least one first device to perform performance supervision on the first target;

[0383] The fourth device sends a performance monitoring request message to the at least one first device to obtain the performance monitoring results of the at least one first device on the first target, that is, the performance information of the at least one first device on the first target.

[0384] The network performance monitoring method provided in this application embodiment involves a fourth device receiving a performance monitoring request message from a fifth device, identifying at least one first device based on the identification information of the first target carried in the performance monitoring request message, and sending a performance monitoring request message to the at least one first device. This allows the fourth device to obtain the performance monitoring results of the at least one first device for processing, thereby realizing network performance monitoring and improving the accuracy of network performance monitoring.

[0385] Figure 7 The sixth flowchart illustrating the network performance monitoring method provided in this application embodiment is as follows: Figure 7 As shown, the network performance monitoring method includes step 603.

[0386] Step 603: The fourth device sends a performance monitoring request message to the first device or the fifth device. The performance monitoring request message is used to request the first device or the fifth device to perform performance monitoring on the first target.

[0387] The network performance monitoring method provided in this application embodiment involves a fourth device sending a performance monitoring request message to a first or fifth device, thereby realizing the network performance monitoring process after introducing an AI platform.

[0388] In some embodiments, the method further includes:

[0389] The fourth device receives performance monitoring feedback information of a first target from the at least one first device;

[0390] The fourth device processes the performance supervision feedback information of the first target from the at least one first device to obtain the first performance supervision feedback information of the first target.

[0391] The fourth device sends the first performance monitoring feedback information of the first target to the fifth device.

[0392] Optionally, processing the performance information of the first target from the at least one first device includes: averaging, weighted averaging, or other data aggregation methods. After processing the performance values ​​in the performance supervision feedback information of the first target from the at least one first device, the fourth device obtains the first performance supervision feedback information of the first target, and sends the first performance supervision feedback information of the first target to the fifth device. The first performance supervision feedback information of the first target is the second data.

[0393] The network performance monitoring method provided in this application embodiment allows a fourth device to process the performance information of at least one first device, thus realizing the network performance monitoring process after introducing an AI platform.

[0394] In some embodiments, the method further includes:

[0395] The fourth device receives second performance monitoring feedback information from the fifth device. The second performance monitoring feedback information is determined by the fifth device based on the first performance monitoring feedback information and / or first data from the first data network element. The first data is performance monitoring related data.

[0396] Understandably, the fifth device, based on the first data and / or the second data, determines the final performance result of the first objective, generates second performance supervision feedback information, and sends this second performance supervision feedback information to the fourth device. The fourth device receives the second performance supervision feedback information from the fifth device.

[0397] In some embodiments, the second performance supervision feedback information includes at least one of the following:

[0398] 1) Performance values;

[0399] Performance values ​​can also be referred to as performance monitoring results or performance outcomes.

[0400] 2) Identification information of the first target, such as the identifier of the AI ​​model or the identifier of the AI ​​task;

[0401] 3) Performance indicators, used to indicate the calculation method of performance values;

[0402] 4) Monitoring range, used to indicate the monitoring range corresponding to the performance value;

[0403] The scope of supervision includes either a time frame or a geographical area.

[0404] 5) Performance monitoring data used to calculate the performance values;

[0405] It may include first data, and performance supervision-related data used to calculate the performance value corresponding to the first performance supervision feedback information;

[0406] 6) Identification information or storage information corresponding to the performance monitoring-related data used to calculate the performance value;

[0407] It should be noted that the performance monitoring data used to calculate the performance value can be sent directly through this message, or the storage identifier and / or the stored device information (i.e., storage information) corresponding to the performance monitoring data can be sent to instruct the peer device to obtain the performance monitoring data.

[0408] 7) Performance degradation indication information, used to indicate a performance degradation of the first target;

[0409] 8) Fourth instruction information, used to instruct the replacement of the first target or the model corresponding to the first target;

[0410] The fourth instruction is an instruction from the model training device requesting a model replacement. For example, the model training device may request the AI ​​platform to replace the model with one that is designed for the same task (such as a model that can solve the same task).

[0411] 9) Fifth instruction information, used to instruct the updating of the first target or the model corresponding to the first target.

[0412] The fifth instruction is an instruction from the model training device requesting an update to the model. For example, the model training device requests the AI ​​platform to retrain the model in order to update the model.

[0413] Optionally, the model corresponding to the first target can be updated by combining the performance supervision-related data used to calculate the performance value and the fifth indication information, or the model corresponding to the first target can be updated based on the performance supervision-related data used to calculate the performance value.

[0414] 10) The sixth instruction information is used to indicate fine-tuning of the first target or the model corresponding to the first target.

[0415] For example, the model training device requests the AI ​​platform to fine-tune the model via a sixth instruction message.

[0416] Optionally, the model corresponding to the first target can be fine-tuned by combining the performance supervision-related data used to calculate the performance value and the sixth indication information, or the model corresponding to the first target can be fine-tuned based on the performance supervision-related data used to calculate the performance value.

[0417] The network performance monitoring method provided in this application embodiment involves a fourth device receiving second performance monitoring feedback information returned by a fifth device, thereby realizing the network performance monitoring process after introducing an AI platform.

[0418] In some embodiments, the method further includes:

[0419] The fourth device receives first performance monitoring feedback information from the first device, which is determined by the first device based on first data from the first data network element.

[0420] It is understandable that after the fourth device sends a performance monitoring request message to the first device, it receives the first performance monitoring feedback information from the first device.

[0421] Optionally, the first performance supervision feedback information includes at least one of the following:

[0422] 1) Performance values;

[0423] The performance value is calculated based on the first data and can also be called the performance monitoring result or performance result.

[0424] 2) Identification information of the first target, such as the identifier of the AI ​​model or the identifier of the AI ​​task;

[0425] 3) Performance indicators;

[0426] Performance metrics are used to indicate how performance values ​​are calculated.

[0427] 4) Monitoring range, used to indicate the monitoring range corresponding to the performance value;

[0428] The scope of supervision includes either a time frame or a geographical area.

[0429] 5) The first data;

[0430] The first data refers to the data used to determine performance values, such as inference input data, inference output data, and label data.

[0431] 6) Identification information of the first data;

[0432] 7) Storage information of the first data;

[0433] It should be noted that the first data can be sent directly through the first performance monitoring feedback information, or it can be sent by sending the storage identifier corresponding to the first data and / or the stored device information (i.e., the storage information of the first data) to instruct the peer device to obtain the first data.

[0434] 8) Performance degradation indication information, used to indicate a performance degradation of the first target;

[0435] Based on the calculated performance value, determine whether the performance of the first target has declined. If it is determined that the performance of the first target has declined, then provide feedback on the performance decline indication.

[0436] 9) First instruction information, used to instruct the replacement of the first target or the model corresponding to the first target;

[0437] The first instruction is an instruction from the model inference device requesting a model replacement. For example, the model inference device requests the AI ​​platform to replace it with a model that is suitable for the same task (such as a model that can solve the same task).

[0438] 10) Second indication information, used to indicate updating the first target or the model corresponding to the first target;

[0439] The second instruction information is used to instruct the model to be updated. For example, the model inference device requests the AI ​​platform or model training device to retrain the model in order to update the model.

[0440] Optionally, the model corresponding to the first target can be updated by combining the first data and the second instruction information, or the model corresponding to the first target can be updated based on the first data.

[0441] 11) Third instruction information, used to indicate fine-tuning of the first target or the model corresponding to the first target.

[0442] For example, the model inference device requests the AI ​​platform or model training device to fine-tune the model via a third instruction.

[0443] Optionally, by combining the first data and the third instruction information, the model corresponding to the first target can be fine-tuned, or the model corresponding to the first target can be fine-tuned based on the first data.

[0444] The network performance monitoring method provided in this application embodiment involves a fourth device receiving first performance monitoring feedback information returned by a first device, thereby realizing the network performance monitoring process after introducing an AI platform.

[0445] Optionally, the method further includes:

[0446] The fourth device performs a second operation, which includes at least one of the following:

[0447] Retrain the first target or the model corresponding to the first target;

[0448] Reselect the first target or the model corresponding to the first target;

[0449] Fine-tune the first target or the model corresponding to the first target;

[0450] Send the model, which has been retrained, reselected, or fine-tuned, to the first or fifth device.

[0451] It is understandable that after receiving performance supervision feedback information from the first or fifth device, the fourth device can retrain, reselect, or fine-tune the first target or the model corresponding to the first target. Furthermore, it can send the model obtained after retraining, reselection, or fine-tuning to the first or fifth device.

[0452] The network performance monitoring method provided in this application embodiment allows a fourth device to receive performance monitoring feedback information, retrain, reselect, or fine-tune the model, and send the model obtained after retraining, reselecting, or fine-tuning, which can optimize network performance.

[0453] Figure 8 The seventh flowchart illustrates the network performance monitoring method provided in this application embodiment. Figure 8 As shown, the network performance monitoring method includes steps 700 and 701.

[0454] Step 700: The second data network element receives a data request message from the first device or the fifth device. The data request message is used to request the acquisition of first data, which is performance monitoring related data.

[0455] It should be noted that the second data network element can be the control module (DPF-CP) in the data plane function, or other devices with data plane capability or data plane control capability.

[0456] In some embodiments, the data request message carries at least one of the following:

[0457] Description information of the first data;

[0458] Information about the first device;

[0459] Supported data transmission protocols and their corresponding protocol types;

[0460] The protocol type corresponding to the desired data transmission protocol;

[0461] Data transmission time, used to indicate the time when the first data was transmitted.

[0462] Step 701: The second data network element sends a data provision message to the first data network element, the data provision message being used to notify the first data network element to send the first data.

[0463] The first data network element can be a data module (DPF-DP) in a data plane function (DPF), or other devices with data plane functions or data plane-related capabilities, such as devices with the ability to process and forward data packets, manage and schedule traffic and / or process network protocols.

[0464] Understandably, after receiving a data request message from the first or fifth device, the second data network element interacts with the first data network element. The second data network element sends a data provision message to the first data network element, which notifies the first data network element to send the first data to the target consumer (i.e., the first or fifth device). The first data network element sends the first data to the target consumer based on the data plane, thereby the first or fifth device receives the first data from the first data network element based on the data plane.

[0465] Optionally, the second data network element is configured separately from the first data network element. After the first device sends a data request message to the second data network element, the second data network element interacts with the first data network element. The second data network element sends a data provision message to the first data network element, which notifies the first data network element to send the first data to the first device. The first data network element sends the first data to the first device based on the data plane, thereby the first device receives the first data from the first data network element based on the data plane.

[0466] Optionally, the second data network element and the first data network element are the same network element. The second data network element and the first data network element can be referred to as a second device, meaning that both functions are implemented through the same network element or device. Alternatively, the functions of the first and second data network elements can be combined, or the first and second data network elements can be considered two logical functional units within the second device. In the case of combined functions, the interaction information between the two will change from explicit external interaction to implicit internal interaction.

[0467] Optionally, the data providing message carries at least one of the following:

[0468] 1) The data protocol information to be used for data transmission selected by the second data network element;

[0469] This section specifies the exact protocol information for DPF-CP, such as a combination of QUIC, HTTP3, and protobuf protocols. DPF-DP uses this protocol to send data to the target consumer (model inference device or model training device).

[0470] 2) Data transmission time, used to indicate the transmission time of the first data;

[0471] 3) Description information of the first data, including data type, data identifier, data function description, data limitation information, etc.

[0472] 4) Information about the destination consumer, including at least one of the following: the destination consumer's identification information, address (IP address or other type of address), port number, and corresponding data plane tunnel endpoint identifier.

[0473] 5) The destination address for storage.

[0474] Optionally, the destination address for storage includes at least one of the following:

[0475] An identifier pointing to the storage space of the first data network element;

[0476] The address pointing to the storage space of the first data network element;

[0477] Data storage identifier;

[0478] The identification information of the first data.

[0479] In some scenarios, the first data network element does not store the first data and temporarily obtains the first data from the data source through the data plane. In this case, the destination address stored is the identification information of the first data.

[0480] The network performance monitoring method provided in this application embodiment involves a second data network element receiving a data request message from a first device or a fifth device and sending a data provision message to the first data network element, thereby realizing data interaction related to performance monitoring based on the data plane.

[0481] In some embodiments, the method further includes:

[0482] The second data network element receives data from the first data network element and provides a response message;

[0483] The data provision response message includes at least one of the following: an indication to accept the data provision message; an indication to reject the data provision message; or information about the first data network element.

[0484] In one implementation, the data provision response message includes: an indication of accepting the data provision message, and / or information about a first data network element.

[0485] Optionally, the information of the first data network element includes at least one of the following: the identification information of the first data network element, the address of the first data network element (IP address or other type of address), the port number of the first data network element, the data plane tunnel identification information corresponding to the first data network element, and the data plane tunnel endpoint identification information corresponding to the first data network element.

[0486] In one implementation, the data provision response message includes: an indication that the data provision message is rejected.

[0487] The first data network element sends the first data to the destination address indicated by the information in the data provision message, such as the data transmission time and the data protocol information to be used for data transmission. The first data may be data already stored by the first data network element, or data obtained from the data source and forwarded by the first data network element.

[0488] The network performance monitoring method provided in this application embodiment involves a second data network element receiving a data request message from a first device or a fifth device, sending a data provision message to a first data network element, and receiving a data provision response message from the first data network element, thereby realizing data interaction related to performance monitoring based on the data plane.

[0489] Figure 9 The eighth flowchart illustrating the network performance monitoring method provided in this application embodiment is as follows: Figure 9 As shown, the network performance monitoring method includes steps 800 and 801.

[0490] Step 800: The first data network element receives a data provision message from the second data network element. The data provision message is used to notify the first data network element to send first data, which is performance monitoring related data.

[0491] It should be noted that the first data network element is the data module DPF-DP in the data plane function, and the second data network element is the control module DPF-CP in the data plane function. DPF-DP receives data provision messages from DPF-CP, and the data provision messages are used to notify DPF-DP to send the first data.

[0492] Optionally, the data providing message carries at least one of the following:

[0493] 1) The data protocol information to be used for data transmission selected by the second data network element;

[0494] This section specifies the exact protocol information for DPF-CP, such as a combination of QUIC, HTTP3, and protobuf protocols. DPF-DP uses this protocol to send data to the destination consumer. Optionally, the destination consumer may include a model inference device or a model training device.

[0495] 2) Data transmission time, used to indicate the transmission time of the first data;

[0496] 3) Description information of the first data, including data type, data identifier, data function description, data limitation information, etc.

[0497] 4) Information about the destination consumer, including at least one of the following: the destination consumer's identification information, address (IP address or other type of address), port number, and corresponding data plane tunnel endpoint identifier.

[0498] In this context, the target consumer refers to the target consumer of the first data, that is, the device that needs to use the first data.

[0499] 5) The destination address for storage.

[0500] Optionally, the destination address for storage includes at least one of the following:

[0501] An identifier pointing to the storage space of the first data network element;

[0502] The address pointing to the storage space of the first data network element;

[0503] Data storage identifier;

[0504] The identification information of the first data.

[0505] In some scenarios, the first data network element does not store the first data and temporarily obtains the first data from the data source through the data plane. In this case, the destination address stored is the identification information of the first data.

[0506] Step 801: The first data network element sends a data provision response message to the second data network element. The data provision response message includes at least one of the following: an indication to accept the data provision message; an indication to reject the data provision message; and information about the first data network element.

[0507] The DPF-CP sends a data provision response message to the DPF-CP. The provision response message includes at least one of the following: an indication to accept the data provision message; an indication to reject the data provision message; or information about the first data network element.

[0508] The network performance monitoring method provided in this application embodiment involves a first data network element receiving a data provision message from a second data network element and sending a data provision response message to the second data network element, thereby realizing data interaction related to performance monitoring based on the data plane.

[0509] Optionally, the information of the first data network element includes at least one of the following: the identification information of the first data network element, the address of the first data network element (IP address or other type of address), the port number of the first data network element, the data plane tunnel identification information corresponding to the first data network element, and the data plane tunnel endpoint identification information corresponding to the first data network element.

[0510] In some embodiments, the method further includes:

[0511] The first data network element sends the first data to the target consumer.

[0512] Understandably, the first data network element sends the first data to the destination address indicated by the information in the data provision message, such as the data transmission time and the data protocol information to be used for data transmission. The first data may be data already stored by the first data network element, or data obtained from the data source and forwarded by the first data network element.

[0513] The network performance monitoring method provided in this application embodiment involves a first data network element receiving a data provision message from a second data network element, sending a data provision response message to the second data network element, and sending the first data to the target consumer, thereby realizing data interaction related to performance monitoring based on the data plane.

[0514] The network performance monitoring method provided in this application can be executed by a network performance monitoring device. This application uses the example of a network performance monitoring device executing the network performance monitoring method to illustrate the network performance monitoring device provided in this application.

[0515] This application provides a network performance monitoring device. As an example, the network performance monitoring device can be a communication device or a component within a communication device, such as a chip. The communication device can be a network-side device or a server, etc. Exemplarily, the network-side device can include, but is not limited to, the types of network-side devices 12 listed above; this application does not impose specific limitations.

[0516] The network performance monitoring device includes a receiving module, a transmitting module, and a processing module. These modules can be implemented in software or hardware. When implemented in hardware, the processing module can be implemented by a processor. For example, the processor can include general-purpose processors, special-purpose processors, etc., such as central processing units (CPUs), microprocessors, digital signal processors (DSPs), artificial intelligence (AI) processors, graphics processing units (GPUs), application-specific integrated circuits (ASICs), network processors (NPs), field-programmable gate arrays (FPGAs), or other programmable logic devices, gate circuits, transistors, discrete hardware components, etc. The receiving and transmitting modules can be implemented by a communication interface, which can include one or more of the following: transceivers, pins, circuits, buses, radio frequency units, etc.

[0517] For details, see Figure 10 When the network performance monitoring device is a network-side device or a component of a network-side device, the network performance monitoring device 900 includes a first receiving module 901, used to receive first data from a first data network element, the first data being performance monitoring related data; and a first processing module 902, used to determine first performance monitoring feedback information for a first target based on the first data, the first target being a model or an artificial intelligence (AI) task.

[0518] The network performance monitoring device provided in this application embodiment receives performance monitoring-related data from a first data network element and determines the performance information of a model or AI task based on the performance monitoring-related data, thereby realizing performance monitoring-related data interaction based on the data plane and network-related performance monitoring.

[0519] Optionally, the network performance monitoring device 900 further includes a first sending module, used for...

[0520] A data request message is sent to the second data network element, the data request message being used to request the acquisition of the first data.

[0521] Optionally, the data request message carries at least one of the following:

[0522] Description information of the first data;

[0523] Information about the first device;

[0524] Supported data transmission protocols and their corresponding protocol types;

[0525] The protocol type corresponding to the desired data transmission protocol;

[0526] Data transmission time, used to indicate the transmission time of the first data.

[0527] Optionally, the description information includes at least one of the following:

[0528] Data type;

[0529] Data identification;

[0530] Data function description;

[0531] Data limitation information.

[0532] Optionally, the information of the first device includes at least one of the following:

[0533] The identifier of the first device;

[0534] The IP address of the first device;

[0535] The port number of the first device;

[0536] The data plane tunnel endpoint identifier corresponding to the first device.

[0537] Optionally, the first receiving module 901 is further configured to:

[0538] Receive a data request response message from the second data network element, the data request response message including at least one of the following: an indication to accept the data request message; an indication to reject the data request message; and a data transmission method.

[0539] Optionally, the first receiving module 901 is further configured to:

[0540] The first device receives a performance monitoring request message from a fourth, fifth, or sixth device, the performance monitoring request message being used to request the first device to perform performance monitoring on the first target.

[0541] Optionally, the performance monitoring request message carries at least one of the following:

[0542] The identification information of the first target;

[0543] The type of the first target;

[0544] Performance metrics are used to indicate the methods for calculating performance values;

[0545] Threshold value, used to indicate the threshold for triggering the reporting of performance monitoring feedback information;

[0546] Conditions for reporting performance monitoring feedback information.

[0547] Optionally, based on the first data, the first device determines first performance supervision feedback information for the first target, including:

[0548] The first device determines the first performance monitoring feedback information of the first target based on the first data and the performance monitoring request message.

[0549] Optionally, the first sending module is further configured to:

[0550] The first device sends the first performance monitoring feedback information to the fourth, fifth, or sixth device.

[0551] Optionally, the first performance supervision feedback information includes at least one of the following:

[0552] Performance values;

[0553] The identification information of the first target;

[0554] Performance metrics;

[0555] Supervision range, used to indicate the supervision range corresponding to the performance value;

[0556] The first data;

[0557] The identification information of the first data;

[0558] Storage information of the first data;

[0559] Performance degradation indication information is used to indicate a performance degradation of the first target;

[0560] The first instruction information is used to instruct the replacement of the first target or the model corresponding to the first target;

[0561] The second instruction information is used to instruct the updating of the first target or the model corresponding to the first target;

[0562] The third instruction information is used to instruct fine-tuning of the first target or the model corresponding to the first target.

[0563] Optionally, the first device is a device with model inference capability, the fourth device is a device with model management capability, the fifth device is a device with model training capability, and the sixth device is a consumer device.

[0564] Optionally, the first data includes at least one of the following: inference input data; inference output data; and truth data.

[0565] The network performance monitoring device provided in this application embodiment can achieve... Figures 2 to 3 The various processes implemented in the method embodiments achieve the same technical effect, and will not be described again here to avoid repetition.

[0566] For details, see Figure 11 When the network performance monitoring device is a network-side device or a component within a network-side device, the network performance monitoring device 1000 includes a second receiving module 1001, used to receive first data from a first data network element, the first data being performance monitoring related data; and / or, to receive second data from a first device or a fourth device, the second data being first performance monitoring feedback information for a first target, the first target being a model or an artificial intelligence (AI) task; and a second processing module 1002, used to determine second performance monitoring feedback information for the first target based on the first data and / or the second data.

[0567] The network performance monitoring device provided in this application embodiment obtains performance monitoring-related data from the data plane and / or determines the performance information of the model or AI task based on the performance monitoring results of other devices, thereby realizing data interaction related to performance monitoring based on the data plane and network-related performance monitoring.

[0568] Optionally, the network performance monitoring device further includes a second sending module, used for:

[0569] A data request message is sent to the second data network element, the data request message being used to request the acquisition of the first data.

[0570] Optionally, the data request message carries at least one of the following:

[0571] Description information of the first data;

[0572] Information about the fifth device;

[0573] Supported data transmission protocols and their corresponding protocol types;

[0574] The protocol type corresponding to the desired data transmission protocol;

[0575] Data transmission time, used to indicate the time when the first data was transmitted.

[0576] Optionally, the description information includes at least one of the following:

[0577] Data type;

[0578] Data identification;

[0579] Data function description;

[0580] Data limitation information.

[0581] Optionally, the information of the fifth device includes at least one of the following:

[0582] The identification information of the fifth device;

[0583] The IP address of the fifth device;

[0584] The port number of the fifth device;

[0585] The data plane tunnel endpoint identifier corresponding to the fifth device.

[0586] Optionally, the second receiving module 1001 is further configured to:

[0587] Receive a data request response message from the second data network element, the data request response message including at least one of the following: an indication to accept the data request message; an indication to reject the data request message; and a data transmission method.

[0588] Optionally, the second receiving module 1001 is further configured to:

[0589] The fifth device receives a performance monitoring request message from the first device or the fourth device, the performance monitoring request message being used to request the fifth device to perform performance monitoring on the first target.

[0590] Optionally, the second sending module is further configured to:

[0591] A performance monitoring request message is sent to the first device or the fourth device, the performance monitoring request message being used to request the first device or the fourth device to perform performance monitoring on the first target.

[0592] Optionally, a performance monitoring request message is sent to the first device, including:

[0593] A performance monitoring request message is sent to the first device via a service-based architecture or through a service.

[0594] or,

[0595] The fourth device sends a performance monitoring request message to the first device.

[0596] Optionally, the second data is the first performance monitoring feedback information of the first target determined by the first device, or the second data is determined by the fourth device based on the performance monitoring feedback information of the first target fed back by at least one first device.

[0597] Optionally, the performance monitoring request message carries at least one of the following:

[0598] The identification information of the first target;

[0599] The type of the first target;

[0600] Performance metrics are used to indicate the methods for calculating performance values;

[0601] Threshold value, used to indicate the threshold for triggering the reporting of performance monitoring feedback information;

[0602] Conditions for reporting performance monitoring feedback information.

[0603] Optionally, the second processing module 1002 is further configured to:

[0604] Perform a first operation, the first operation including at least one of the following:

[0605] Send the second performance monitoring feedback information to the first device or the fourth device;

[0606] Retrain the first target or the model corresponding to the first target;

[0607] Reselect the first target or the model corresponding to the first target;

[0608] Fine-tune the first target or the model corresponding to the first target;

[0609] Send the model, which has been retrained, reselected, or fine-tuned, to the first or fourth device.

[0610] Optionally, the second performance supervision feedback information includes at least one of the following:

[0611] Performance values;

[0612] The identification information of the first target;

[0613] Performance metrics are used to indicate the methods for calculating performance values;

[0614] Supervision range, used to indicate the supervision range corresponding to the performance value;

[0615] Performance monitoring data used to calculate the performance values;

[0616] Identification information or storage information corresponding to the performance supervision-related data used to calculate the performance value;

[0617] Performance degradation indication information is used to indicate a performance degradation of the first target;

[0618] The fourth instruction information is used to instruct the replacement of the first target or the model corresponding to the first target;

[0619] The fifth instruction information is used to instruct the updating of the first target or the model corresponding to the first target;

[0620] The sixth instruction is used to instruct fine-tuning of the first target or the model corresponding to the first target.

[0621] Optionally, the first device is a device with model inference capability, the fourth device is a device with model management capability, and the fifth device is a device with model training capability.

[0622] Optionally, the performance supervision related data includes at least one of the following: inference input data, inference output data, and truth data.

[0623] The network performance monitoring device provided in this application embodiment can achieve... Figures 4 to 5 The various processes implemented in the method embodiments achieve the same technical effect, and will not be described again here to avoid repetition.

[0624] For details, see Figure 12 When the network performance monitoring device is a network-side device or a component within a network-side device, the network performance monitoring device 1100 includes:

[0625] The third receiving module 1101 is used to receive a performance supervision request message from the fifth device. The performance supervision request message is used to request performance supervision of a first target, which is a model or an artificial intelligence (AI) task.

[0626] The third processing module 1102 is used to determine at least one first device based on the identification information of the first target;

[0627] The third sending module 1103 is used to send a performance supervision request message to the at least one first device, the performance supervision request message being used to request the at least one first device to perform performance supervision on the first target;

[0628] or,

[0629] The third sending module 1103 is used to send a performance supervision request message to the first device or the fifth device, the performance supervision request message being used to request the first device or the fifth device to perform performance supervision on the first target.

[0630] The network performance monitoring device provided in this application embodiment receives a performance monitoring request message from a fifth device, determines at least one first device based on the identification information of the first target carried in the performance monitoring request message, and sends a performance monitoring request message to the at least one first device, thereby obtaining the performance monitoring results of at least one first device for processing, realizing network performance monitoring and improving the accuracy of network performance monitoring; by sending a performance monitoring request message to the first device or the fifth device, the network performance monitoring process after introducing an AI platform is realized.

[0631] Optionally, the third receiving module 1101 is further configured to: receive performance monitoring feedback information of the first target from the at least one first device;

[0632] The third processing module 1102 is further configured to: process the performance supervision feedback information of the first target from the at least one first device to obtain the first performance supervision feedback information of the first target;

[0633] The third sending module 1103 is further configured to: send the first performance monitoring feedback information of the first target to the fifth device.

[0634] Optionally, the third receiving module 1101 is further configured to:

[0635] The device receives second performance monitoring feedback information from a fifth device. The second performance monitoring feedback information is determined by the fifth device based on the first performance monitoring feedback information and / or first data from a first data network element. The first data is performance monitoring related data.

[0636] Optionally, the second performance supervision feedback information includes at least one of the following:

[0637] Performance values;

[0638] The identification information of the first target;

[0639] Performance metrics, used to indicate the method of calculating the performance values;

[0640] Supervision range, used to indicate the supervision range corresponding to the performance value;

[0641] Performance monitoring data used to calculate the performance values;

[0642] Identification information or storage information corresponding to the performance supervision-related data used to calculate the performance value;

[0643] Performance degradation indication information is used to indicate a performance degradation of the first target;

[0644] The fourth instruction is used to instruct the replacement of the first target or the model corresponding to the first target;

[0645] The fifth instruction is used to instruct the updating of the first target or the model corresponding to the first target;

[0646] The sixth instruction is used to instruct fine-tuning of the first target or the model corresponding to the first target.

[0647] Optionally, the third receiving module 1101 is further configured to:

[0648] The system receives first performance monitoring feedback information from a first device, which is determined by the first device based on first data from a first data network element.

[0649] Optionally, the third processing module 1102 is further configured to: perform a second operation, the second operation including at least one of the following:

[0650] Retrain the first target or the model corresponding to the first target;

[0651] Reselect the first target or the model corresponding to the first target;

[0652] Fine-tune the first target or the model corresponding to the first target;

[0653] Send the model, which has been retrained, reselected, or fine-tuned, to the first or fifth device.

[0654] Optionally, the performance monitoring request message carries at least one of the following:

[0655] The identification information of the first target;

[0656] The type of the first target;

[0657] Performance metrics, used to indicate the calculation method of performance metrics;

[0658] Threshold value, used to indicate the threshold for triggering the reporting of performance monitoring feedback information;

[0659] Conditions for reporting performance monitoring feedback information.

[0660] The network performance monitoring device provided in this application embodiment can achieve... Figures 6 to 7 The various processes implemented in the method embodiments achieve the same technical effect, and will not be described again here to avoid repetition.

[0661] For details, see Figure 13When the network performance monitoring device is a network-side device or a component within a network-side device, the network performance monitoring device 1200 includes: a fourth receiving module 1201, used to receive a data request message from a first device or a fifth device, the data request message being used to request the acquisition of first data, the first data being performance monitoring related data; and a fourth sending module 1202, used to send a data providing message to a first data network element, the data providing message being used to notify the first data network element to send the first data.

[0662] The network performance monitoring device provided in this application embodiment receives a data request message from a first device or a fifth device and sends a data provision message to a first data network element, thereby realizing data interaction related to performance monitoring based on the data plane.

[0663] Optionally, the data request message carries at least one of the following:

[0664] Description information of the first data;

[0665] Information about the first device;

[0666] Supported data transmission protocols and their corresponding protocol types;

[0667] The protocol type corresponding to the desired data transmission protocol;

[0668] Data transmission time, used to indicate the time when the first data was transmitted.

[0669] Optionally, the data providing message carries at least one of the following:

[0670] The data protocol information to be used for data transmission selected by the second data network element;

[0671] Data transmission time is used to indicate the transmission time of the first data;

[0672] Description information of the first data;

[0673] Information about the target consumer;

[0674] The destination address for storage.

[0675] Optionally, the destination address for storage includes at least one of the following:

[0676] An identifier pointing to the storage space of the first data network element;

[0677] The address pointing to the storage space of the first data network element;

[0678] Data storage identifier;

[0679] The identification information of the first data.

[0680] Optionally, the fourth receiving module 1201 is further configured to:

[0681] Receive data from the first data network element and provide a response message;

[0682] The data provision response message includes at least one of the following: an indication to accept the data provision message; an indication to reject the data provision message; or information about the first data network element.

[0683] The network performance monitoring device provided in this application embodiment can achieve... Figure 8 The various processes implemented in the method embodiments achieve the same technical effect, and will not be described again here to avoid repetition.

[0684] For details, see Figure 14 When the network performance monitoring device is a network-side device or a component within a network-side device, the network performance monitoring device 1300 includes: a fifth receiving module 1301, configured to receive a data provision message from a second data network element, the data provision message being used to notify the first data network element to send first data, the first data being performance monitoring related data; and a fifth sending module 1302, configured to send a data provision response message to the second data network element, the data provision response message including at least one of the following: an indication to accept the data provision message; an indication to reject the data provision message; and information about the first data network element.

[0685] The network performance monitoring device provided in this application embodiment realizes data interaction related to performance monitoring based on the data plane by receiving a data provision message from a second data network element and sending a data provision response message to the second data network element.

[0686] Optionally, the data providing message carries at least one of the following:

[0687] The data protocol information to be used for data transmission selected by the second data network element;

[0688] Data transmission time is used to indicate the transmission time of the first data;

[0689] Description information of the first data;

[0690] Information about the target consumer;

[0691] The destination address for storage.

[0692] Optionally, the fifth sending module 1302 is further configured to:

[0693] The first data is sent to the target consumer.

[0694] The network performance monitoring device provided in this application embodiment can achieve... Figure 9 The various processes implemented in the method embodiments achieve the same technical effect, and will not be described again here to avoid repetition.

[0695] like Figure 15 As shown, this application embodiment also provides a communication device 1400, including a processor 1401 and a memory 1402. The memory 1402 stores a program or instructions that can run on the processor 1401. When the communication device 1400 is a network-side device, when the program or instructions are executed by the processor 1401, they implement the various steps of the above-described network performance monitoring method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0696] This application embodiment also provides a network-side device, including a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement, for example... Figures 2-9 The steps of the method embodiment shown are illustrated. This network-side device embodiment corresponds to the above-described network-side device method embodiment. All implementation processes and methods of the above-described method embodiments can be applied to this network-side device embodiment and can achieve the same technical effect.

[0697] Specifically, embodiments of this application also provide a network-side device. For example... Figure 16 As shown, the network-side device 1500 includes: a processor 1501, a network interface 1502, and a memory 1503. This network-side device can be... Figures 10-14 The network performance monitoring device shown is described. The network interface 1502 is, for example, a common public radio interface (CPRI).

[0698] Among them, network interface 1502 is used to receive first data from the first data network element, and the first data is performance monitoring related data;

[0699] The processor 1501 is used to determine first performance supervision feedback information for a first target, which is a model or an artificial intelligence (AI) task, based on the first data.

[0700] Optionally, the network interface 1502 is used to receive first data from a first data network element, the first data being performance supervision related data; and / or, to receive second data from a first device or a fourth device, the second data being first performance supervision feedback information of a first target, the first target being a model or an artificial intelligence (AI) task.

[0701] The processor 1501 is used to determine second performance supervision feedback information for the first target based on the first data and / or the second data.

[0702] Optional,

[0703] Network interface 1502 is used to receive a performance supervision request message from a fifth device, the performance supervision request message being used to request performance supervision of a first target, the first target being a model or an artificial intelligence (AI) task;

[0704] The processor 1501 is configured to determine at least one first device based on the identification information of the first target;

[0705] Network interface 1502 is used to send a performance monitoring request message to the at least one first device, the performance monitoring request message being used to request the at least one first device to perform performance monitoring on the first target;

[0706] or,

[0707] Network interface 1502 is used to send a performance monitoring request message to a first device or a fifth device, the performance monitoring request message being used to request the first device or the fifth device to perform performance monitoring on the first target.

[0708] Optionally, network interface 1502 is used for:

[0709] Receive a data request message from a first device or a fifth device, the data request message being used to request the acquisition of first data, the first data being performance monitoring related data;

[0710] A data provision message is sent to the first data network element, the data provision message being used to notify the first data network element to send the first data.

[0711] Optionally, network interface 1502 is used for:

[0712] Receive a data provision message from a second data network element, the data provision message being used to notify the first data network element to send first data, the first data being performance monitoring related data;

[0713] A data provision response message is sent to the second data network element, the data provision response message including at least one of the following: an indication to accept the data provision message; an indication to reject the data provision message; and information about the first data network element.

[0714] Furthermore, the network-side device 1500 in this embodiment of the application also includes: a program or instructions stored in a memory 1503 and executable on a processor 1501, wherein the processor 1501 calls the program or instructions in the memory 1503 to execute. Figures 10-14 The methods executed by each module shown achieve the same technical effect, and to avoid repetition, they will not be described in detail here.

[0715] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described network performance monitoring method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0716] The processor mentioned above is either the processor in the terminal described in the above embodiments or the processor in the network-side device. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk. In some examples, the readable storage medium may be a non-transient readable storage medium.

[0717] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described network performance monitoring method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0718] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0719] This application also provides a computer program / program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described network performance monitoring method embodiments, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0720] This application also provides a wireless communication system, including a network-side device, which can be used to perform the steps of the network performance monitoring method described above.

[0721] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0722] From the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of computer software products plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.), and the computer software product includes several instructions to cause the terminal or network-side device to execute the methods described in the various embodiments of this application.

[0723] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other implementations under the guidance of this application without departing from the spirit and scope of the claims. All of these implementations are within the protection scope of this application.

Claims

1. A network performance monitoring method, characterized in that, include: The first device receives first data from the first data network element, and the first data is performance monitoring related data. Based on the first data, the first device determines the first performance supervision feedback information of the first target, which is a model or an artificial intelligence (AI) task.

2. The method according to claim 1, characterized in that, Before the first device receives the first data from the first data network element, the method further includes: The first device sends a data request message to the second data network element, the data request message being used to request the acquisition of the first data.

3. The method according to claim 2, characterized in that, The data request message carries at least one of the following: Description information of the first data; Information about the first device; Supported data transmission protocols and their corresponding protocol types; The protocol type corresponding to the desired data transmission protocol; Data transmission time, used to indicate the transmission time of the first data.

4. The method according to claim 3, characterized in that, The descriptive information includes at least one of the following: Data type; Data identification; Data function description; Data limitation information.

5. The method according to claim 3, characterized in that, The information of the first device includes at least one of the following: The identifier of the first device; The IP address of the first device; The port number of the first device; The data plane tunnel endpoint identifier corresponding to the first device.

6. The method according to any one of claims 2-5, characterized in that, The method further includes: the first device receiving a data request response message from the second data network element, the data request response message including at least one of the following: an indication to accept the data request message; an indication to reject the data request message; and a data transmission method.

7. The method according to any one of claims 1-6, characterized in that, Before the first device receives the first data from the first data network element, the method further includes: The first device receives a performance monitoring request message from a fourth, fifth, or sixth device, the performance monitoring request message being used to request the first device to perform performance monitoring on the first target.

8. The method according to claim 7, characterized in that, The performance monitoring request message carries at least one of the following: The identification information of the first target; The type of the first target; Performance metrics are used to indicate the methods for calculating performance values; Threshold value, used to indicate the threshold for triggering the reporting of performance monitoring feedback information; Conditions for reporting performance monitoring feedback information.

9. The method according to claim 8, characterized in that, Based on the first data, the first device determines performance monitoring feedback information for the first target, including: The first device determines the performance monitoring feedback information of the first target based on the first data and the performance monitoring request message.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: The first device sends the first performance monitoring feedback information to the fourth, fifth, or sixth device.

11. The method according to any one of claims 1-10, characterized in that, The first performance supervision feedback information includes at least one of the following: Performance values; The identification information of the first target; Performance metrics; Supervision range, used to indicate the supervision range corresponding to the performance value; The first data; The identification information of the first data; Storage information of the first data; Performance degradation indication information is used to indicate a performance degradation of the first target; The first instruction information is used to instruct the replacement of the first target or the model corresponding to the first target; The second instruction information is used to instruct the updating of the first target or the model corresponding to the first target; The third instruction information is used to instruct fine-tuning of the first target or the model corresponding to the first target.

12. The method according to claim 7, 10 or 11, characterized in that, The first device is a device with model inference capability, the fourth device is a device with model management capability, the fifth device is a device with model training capability, and the sixth device is a consumer device.

13. The method according to any one of claims 1-12, characterized in that, The first data includes at least one of the following: inference input data; inference output data; truth data.

14. A network performance monitoring method, characterized in that, include: The fifth device receives first data from the first data network element, and the first data is performance monitoring related data; And / or, the fifth device receives second data from the first device or the fourth device, the second data being first performance supervision feedback information for the first target, the first target being a model or an artificial intelligence (AI) task; The fifth device determines the second performance monitoring feedback information of the first target based on the first data and / or the second data.

15. The method according to claim 14, characterized in that, Before the fifth device receives the first data from the first data network element, the method further includes: The fifth device sends a data request message to the second data network element, the data request message being used to request the acquisition of the first data.

16. The method according to claim 15, characterized in that, The data request message carries at least one of the following: Description information of the first data; Information about the fifth device; Supported data transmission protocols and their corresponding protocol types; The protocol type corresponding to the desired data transmission protocol; Data transmission time, used to indicate the time when the first data was transmitted.

17. The method according to claim 16, characterized in that, The descriptive information includes at least one of the following: Data type; Data identification; Data function description; Data limitation information.

18. The method according to claim 16, characterized in that, The information of the fifth device includes at least one of the following: The identification information of the fifth device; The IP address of the fifth device; The port number of the fifth device; The data plane tunnel endpoint identifier corresponding to the fifth device.

19. The method according to any one of claims 15-18, characterized in that, The method further includes: The fifth device receives a data request response message from the second data network element. The data request response message includes at least one of the following: an indication to accept the data request message; an indication to reject the data request message; and a data transmission method.

20. The method according to any one of claims 14-19, characterized in that, Before the fifth device receives the first data from the first data network element, the method further includes: The fifth device receives a performance monitoring request message from the first device or the fourth device, the performance monitoring request message being used to request the fifth device to perform performance monitoring on the first target.

21. The method according to any one of claims 14-19, characterized in that, Before the fifth device receives the second data from the first or fourth device, the method further includes: The fifth device sends a performance monitoring request message to the first device or the fourth device, the performance monitoring request message being used to request the first device or the fourth device to perform performance monitoring on the first target.

22. The method according to claim 21, characterized in that, The fifth device sends a performance monitoring request message to the first device, including: The fifth device sends a performance monitoring request message to the first device through a service-based architecture or through a service. or, The fifth device sends a performance monitoring request message to the first device through the fourth device.

23. The method according to claim 21 or 22, characterized in that, The second data is the first performance monitoring feedback information of the first target determined by the first device, or the second data is determined by the fourth device based on the performance monitoring feedback information of the first target fed back by at least one first device.

24. The method according to any one of claims 20-23, characterized in that, The performance monitoring request message carries at least one of the following: The identification information of the first target; The type of the first target; Performance metrics are used to indicate the methods for calculating performance values; Threshold value, used to indicate the threshold for triggering the reporting of performance monitoring feedback information; Conditions for reporting performance monitoring feedback information.

25. The method according to any one of claims 14-24, characterized in that, The method further includes: The fifth device performs a first operation, the first operation including at least one of the following: Send the second performance monitoring feedback information to the first device or the fourth device; Retrain the first target or the model corresponding to the first target; Reselect the first target or the model corresponding to the first target; Fine-tune the first target or the model corresponding to the first target; Send the model, which has been retrained, reselected, or fine-tuned, to the first or fourth device.

26. The method according to any one of claims 14-25, characterized in that, The second performance supervision feedback information includes at least one of the following: Performance values; The identification information of the first target; Performance metrics are used to indicate the methods for calculating performance values; Supervision range, used to indicate the supervision range corresponding to the performance value; Performance monitoring data used to calculate the performance values; Identification information or storage information corresponding to the performance supervision-related data used to calculate the performance value; Performance degradation indication information is used to indicate a performance degradation of the first target; The fourth instruction information is used to instruct the replacement of the first target or the model corresponding to the first target; The fifth instruction information is used to instruct the updating of the first target or the model corresponding to the first target; The sixth instruction is used to instruct fine-tuning of the first target or the model corresponding to the first target.

27. The method according to any one of claims 14-26, characterized in that, The first device is a device with model reasoning capability, the fourth device is a device with model management capability, and the fifth device is a device with model training capability.

28. The method according to claim 14 or 26, characterized in that, The performance supervision-related data includes at least one of the following: inference input data, inference output data, and truth data.

29. A network performance monitoring method, characterized in that, include: The fourth device receives a performance supervision request message from the fifth device. The performance supervision request message is used to request performance supervision of a first target, which is a model or an artificial intelligence (AI) task. The fourth device determines at least one first device based on the identification information of the first target; The fourth device sends a performance supervision request message to the at least one first device, the performance supervision request message being used to request the at least one first device to perform performance supervision on the first target; or, The fourth device sends a performance monitoring request message to the first device or the fifth device, the performance monitoring request message being used to request the first device or the fifth device to perform performance monitoring on the first target.

30. The method according to claim 29, characterized in that, The method further includes: The fourth device receives performance monitoring feedback information of a first target from the at least one first device; The fourth device processes the performance monitoring feedback information of the first target from the at least one first device to determine the first performance monitoring feedback information of the first target; The fourth device sends the first performance monitoring feedback information of the first target to the fifth device.

31. The method according to claim 30, characterized in that, The method further includes: The fourth device receives second performance monitoring feedback information from the fifth device. The second performance monitoring feedback information is determined by the fifth device based on the first performance monitoring feedback information and / or first data from the first data network element. The first data is performance monitoring related data.

32. The method according to claim 31, characterized in that, The second performance supervision feedback information includes at least one of the following: Performance values; The identification information of the first target; Performance metrics, used to indicate the method of calculating the performance values; Supervision range, used to indicate the supervision range corresponding to the performance value; Performance monitoring data used to calculate the performance values; Identification information or storage information corresponding to the performance supervision-related data used to calculate the performance value; Performance degradation indication information is used to indicate a performance degradation of the first target; The fourth instruction is used to instruct the replacement of the first target or the model corresponding to the first target; The fifth instruction is used to instruct the updating of the first target or the model corresponding to the first target; The sixth instruction is used to instruct fine-tuning of the first target or the model corresponding to the first target.

33. The method according to claim 29, characterized in that, The method further includes: The fourth device receives first performance monitoring feedback information from the first device, which is determined by the first device based on first data from the first data network element.

34. The method according to any one of claims 31-33, characterized in that, The method further includes: The fourth device performs a second operation, which includes at least one of the following: Retrain the first target or the model corresponding to the first target; Reselect the first target or the model corresponding to the first target; Fine-tune the first target or the model corresponding to the first target; Send the model, which has been retrained, reselected, or fine-tuned, to the first or fifth device.

35. The method according to any one of claims 29-34, characterized in that, The performance monitoring request message carries at least one of the following: The identification information of the first target; The type of the first target; Performance metrics, used to indicate the calculation method of performance metrics; Threshold value, used to indicate the threshold for triggering the reporting of performance monitoring feedback information; Conditions for reporting performance monitoring feedback information.

36. A network performance monitoring method, characterized in that, include: The second data network element receives a data request message from the first device or the fifth device. The data request message is used to request the acquisition of first data, which is performance monitoring related data. The second data network element sends a data provision message to the first data network element, the data provision message being used to notify the first data network element to send the first data.

37. The method according to claim 36, characterized in that, The data request message carries at least one of the following: Description information of the first data; Information about the first device; Supported data transmission protocols and their corresponding protocol types; The protocol type corresponding to the desired data transmission protocol; Data transmission time, used to indicate the time when the first data was transmitted.

38. The method according to claim 36, characterized in that, The data-providing message carries at least one of the following: The data protocol information to be used for data transmission selected by the second data network element; Data transmission time is used to indicate the transmission time of the first data; Description information of the first data; Information about the target consumer; The destination address for storage.

39. The method according to claim 38, characterized in that, The destination address for storage includes at least one of the following: An identifier pointing to the storage space of the first data network element; The address pointing to the storage space of the first data network element; Data storage identifier; The identification information of the first data.

40. The method according to any one of claims 36-39, characterized in that, The method further includes: The second data network element receives data from the first data network element and provides a response message; The data provision response message includes at least one of the following: an indication to accept the data provision message; an indication to reject the data provision message; or information about the first data network element.

41. A network performance monitoring method, characterized in that, include: The first data network element receives a data provision message from the second data network element. The data provision message is used to notify the first data network element to send first data, which is performance monitoring related data. The first data network element sends a data provision response message to the second data network element. The data provision response message includes at least one of the following: an indication to accept the data provision message; an indication to reject the data provision message; and information about the first data network element.

42. The method according to claim 41, characterized in that, The data-providing message carries at least one of the following: The data protocol information to be used for data transmission selected by the second data network element; Data transmission time is used to indicate the transmission time of the first data; Description information of the first data; Information about the target consumer; The destination address for storage.

43. The method according to any one of claims 41-42, characterized in that, The method further includes: The first data network element sends the first data to the target consumer.

44. A network performance monitoring device, characterized in that, include: The first receiving module is used to receive first data from the first data network element, wherein the first data is performance monitoring related data. The first processing module is used to determine first performance supervision feedback information for a first target based on the first data, wherein the first target is a model or an artificial intelligence (AI) task.

45. A network performance monitoring device, characterized in that, include: The second receiving module is used to receive first data from the first data network element, wherein the first data is performance monitoring related data. And / or, receive second data from a first device or a fourth device, the second data being first performance supervision feedback information for a first target, the first target being a model or an artificial intelligence (AI) task; The second processing module is used to determine the second performance supervision feedback information of the first target based on the first data and / or the second data.

46. ​​A network performance monitoring device, characterized in that, include: The third receiving module is used to receive a performance supervision request message from the fifth device. The performance supervision request message is used to request performance supervision of a first target, which is a model or an artificial intelligence (AI) task. The third processing module is used to determine at least one first device based on the identification information of the first target; The third sending module is used to send a performance supervision request message to the at least one first device, the performance supervision request message being used to request the at least one first device to perform performance supervision on the first target; or, The third sending module is used to send a performance supervision request message to the first device or the fifth device, the performance supervision request message being used to request the first device or the fifth device to perform performance supervision on the first target.

47. A network performance monitoring device, characterized in that, include: The fourth receiving module is used to receive a data request message from the first device or the fifth device, wherein the data request message is used to request the acquisition of first data, and the first data is performance monitoring related data; The fourth sending module is used to send a data provision message to the first data network element, the data provision message being used to notify the first data network element to send the first data.

48. A network performance monitoring device, characterized in that, include: The fifth receiving module is used to receive a data providing message from the second data network element. The data providing message is used to notify the first data network element to send first data, which is performance monitoring related data. The fifth sending module is used to send a data provision response message to the second data network element. The data provision response message includes at least one of the following: an indication to accept the data provision message; an indication to reject the data provision message; and information about the first data network element.

49. A network-side device, characterized in that, The method includes a processor and a memory, the memory storing a program or instructions executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the network performance monitoring method as described in any one of claims 1 to 13, or implement the steps of the network performance monitoring method as described in any one of claims 14 to 28, or implement the steps of the network performance monitoring method as described in any one of claims 29 to 35, or implement the steps of the network performance monitoring method as described in any one of claims 36 to 40, or implement the steps of the network performance monitoring method as described in any one of claims 41 to 43.

50. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the network performance monitoring method as described in any one of claims 1 to 13, or the steps of the network performance monitoring method as described in any one of claims 14 to 28, or the steps of the network performance monitoring method as described in any one of claims 29 to 35, or the steps of the network performance monitoring method as described in any one of claims 36 to 40, or the steps of the network performance monitoring method as described in any one of claims 41 to 43.

Images