Communication method and communication device

Abstract

The invention provides a communication method and a communication device, relates to the field of communication, and can provide support for application of reinforcement learning in a communication system. The method comprises the steps that first information is determined, the first information is used for training a first model, and the training method of the first model is reinforcement learning; and sending first information to first equipment, wherein the first equipment is equipment for training the first model. In the embodiment of the invention, the user of the model training sends the information related to the model training based on the reinforcement learning to the provider of the model training, so that the support can be provided for the model training through the reinforcement learning method. For example, the first information can comprise related information of the environment of model interaction in the reinforcement learning process, so that the environment of reinforcement learning model training can be determined. For example, the first information can comprise relevant information of the model behaviors in the reinforcement learning process, so that the behaviors of the model in the training process can be determined.

Description

Technical Field

[0001] This application relates to the field of communications, and more specifically, to a communication method and a communication device. Background Technology

[0002] Reinforcement learning (RL) is an important branch of machine learning. Due to its numerous advantages, RL has been widely applied in fields such as games, automatic control, robot control, finance, and healthcare. Therefore, how to apply RL to communication systems is a problem that needs to be solved. Summary of the Invention

[0003] This application provides a communication method and a communication device that can support the application of reinforcement learning in communication systems, such as the training of reinforcement learning models.

[0004] In a first aspect, embodiments of this application provide a communication method that can be applied to a second device, which may be a physical device, a virtualization functional module, a software function, a circuit, a chip (such as a system on-chip (SoC) chip or a system-in-package (SIP) chip), etc.

[0005] For example, the second device is the user of the model training.

[0006] For example, the second device can be a cross-domain management function unit, also known as a network management system (NMS).

[0007] The method includes: determining first information, the first information being used to train a first model, the training method of the first model being reinforcement learning; and sending the first information to a first device, the first device being the device used to train the first model.

[0008] In this embodiment of the application, the user of model training sends information related to reinforcement learning-based model training to the model training provider, which helps to support model training using reinforcement learning methods. For example, the first information may include information about the environment in which the model interacts during reinforcement learning, thereby helping to determine the environment for reinforcement learning model training. Alternatively, the first information may include information about the model's behavior during reinforcement learning, thereby helping to determine the model's behavior during training.

[0009] In some embodiments, during model training, the first model interacts with a first environment, and the first information includes one or more of the following: a first metric list, the state of the first environment being determined by one or more metrics in the first metric list; a first parameter list, the behavior of the first model being determined by one or more parameters in the first parameter list; a first training environment, the training environment of the first model being the first training environment, or the training environment of the first model being one of the first training environments; the function of the first model; the state information of the first environment; the behavior information of the first model; or the performance metrics of the first model.

[0010] In traditional machine learning, training a model using training data from a single class allows for the acquisition of a one-to-one mapping relationship. Taking supervised learning as an example, training data from a single class can refer to training data for a single feature; by training the model, the mapping relationship between that feature and the label can be obtained.

[0011] Unlike traditional machine learning, reinforcement learning involves the interaction between an agent and its environment, where changes in the environment affect subsequent actions. Traditional machine learning training data often fails to represent the environment involved in reinforcement learning. In other words, the training methods of traditional machine learning models are not applicable to training reinforcement learning-based models.

[0012] The embodiments of this application, through information such as the first index list and the function of the first model, can provide the model training provider with a basis for determining the above-mentioned environment, thereby helping to solve the problem that traditional training data cannot be applied to the training of reinforcement learning-based models.

[0013] In some embodiments, the first information includes the first indicator list, and determining the first information includes: determining the first information based on model training requirement information, wherein the model training requirement information includes a second indicator list, and the indicators in the second indicator list are indicators expected to be used to determine the first environment; wherein the first indicator list is the second indicator list.

[0014] Determining the first indicator list using the above method and training the model based on the first indicator list helps the training results of the first model meet the requirements of the party proposing the training needs, that is, meet the user's needs.

[0015] In some embodiments, the capability information of the first device includes a third indicator list, which includes indicators that the first device supports using when determining the interaction environment of the model. Determining the first information based on the model training requirement information includes: determining the first indicator list based on the model training requirement information and the capability information of the first device, wherein: if the third indicator list includes the second indicator list, then the first indicator list is the second indicator list; if the third indicator list includes some indicators in the second indicator list, then the first indicator list includes indicators common to both the second and third indicator lists.

[0016] The method described above for determining the first list of indicators can avoid situations where the first device does not support the configured indicators, thereby helping to improve the feasibility of model training.

[0017] In some embodiments, the first information includes the functionality of the first model, which is determined based on the model training requirements. This approach is simple to implement.

[0018] In some embodiments, the first information includes the first parameter list, and the capability information of the first device includes the correspondence between indicators in the third indicator list and parameters in the second parameter list. The third indicator list includes indicators supported by the first device when determining the interaction environment of the model, and the second parameter list includes parameters supported by the first device when determining the behavior of the model. Determining the first information includes: the first parameter list is determined based on the indicators in the first indicator list and the correspondence, wherein the first parameter list includes parameters that have the correspondence with the indicators in the first indicator list.

[0019] The correspondence between metrics and parameters can be understood as follows: when training the first model in an environment defined by a metric, it can support the behavior defined by that parameter; or, in other words, changes in the parameter will affect the state of the metric. This is because, when training the first model in environments with different defined metrics, the behaviors supported by those parameters may be the same or different. Therefore, determining the first parameter list based on this correspondence helps improve the rationality of the determined behaviors.

[0020] In addition, different devices or different device manufacturers may support different correspondences as described above. Therefore, determining the first parameter list based on the above correspondences can configure the first parameter list supported by different devices, which helps to improve the compatibility of the model training configuration information with the training device.

[0021] In some embodiments, the method further includes: obtaining capability information of the first device, the capability information of the first device being related to reinforcement learning, wherein the capability information of the first device includes one or more of the following: a first function, wherein the first device supports training a model to implement the first function; a second environment, an environment supported by the first device, the second environment being used for model training; a third metric list, the third metric list including metrics supported by the first device when determining the interaction environment of the model; a second parameter list, the second parameter list including parameters supported by the first device when determining the behavior of the model; or a correspondence between metrics in the third metric list and parameters in the second parameter list.

[0022] For example, obtaining the capability information of the first device can also be replaced by receiving the capability information of the first device or subscribing to the capability information of the first device.

[0023] In this embodiment of the application, by reporting the capability information of the model training provider (such as capability information related to reinforcement learning model training), it helps the model training user to determine the model training configuration information that matches the capability information of the model training provider, thereby helping to improve the feasibility of reinforcement learning model training.

[0024] In some embodiments, sending the first information to the first device includes: sending a first request to the first device, the first request being used to request training of the first model; wherein the first request includes a first attribute, the first attribute being used to indicate the first information.

[0025] By including initial information in the model training request, the degree of modification required to the relevant technologies can be reduced. Furthermore, this approach is relatively easy to implement.

[0026] In some embodiments, the method further includes: receiving a model training report, the model training report including a second attribute and / or a third attribute, the second attribute indicating the training environment of the first model, and the third attribute indicating the performance metrics of the first model.

[0027] By including the model's training environment and / or the performance metrics of the first model in the model training report, the second device can better understand the model training process and support its model management. For example, based on the performance metrics of the first model, management strategies for the model inference process can be determined, such as identifying acceptable performance metrics during inference.

[0028] In some embodiments, the method further includes sending a first threshold to a first device, the first threshold being used to indicate an acceptable performance metric of the first model during model inference.

[0029] An acceptable performance metric can refer to, for example, the minimum allowed value of a performance metric. Taking the performance metric as a reward value as an example, an acceptable performance metric can refer to the minimum allowed reward value during model inference based on the first model.

[0030] Setting the first threshold as described above helps to improve the reliability of model inference.

[0031] For example, since the reward values ​​in the last few rounds of model training are close to the reward values ​​that can be obtained in the model inference process, the first threshold can be determined based on the reward values ​​in the last few rounds of model training, which helps to improve the accuracy of the first threshold.

[0032] For example, the first threshold can be determined based on the performance metrics of the first model during model training.

[0033] In some embodiments, the method further includes: sending second information to a first device, the second information being used to instruct the execution of model inference based on the first model; and receiving a model inference report, the model inference report including performance metrics of the first model during the model inference process.

[0034] Reporting performance metrics during the model inference process helps users understand the model's inference status and the accuracy of the inference results, and provides support for the management and monitoring of the model inference process.

[0035] Secondly, embodiments of this application provide a communication method that can be applied to a first device, which may be a physical device, a virtualization functional module, a software function, a circuit, a chip (such as a system on-chip (SoC) chip or a system-in-package (SIP) chip), etc.

[0036] For example, the first device is a provider of model training, or in other words, model training is deployed on the first device.

[0037] For example, the first device is both a provider of model training and a provider of model inference, or both model training and model inference are deployed in the first device.

[0038] For example, the first device can be a domain management function unit, or an element management system (EMS).

[0039] The method includes: receiving first information, the first information being used to train a first model, the training method of the first model being reinforcement learning; and training the first model based on the first information.

[0040] In this embodiment of the application, the user of model training sends information related to reinforcement learning-based model training to the model training provider, which helps to support model training using reinforcement learning methods. For example, the first information may include information about the environment in which the model interacts during reinforcement learning, thereby helping to determine the environment for reinforcement learning model training. Alternatively, the first information may include information about the model's behavior during reinforcement learning, thereby helping to determine the model's behavior during training.

[0041] In some embodiments, during model training, the first model interacts with a first environment, and the first information includes one or more of the following: a first metric list, the state of the first environment being determined by one or more metrics in the first metric list; a first parameter list, the behavior of the first model being determined by one or more parameters in the first parameter list; a first training environment, the training environment of the first model being the first training environment, or the training environment of the first model being one of the first training environments; the function of the first model; the state information of the first environment; the behavior information of the first model; or the performance metrics of the first model.

[0042] In traditional machine learning, training a model using training data from a single class allows for the acquisition of a one-to-one mapping relationship. Taking supervised learning as an example, training data from a single class can refer to training data for a single feature; by training the model, the mapping relationship between that feature and the label can be obtained.

[0043] Unlike traditional machine learning, reinforcement learning involves the interaction between an agent and its environment, where changes in the environment affect subsequent actions. Traditional machine learning training data often fails to represent the environment involved in reinforcement learning. In other words, the training methods of traditional machine learning models are not applicable to training reinforcement learning-based models.

[0044] The embodiments of this application, through information such as the first index list and the function of the first model, can provide the model training provider with a basis for determining the above-mentioned environment, thereby helping to solve the problem that traditional training data cannot be applied to the training of reinforcement learning-based models.

[0045] In some embodiments, the first information includes the functionality of the first model, and training the first model based on the first information includes: determining third information based on the functionality of the first model; and training the first model based on the third information, wherein the third information includes one or more of the following: one or more metrics for determining the state of the first environment; one or more parameters for determining the behavior of the first model; the first environment; the behavior of the first model; state information of the first environment; behavioral information of the first model; or the training environment of the first model.

[0046] In this embodiment of the application, by encapsulating reinforcement learning capabilities as a function, the second device does not need to care about the underlying information in the reinforcement learning model training process, which helps to reduce the processing complexity of the second device and reduce transmission overhead.

[0047] In some embodiments, the method further includes: notifying the first device of capability information, the capability information of the first device being related to reinforcement learning, wherein the capability information of the first device includes one or more of the following: a first function, wherein the first device supports training a model that implements the first function; a second environment, wherein the first device supports an environment for training the model; a third metric list, wherein the third metric list includes metrics that the first device supports using when determining the interaction environment of the model; a second parameter list, wherein the second parameter list includes parameters that the first device supports when determining the behavior of the model; or a correspondence between metrics in the third metric list and parameters in the second parameter list.

[0048] For example, notifying the first device of its capability information can also be replaced by reporting the first device's capability information or sending the first device's capability information.

[0049] In this embodiment of the application, by notifying the model training provider of capability information (such as capability information related to reinforcement learning model training), it helps the model training user to determine the model training configuration information that matches the capability information of the model training provider, thereby helping to improve the feasibility of reinforcement learning model training.

[0050] In some embodiments, receiving the first information includes: receiving a first request, the first request being used to request training of the first model; wherein the first request includes a first attribute, the first attribute being used to indicate the first information.

[0051] By including initial information in the model training request, the degree of modification required to the relevant technologies can be reduced. Furthermore, this approach is relatively easy to implement.

[0052] In some embodiments, the method further includes: sending a model training report to a second device, the model training report including a second attribute and / or a third attribute, the second attribute indicating the training environment of the first model, and the third attribute indicating the performance metrics of the first model.

[0053] By including the model's training environment and / or the performance metrics of the first model in the model training report, the second device can better understand the model training process and support its model management. For example, based on the performance metrics of the first model, management strategies for the model inference process can be determined, such as identifying acceptable performance metrics during inference.

[0054] In some embodiments, the method further includes receiving a first threshold, the first threshold being used to indicate an acceptable performance metric for the first model during model inference.

[0055] An acceptable performance metric can refer to, for example, the minimum allowed value of a performance metric. Taking the performance metric as a reward value as an example, an acceptable performance metric can refer to the minimum allowed reward value during model inference based on the first model.

[0056] Setting the first threshold as described above helps to improve the reliability of model inference.

[0057] For example, since the reward values ​​in the last few rounds of model training are close to the reward values ​​that can be obtained in the model inference process, the first threshold can be determined based on the reward values ​​in the last few rounds of model training, which helps to improve the accuracy of the first threshold.

[0058] For example, the first threshold can be determined based on the performance metrics of the first model during model training.

[0059] In some embodiments, the method further includes: receiving second information, the second information being used to instruct the execution of model inference based on the first model; and sending a model inference report to a second device, the model inference report including performance metrics of the first model during the model inference process.

[0060] Reporting performance metrics during the model inference process helps users understand the model's inference status and the accuracy of the inference results, and provides support for the management and monitoring of the model inference process.

[0061] In some embodiments, if the first device is not the provider of model inference (i.e., in a scenario where model inference and model training are deployed separately), the first device can forward the model inference-related information sent by the second device to the provider of model inference (referred to as the third device), and the first device can send the model inference-related information reported by the third device to the second device.

[0062] The model inference information sent by the second device may include, for example, the second information and / or the first threshold. The model inference information reported by the third device may include, for example, the model inference report and / or notification message.

[0063] For example, the second device may send the aforementioned second information and / or first threshold to the first device, and the first device may send the aforementioned second information and / or first threshold to the third device.

[0064] For example, the third device may send the aforementioned model inference report and / or notification message to the first device, and the first device may send the aforementioned model inference report and / or notification message to the second device.

[0065] Thirdly, this application provides a communication device. In one possible design, the communication device has the functions to implement the first or second aspect described above. For example, the communication device includes modules, units, or means corresponding to the operations involved in the first or second aspect described above. These modules, units, or means can be implemented by software, hardware, or a combination of software and hardware.

[0066] Fourthly, this application provides a communication device including an interface circuit and one or more processors. The one or more processors are coupled to a memory. The memory stores part or all of the necessary computer programs or instructions for implementing the functions described in the first or second aspect. The one or more processors can execute the computer programs or instructions, which, when executed, cause the communication device to implement the methods in any possible design or implementation of the first or second aspect. The interface circuit is used to implement the communication functions within the communication device and / or the communication functions between the communication device and other devices or components.

[0067] In one possible implementation, the processor is used to communicate with other devices or components through the interface circuit.

[0068] In one possible implementation, the communication device may also include the memory.

[0069] Fifthly, this application provides a communication system including a first device and a second device. The first device or the second device may be a communication device provided in the fourth or third aspect.

[0070] In one possible implementation, the second device can execute the method provided in the first aspect, and the first device can execute the method provided in the second aspect.

[0071] Sixthly, this application provides a computer-readable storage medium storing computer-readable instructions that, when executed, cause the method in any of the above aspects or any possible implementations of any of the above aspects to be performed.

[0072] Seventhly, this application provides a computer program product including computer program instructions that, when executed, cause the method in any of the above aspects or any possible implementations of any of the above aspects to be implemented.

[0073] Eighthly, this application provides a chip including a processor, which, when executing a program or instructions, causes the methods in any possible implementation of the first and second aspects to be performed. Attached Figure Description

[0074] Figure 1 This is a diagram of an autonomous, hierarchical operation and maintenance system architecture;

[0075] Figure 2 A diagram illustrating the AI / ML workflow;

[0076] Figure 3A An example diagram illustrating how model inference and model training can be deployed;

[0077] Figure 3B Another example diagram illustrating how model inference and model training can be deployed;

[0078] Figure 3C Another example diagram illustrating how model inference and model training can be deployed;

[0079] Figure 4 A schematic diagram illustrating the principles of reinforcement learning;

[0080] Figure 5 This is a schematic flowchart illustrating a communication method provided in an embodiment of this application;

[0081] Figure 6 A flowchart illustrating another communication method provided in an embodiment of this application;

[0082] Figure 7 A flowchart illustrating yet another communication method provided in an embodiment of this application;

[0083] Figure 8 A flowchart illustrating yet another communication method provided in an embodiment of this application;

[0084] Figure 9 A flowchart illustrating yet another communication method provided in an embodiment of this application;

[0085] Figure 10 A schematic block diagram of a communication device provided in an embodiment of this application;

[0086] Figure 11 This is a schematic diagram of another communication device provided in an embodiment of this application. Detailed Implementation

[0087] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0088] In the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. "And / or" in this application merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. Additionally, to facilitate a clear description of the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or execution order, and that "first," "second," etc., do not necessarily imply differences. In this application, descriptions such as "under certain circumstances," "if," "when," "if," etc., can be used interchangeably. Furthermore, these descriptions all refer to situations where corresponding processing will be performed, not to time constraints, nor do they require a judgment action during implementation, nor do they imply any other limitations. In this application, "greater than or equal to" can be replaced with "greater than," and correspondingly, "less than" can also be replaced with "less than or equal to."

[0089] In the various method embodiments of this application, the order of the sequence numbers does not imply the order of execution. The execution order should be determined by its function and internal logic, and should not impose any limitations on the implementation process of the embodiments of this application.

[0090] It is understood that some optional features in the embodiments of this application can be implemented independently in certain scenarios without relying on other features, such as the current solution on which they are based, to solve the corresponding technical problems and achieve the corresponding effects. Alternatively, they can be combined with other features as needed in certain scenarios. Correspondingly, the apparatus given in the embodiments of this application can also implement these features or functions, which will not be elaborated here.

[0091] In this application, unless otherwise specified, the same or similar parts between the various embodiments can be referred to each other. In the various embodiments of this application, and in the various implementation methods / methods / implementations within each embodiment, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments and between the various implementation methods / methods / implementations within each embodiment are consistent and can be mutually referenced. The technical features in different embodiments and the various implementation methods / methods / implementations within each embodiment can be combined according to their inherent logical relationships to form new embodiments, implementation methods, methods, or implementation approaches. The embodiments described below do not constitute a limitation on the scope of protection of this application.

[0092] The communication method provided in this application can be applied to... Figure 1 In the architecture shown. See also Figure 1 The diagram shows an autonomous hierarchical operation and maintenance system architecture, which may include cross-domain management functional units, domain management functional units, and network elements. In some embodiments, the autonomous hierarchical operation and maintenance system architecture may also include a service operation unit. The functions of each unit and network element are briefly described below.

[0093] (1) Business Operation Unit: Also known as the communication service management function, it provides functions and management services such as billing, settlement, accounting, customer service, sales, network monitoring, communication service lifecycle management, and service intent translation. The business operation unit may include the operator's operating system or a vertical industry operating system.

[0094] (2) Cross-Domain Management Function Unit: Also known as Network Management Function (NMF). The Cross-Domain Management Function Unit can provide one or more of the following functions or management services: network lifecycle management, network deployment, network fault management, network performance management, network configuration management, network assurance, network optimization, translation of network intents from communication service providers (intent-CSP), translation of network intents from communication service consumers (intent-CSC), etc. The network here can include one or more network elements, subnetworks, or network slices. For example, the Cross-Domain Management Function Unit can be a network slice management function (NSMF), a management data analytical function (MDAF), a cross-domain self-organizing network function (SON-function), or a cross-domain intent management function unit.

[0095] For example, in certain deployment scenarios, the cross-domain management function unit may also provide one or more of the following management functions or services: subnetwork lifecycle management, subnetwork deployment, subnetwork fault management, subnetwork performance management, subnetwork configuration management, subnetwork assurance, subnetwork optimization functions, translation of subnetwork intents from communication service providers, and translation of subnetwork intents from communication service users. A subnetwork may consist of multiple smaller subnetworks or multiple network slice subnetworks.

[0096] For example, in implementation, the cross-domain management function unit can be implemented by a management platform and multiple management applications, wherein the management applications implement the management functions of one or more networks or sub-networks mentioned above.

[0097] (3) Domain Management Function Unit: Also known as a subnetwork management function (subnetwork NMF) or network element / function management function. The domain management function unit can provide one or more of the following functions or management services: subnetwork or network element lifecycle management, subnetwork or network element deployment, subnetwork or network element fault management, subnetwork or network element performance management, subnetwork or network element assurance, subnetwork or network element optimization management, subnetwork or network element intent translation, etc. Here, a subnetwork includes one or more network elements. Alternatively, a subnetwork may include one or more subnetworks, i.e., one or more subnetworks forming a larger coverage subnetwork. Or, a subnetwork may include one or more network slice subnetworks. A subnetwork can be described in one of the following ways:

[0098] A network within a specific technology domain, such as a wireless access network, core network, or transmission network.

[0099] A network of a certain standard, such as the Global System for Mobile Communications (GSM) network, the Long Term Evolution (LTE) network, and the 5th generation mobile communication technology (5G) network;

[0100] A network provided by a specific equipment vendor, such as a network provided by equipment vendor X;

[0101] A network for a specific geographical area, such as the network of factory A or the network of prefecture-level city B.

[0102] (4) Network element (NE): An entity that provides network services, which may include core network elements and / or access network elements. For example, core network elements may include, but are not limited to, access and mobility management function (AMF) entities, session management function (SMF) entities, policy control function (PCF) entities, network data analysis function (NWDAF) entities, network repository function (NRF) entities, gateways, etc. Access network elements may include, but are not limited to: various types of base stations (e.g., next-generation node B (gNB), evolved Node B (eNB), central unit control panel (CUCP), central unit (CU), distributed unit (DU), central unit user panel (CUUP), etc.

[0103] It should be understood that Figure 1 The number of each unit or network element shown is merely an example, and this application does not limit the number of each unit or network element.

[0104] To improve the intelligence and automation of networks, artificial intelligence (AI) and machine learning (ML) technologies are being applied in an increasing number of fields. The 3rd generation partnership project (3GPP) working group is researching several related topics for network intelligence. Among these, the research focuses on model lifecycle management.

[0105] Figure 2 This is a diagram illustrating the AI / ML workflow. Figure 2 The process shown mainly includes model training 210, model testing 220, model simulation 230, model deployment 240, and model inference 250.

[0106] Generally, a model training consumer (MLT consumer) can call the model training service provided by a model training provider (MLTproducer).

[0107] Taking the 3GPP network domain as an example, the network management system (i.e., the cross-domain management function unit mentioned above) can act as a user of model training. The model training provider can be a network element management device (i.e., the single-domain management function unit mentioned above), or a network element managed by the EMS, such as a radio access network (RAN), a base station (gNodeB, gNB), or a core network (CN) element, such as a network data analytics function (NWDAF), etc.

[0108] In this context, a base station can refer to a device in a mobile communication system that connects the fixed and wireless components and connects to mobile terminals via an over-the-air wireless channel. NWDAF (Network NodeB-Aided AF) possesses various intelligent computing functions such as AI training and inference. Network element can broadly refer to various network elements such as gNB (Gateway NodeB) and NWDAF.

[0109] Furthermore, in the ORAN network domain, the Service Management and Orchestration (SMO) function can act as a model training user, and the network elements directly managed by the SMO can act as model training providers. The SMO's role in the network architecture is similar to that of the NMS, responsible for the operation, management, and maintenance of various network services and orchestration functions. The network elements it directly manages can be heterogeneous, such as the EMS, gNB, and NWDAF. For example, the provider of model training can be called a Machine Learning Training Function (MLTF), generally referring to network elements with ML training capabilities.

[0110] Model inference and model training can be deployed on the same device or on different devices. Taking the 3GPP network domain (RAN domain) as an example... Figures 3A to 3C The diagrams illustrate different deployment methods for model inference and model training.

[0111] See Figure 3A Both model inference and model training are deployed on the domain management functional unit, namely EMS.

[0112] See Figure 3B Model inference is deployed on network elements, such as gNBs, while model training is deployed on domain management functional units.

[0113] See Figure 3C Both model inference and model training are deployed on network elements, such as gNB.

[0114] In some embodiments, a model training request can be initiated by the model training user to the model training provider. In the model training request, the model training user can specify information such as candidate training data source and performance requirements, as shown in Table 1. The performance requirements data is in the format of ModelPerformance and may include performance metrics, which are used to specify the loss function during training, as shown in Table 2.

[0115] Table 1

[0116]

[0117] Referring to Table 1, the support qualifiers for a given content indicate whether that content is mandatory. If it is mandatory (M), it must be included; if it is optional (O), it can be omitted; if it is conditional (C), it can be included / not included under certain conditions; and if it is CM, it must be included under certain conditions. A content readable / unreadable indicates whether it can be read / not read, and a content writable / unwritable indicates whether it can be written / not written. These can be indicated by true (T) or false (F).

[0118] Table 2

[0119]

[0120]

[0121] Referring to Table 2, T / F (NOTE) indicates that when this attribute is used in a model training request, the qualifier for isWritable is T; otherwise, the qualifier for isWritable is F.

[0122] 3GPP projects have begun to focus on reinforcement learning and are studying the full-process management and operation capabilities of AI / ML in 5G systems to support various AI / ML technologies, including reinforcement learning.

[0123] Reinforcement learning is an important branch of machine learning that focuses on how to take actions in an environment to maximize a cumulative reward; that is, the training objective of reinforcement learning is to maximize reward values. The core of reinforcement learning is that an agent learns optimal behavioral policies through interaction with the environment. For example, an agent can choose an appropriate action based on the current state of the environment (which can be acquired by an interpreter) and observe the environment's feedback on that action (such as changes in the environment's state), thereby learning how to optimize its decisions. Figure 4 As shown.

[0124] Reinforcement learning offers numerous advantages, such as its ability to adjust behavior according to changes in the environment, demonstrating strong adaptability and flexibility. Therefore, it has been widely applied in various fields, including gaming, automatic control, robotics, finance, and healthcare.

[0125] It is evident that introducing reinforcement learning into communication systems can help improve their performance. Therefore, how to apply reinforcement learning to communication systems is a problem that needs to be solved.

[0126] To address the aforementioned problems, embodiments of this application provide a communication method. In this method, the user of model training sends information related to reinforcement learning-based model training to the model training provider, which helps support model training using reinforcement learning methods. For example, the first information may include information about the environment in which the model interacts during reinforcement learning, thereby helping to determine the environment for reinforcement learning model training. Alternatively, the first information may include information about the model's behavior during reinforcement learning, thereby helping to determine the model's behavior during training.

[0127] The method provided in this application will be further described below with reference to the accompanying drawings.

[0128] Figure 5 This is a flowchart illustrating a communication method provided in an embodiment of this application. Figure 5 The method shown involves the interaction between a first device and a second device. The first device can be a provider of model training, and the second device can be a user of model training. The first device can be, for example, the EMS mentioned earlier, and the second device can be, for example, the NMS mentioned earlier.

[0129] It should be understood that the first or second device in the method provided in this application may be a physical device, a virtualization functional module, a software function, a circuit, a chip (such as a system on chip (SoC) chip or a system in package (SIP) chip), etc.

[0130] It should be noted that the following description is based on the interaction between the second device and the first device, and introduces the method provided in the embodiments of this application.

[0131] Figure 5 The method shown may include steps S510 and S520, and the steps are described below.

[0132] S510, the second device determines the first information.

[0133] The aforementioned first information can be used to train a first model, and the training method for the first model is reinforcement learning, or in other words, the first model is trained using reinforcement learning.

[0134] Based on the principles of reinforcement learning mentioned above, the first model can be understood as an intelligent agent, or an ML model that makes up the intelligent agent. For ease of description, the object that the first model interacts with during the model training process, i.e., the environment mentioned above, will be called the first environment; and the output of the first model, i.e., the behavior mentioned above, will be called the first behavior.

[0135] In some embodiments, the first information may include one or more of the following: a first metric list; a first parameter list; a first training environment; the functionality of the first model; the state information of the first environment; the behavioral information of the first model; or the performance metrics of the first model. The above information will be described in detail below.

[0136] First indicator list

[0137] For example, the metrics mentioned in the embodiments of this application can be a set of network performance measurement metrics or key performance indicators (KPIs).

[0138] For example, the state of the first environment can be determined by one or more indicators in the first indicator list. Alternatively, the state parameters of the first environment can include one or more indicators in the first indicator list. Furthermore, the state information of the first environment can be derived from the state parameters of the first environment and the values ​​of those parameters.

[0139] As an example, the metrics in the first metric list may include those specified in 3GPP Management Data Analytics (MDA) (refer to 3GPP TS28.104), those specified in 3GPP NWDAF (refer to 3GPP TS23.288), and those in 3GPP RAN.

[0140] For example, the metrics in the first metric list may include one or more of the following: reference signal received power (RSRP), physical resource block (PRB) utilization, or handover success rate.

[0141] Optionally, the state of the first environment can be determined by all indicators in the first indicator list; that is, the state information of the first environment can be determined by the set of values ​​for all indicators in the first indicator list. In this case, the second device provides the indicators used to determine the state of the first environment.

[0142] Taking the first indicator list, which includes RSRP, PRB utilization, and handover success rate, as an example, the status of the first environment can be determined by RSRP, PRB utilization, and handover success rate. The status information of the first environment can be determined by the set of values ​​for RSRP, PRB utilization, and handover success rate.

[0143] Optionally, the state of the first environment can be determined by a subset of indicators from the first indicator list; that is, the state information of the first environment can be determined by the set of values ​​for a subset of indicators in the first indicator list. In this case, the second device provides candidate indicators for determining the first environment, and the first device can select the indicator used to determine the state of the first environment from the candidate indicators.

[0144] Taking the first indicator list, which includes RSRP, PRB utilization, and handover success rate, as an example, the state of the first environment can be determined, for example, by RSRP and PRB utilization, or by RSRP and handover success rate. The indicators for determining the first environment can be determined by the first device.

[0145] If the state of the first environment is determined by RSRP and PRB utilization, where the set of values ​​for RSRP is X1, X2, X3, X4, X5, X6, and the set of values ​​for PRB utilization is 20%, 30%, 40%, 60%, 80%, 90%, then the state information of the first environment consists of the above values ​​of RSRP and PRB utilization.

[0146] It should be noted that the values ​​of the above indicators are given as examples only. The values ​​of the indicators can be discrete or continuous, and this application does not limit them.

[0147] First parameter list

[0148] For example, the behavior of the first model can be determined by one or more parameters in the first parameter list. In other words, the action parameters of the first model can include one or more parameters in the first parameter list. For example, the parameters in the first parameter list can include antenna settings, cell settings, beam settings, etc. Furthermore, the parameters in the first parameter list can include parameters specified in 3GPP management data analytics (MDA) (refer to 3GPP TS28.104), parameters specified in 3GPP NWDAF (refer to 3GPP TS23.288), and performance parameters in 3GPP RAN.

[0149] Optionally, the behavior of the first model can be determined by all parameters in the first parameter list; that is, the behavior information of the first model can be determined by the set of values ​​for all parameters in the first parameter list. In this case, the parameters for determining the first behavior are provided by the second device.

[0150] Taking the first parameter list as including antenna settings, cell settings, and beam settings as an example, the first row can be determined by antenna settings, cell settings, and beam settings, and the behavior information of the first model can be determined by the set of values ​​for antenna settings, cell settings, and beam settings.

[0151] Optionally, the behavior of the first model can be determined by a subset of parameters in the first parameter list; that is, the behavior information of the first model can be determined by the set of values ​​for a subset of parameters in the first parameter list. In this case, the second device provides candidate parameters for determining the first behavior, and the first device can select parameters from the candidate parameters to determine the first behavior.

[0152] Taking the first parameter list, which includes antenna settings, cell settings, and beam settings, as an example, the first row can be determined by, for example, antenna settings and beam settings, or the first row can be determined by, for example, cell settings and beam settings. The parameters used to determine the first row can be determined by the first device.

[0153] First training environment

[0154] The training environment of the first model can refer to an environment platform that can provide the first environment, or in other words, the training environment of the first model can provide the first environment for model training. Specifically, the environment platform can provide the state of the first environment, and the state of the first environment provided by the environment platform can respond according to different behaviors.

[0155] For example, the first training environment may include training environments for one or more first models, such as simulation environments or real-world environments. For instance, the first training environment may include simulation functions, such as network digital twin functions (NDT functions), available live network environments, etc. Optionally, the first information may indicate the first training environment through an identifier of a simulation function or an identifier of an available live network environment to save on indication overhead.

[0156] Optionally, the training environment for the first model can be a first training environment, or the training environment for the first model can be one of the first training environments. If the first training environment includes only one training environment, then the training environment for the first model can be the first training environment; if the first training environment includes multiple training environments, then the first device can select the training environment for the first model from the multiple training environments. This approach helps to improve the flexibility of model training.

[0157] For example, the second device can determine an available live network environment based on the functionality of the first model and / or indicators that determine the state of the first environment. For instance, if the indicators for determining the state of the first environment include RSRP, then a live network environment that supports RSRP collection or has RSRP collection information can be used as the first training environment. Alternatively, indicators used to determine the first environment can be determined based on the functionality of the first model, and then an available live network environment can be determined based on these indicators. Another example is using a live network environment that supports model training for the aforementioned functionality as the first training environment.

[0158] For example, the first device can select the training environment for the first model based on its capability information. As an example, for a first device that supports training the model in a live network environment, the live network environment can be selected as the training environment for the first model. As another example, for a first device that supports training the model using simulation capabilities, the simulation capability can be selected as the training environment for the first model.

[0159] Considering that it is relatively easy to obtain information about the current network environment from the second device, and that the second device usually has stronger information processing capabilities, it is simpler to implement the first training environment by having the second device provide it, which helps to reduce the processing complexity of the first device.

[0160] The function of the first model

[0161] For example, the first device can train the first model based on the functions of the first model issued by the second device. In this way, the information related to training the first model can be determined by the first device, thereby helping to improve the freedom and flexibility of the first device in training the first model.

[0162] For example, the first device can determine information related to the training of the first model (which can be referred to as third information) based on the functionality of the first model, and then train the first model based on the third information. The third information may include one or more of the following: one or more metrics for determining the state of the first environment; one or more parameters for determining the behavior of the first model; the first environment; the behavior of the first model; state information of the first environment; behavioral information of the first model; or the training environment of the first model.

[0163] Optionally, the third information may also include a list of first indicators and / or a list of first parameters.

[0164] Optionally, the third information may include the performance metrics of the first model. The meaning of the performance metrics of the first model can be found in the following description, and will not be repeated here for the sake of brevity.

[0165] The following is an exemplary description of a method for determining third information based on the functions of the first model.

[0166] As an example, the indicators used to determine the state of the first environment can be determined based on the correlation between the functions of the first model and the indicators (i.e., the indicators that determine the state of the environment).

[0167] As another example, the first list of indicators can be determined based on the relationship between the function of the first model and the list of indicators. Here, the indicators for determining the environmental state can be considered as the KPIs used for model training using reinforcement learning methods.

[0168] Furthermore, the first device may construct the first environment based on the first indicator list, or the aforementioned indicators used to determine the state of the first environment.

[0169] Optionally, the relationship between the model's functionality and the indicators, and / or the relationship between the model's functionality and the list of indicators, can be pre-stored in the first device.

[0170] As another example, the parameters used to determine the first behavior can be determined based on the association between the functions and parameters (i.e., the parameters that determine the behavior) of the first model. Optionally, the association between the functions and parameters can be pre-stored in the first device.

[0171] As another example, during model training, the first device can adjust the behavior strategy of the first model based on the current state of the environment and the functionality of the first model, which helps to speed up the training of the model.

[0172] It should be understood that the third information may also include other information used for training the first model, and this application does not limit this.

[0173] State information of the first environment / Behavioral information of the first model

[0174] In some embodiments, the first information may further include the state information of the first environment and / or the behavioral information of the first model. Compared to the first device determining this information, having the second device provide the state information of the first environment and / or the behavioral information of the first model helps reduce the processing complexity of the first device.

[0175] Performance metrics of the first model

[0176] In some embodiments, the performance metrics of the first model may include, for example, a reward metric and a reward function. The reward metric may include, for example, a single reward and a cumulative reward. The cumulative reward may include the cumulative reward within a certain time granularity and / or the cumulative reward throughout the entire training process. The reward function defines the feedback signal that the agent, or the ML model comprising the agent, can obtain after performing an action.

[0177] For example, the reward function of the first model is the initial reward function, which can be adjusted and iterated according to the training status of the model.

[0178] For example, the performance metric of the first model can characterize the mapping relationship between the reward value and the value of the metric that determines the first environment (or the amount of change in the value of that metric). The reward value mentioned here may include a single reward and / or a cumulative reward.

[0179] It should be understood that the first information may include one or more of the above information, and the first information may also include other information for reinforcement learning not shown in this application, which is not limited in this application.

[0180] S520, the second device sends first information to the first device, and correspondingly, the first device receives the first information sent by the second device. Here, the first device is the device that trains the first model, i.e., the model training provider mentioned above.

[0181] In some embodiments, the first information may be carried in a model training request. That is, the second device may send a first request to the first device, which may be used to request the training of a first model, wherein the first request includes the first information. Exemplarily, the first request may include a first attribute, which may be used to indicate the first information, such as indicating a first training environment.

[0182] Table 3 shows an example of a model training request provided in the embodiments of this application.

[0183] Table 3

[0184]

[0185] Referring to Table 3, the support qualifier for the first attribute is M / O. For example, when requesting reinforcement learning, the support qualifier for the first attribute can be M, meaning the first attribute is mandatory; in other cases, the first attribute can be optional.

[0186] Table 3 provides an example of a qualifier related to the first attribute. It should be understood that the qualifier related to the first attribute can also be of other types, and this application does not limit this.

[0187] By including initial information in the model training request, the degree of modification required to the relevant technologies can be reduced. Furthermore, this approach is relatively easy to implement.

[0188] In some embodiments, the first information may be carried in the model performance. For example, the model performance may include a fourth attribute, which may be used to indicate the first information, such as a list of first metrics and / or a list of first parameters.

[0189] Table 4 provides an example of the model performance provided in the embodiments of this application.

[0190] Table 4

[0191]

[0192]

[0193] Table 4 provides an example of a qualifier related to the fourth attribute. It should be understood that the qualifier related to the fourth attribute can also be of other types, and this application does not limit this.

[0194] It should be understood that multiple pieces of information in the first message can be contained within the same message or in different messages. For example, the first training environment can be contained within the model training request, and the first metric list can be contained within the model performance.

[0195] It should be understood that the first information can also be carried in other types of information, and this application does not limit this.

[0196] In this embodiment of the application, users of model training can help support the application of reinforcement learning technology by sending information related to reinforcement learning-based model training to the model training provider.

[0197] In traditional machine learning, training a model using training data from a single class allows for the acquisition of a one-to-one mapping relationship. Taking supervised learning as an example, training data from a single class can refer to training data for a single feature; by training the model, the mapping relationship between that feature and the label can be obtained.

[0198] Unlike traditional machine learning, reinforcement learning involves the interaction between an agent and its environment, where changes in the environment affect subsequent actions. Traditional machine learning training data often fails to represent the environment in which reinforcement learning interacts. In other words, training methods for traditional machine learning models are not applicable to training reinforcement learning-based models.

[0199] The embodiments of this application, through information such as the first index list and the function of the first model, can provide the model training provider with a basis for determining the above-mentioned environment, thereby helping to solve the problem that traditional training data cannot be applied to the training of reinforcement learning-based models.

[0200] There are various ways to determine the first piece of information, and the methods for determining different pieces of information within the first piece of information may be the same or different. The methods for determining the first piece of information are described in detail below.

[0201] In some embodiments, the first information can be determined based on the model training requirement information, such as the second device receiving the model training requirement information and then determining the first information based on the model training requirement information.

[0202] For example, the model training requirements may include the functionality of a first model, or the model training requirements may include the functionality of a first model and a second list of metrics. The metrics in the second list are those expected to be used to determine the state of the first environment.

[0203] For example, the function of the first model in the first information is the same as the function of the first model included in the model training requirement information.

[0204] For example, the first information includes a first list of indicators, and the first device can determine the first list of indicators based on the functions of the first model.

[0205] As an example, the first indicator list can be determined based on the relationship between the functions of the first model and the indicators.

[0206] Table 5 shows an example of the relationship between the model's function and the indicators.

[0207] Table 5

[0208]

[0209] Referring to Table 5, Function 1 is related to Indicator 1 and Indicator 2; Function 2 is related to Indicators 3 to 5; and Function 3 is related to Indicators 6 to 8. If the function of the first model is Function 2, then the first indicator list may include Indicators 3 to 5.

[0210] As another example, the first indicator list can be determined based on the relationship between the functions of the first model and the indicator list. Table 6 shows an example of the relationship between the functions of the model and the indicator list.

[0211] Table 6

[0212]

[0213]

[0214] Referring to Table 6, Function 1 is associated with Indicator List 1, Function 2 is associated with Indicator List 2, and Function 3 is associated with Indicator List 3. If the function of the first model is Function 3, then the first indicator list is Indicator List 3.

[0215] Furthermore, the indicators in the first indicator list can be determined based on the correspondence between the first indicator list and the indicators. As an example, the first information may include the identifier of the first indicator list, and the first device can determine the indicators in the first indicator list based on the correspondence between the first indicator list and the indicators. Alternatively, the second device can determine the indicators in the first indicator list based on the correspondence between the first indicator list and the indicators. In this case, the first information may include each indicator in the first indicator list.

[0216] Optionally, the relationship between the model's functions and the indicators, and / or the relationship between the model's functions and the list of indicators, can be pre-stored in the second device and / or the first device.

[0217] For example, if the first information includes a first list of indicators, and the requirements for model training include a second list of indicators, then the first list of indicators can be determined based on the second list of indicators, such as the first list of indicators being the second list of indicators.

[0218] For example, the first information may include a first parameter list, which the first device can determine based on the functionality of the first model. As an example, the first parameter list can be determined based on the relationship between the functionality of the first model and the parameters. As another example, the first parameter list can be determined based on the relationship between the functionality of the first model and the parameter list.

[0219] Optionally, the first information may include an identifier for the parameter list, and the first device determines the parameters in the first parameter list based on the identifier and the correspondence between the first parameter list and the parameters. Alternatively, the second device may determine the parameters in the first parameter list based on the correspondence between the first parameter list and the parameters. In this case, the first information includes each parameter in the first parameter list.

[0220] Optionally, the relationship between the model's functions and parameters, and / or the relationship between the model's functions and parameter lists, can be pre-stored in the second device and / or the first device.

[0221] It should be understood that the method of determining the first parameter list based on the function of the first model is similar to the method of determining the first indicator list based on the function of the first model mentioned above. For the sake of brevity, it will not be described again here.

[0222] It should be understood that the method for determining the first parameter list based on the function of the first model can also include: first determining the first indicator list based on the function of the first model, and then determining the first parameter list according to the correspondence between the indicators and parameters in the first indicator list. This method will be described in detail later and will not be repeated here.

[0223] For example, the first information includes the state information of the first environment. As one example, the state information of the first environment can be determined based on the first indicator list, which in turn can be determined based on the methods described earlier. As another example, the state information of the first environment can be directly determined based on the function of the first model, such as by determining it based on the correlation between the model's function and the state information of the first environment.

[0224] For example, the first information includes the behavioral information of the first model. As one example, the behavioral information of the first model can be determined based on the first parameter list, which in turn can be determined using the methods described earlier. As another example, the behavioral information of the first model can be directly determined based on the function of the first model, such as based on the relationship between the model's function and the behavioral information of the first model.

[0225] For example, the first information includes the first training environment, which can be determined based on the function of the first model.

[0226] For example, the first information includes the performance indicators of the first model, and the performance indicators of the first model can be determined based on the functions of the first model.

[0227] In some embodiments, the first information can be determined based on the capability information of the first device. For example, the second device can obtain the capability information of the first device and then determine the first information based on the capability information of the first device. The capability information of the first device can be referred to in the relevant description below, and will not be repeated here.

[0228] The capability information of the first device may include, for example, a third indicator list, which may include indicators that the first device supports using when determining the interaction environment of the model. In other words, the capability information of the first device is used to indicate which indicators (i.e., the indicators in the third indicator list) the first device supports using to determine the interaction object of the first model, i.e., the state of the environment.

[0229] Optionally, the list of third metrics supported by the first device is associated with the training environment. That is, the list of third metrics supported by the first device may be the same or different in different training environments.

[0230] Taking the first information including the functions of the first model as an example, if the capability information of the first device includes a third indicator list, the first indicator list can be determined based on the correspondence between the indicators in the third indicator list and the functions of the first model. For example, the first indicator list may include the indicators in the third indicator list that correspond to the functions of the first model.

[0231] In some embodiments, the first information can be determined based on the capability information of the first device and the model training requirements. For example, the first indicator list can be determined based on a second indicator list and a third indicator list, such that the first indicator list includes some indicators from the third indicator list.

[0232] As an example, if the third indicator list includes the second indicator list, then the first indicator list can be the second indicator list.

[0233] Determining the first indicator list using the above method and training the model based on the first indicator list helps the training results of the first model meet the requirements of the party proposing the training needs, that is, meet the user's needs.

[0234] As another example, if the third indicator list includes some of the indicators in the second indicator list, then the first indicator list includes the indicators common to both the second and third indicator lists. In other words, the indicators in the first indicator list belong to both the second and third indicator lists.

[0235] The method described above for determining the first list of indicators can avoid situations where the first device does not support the configured indicators, thereby helping to improve the feasibility of model training.

[0236] If the third metric list does not include the second metric list, the first metric list can be determined based on the third metric list, i.e., the first metric list is a subset of the third metric list. Further, the first device can send a response message to the party proposing the model training request. This response message can, for example, indicate that: the metrics provided by the party proposing the model training request were not used, and / or, the first metric list used.

[0237] The capability information of the first device may also include a second parameter list, which may include the parameters supported by the first device when determining the behavior of the model. In other words, the capability information of the first device is used to indicate which parameters (i.e., the parameters in the second parameter list) the first device supports for determining the behavior of the model.

[0238] If the first information includes a first parameter list, then if the capability information of the first device includes a second parameter list, the first parameter list can be determined based on the correspondence between the parameters in the second parameter list and the functions of the first model. Alternatively, the first parameter list can be determined based on the parameter list provided by the proposer of the model training requirements and the second parameter list. For example, the first parameter list may include parameters common to both the second parameter list and the parameter list provided by the proposer of the training requirements.

[0239] In some embodiments, the capability information of the first device includes a third index list, a second parameter list, and a correspondence between the indices in the third index list and the parameters in the second parameter list (for ease of description, this can be referred to as a first correspondence). In this case, the first parameter list can be determined based on the indices in the first index list and the first correspondence, wherein the first parameter list includes parameters that have the said correspondence with the indices in the first index list.

[0240] The correspondence between metrics and parameters can be understood as follows: when training the first model in an environment defined by a metric, it can support the behavior defined by that parameter; or, in other words, changes in the parameter will affect the state of the metric. This is because, when training the first model in environments with different defined metrics, the behaviors supported by those parameters may be the same or different. Therefore, determining the first parameter list based on this correspondence helps improve the rationality of the determined behaviors.

[0241] As an example, the first correspondence can be stored in the form of a list, as shown in Table 7. Table 7 provides an example of the first correspondence.

[0242] Table 7

[0243]

[0244] Referring to Table 7, index 1 corresponds to parameter 1 and parameter 2; index 2 corresponds to parameter 3; and index 3 corresponds to parameter 4, parameter 5, and parameter 6.

[0245] Taking the first correspondence shown in Table 7 as an example, if the third indicator list includes indicator 1, indicator 2, and indicator 3, the second parameter list includes parameters 1 to 6. If the first indicator list includes indicator 1 and indicator 2, then the first parameter list may include parameters 1, 2, and 3.

[0246] In addition, different devices or different device manufacturers may support different correspondences as described above. Therefore, determining the first parameter list based on the above correspondences can configure the first parameter list supported by different devices, which helps to improve the compatibility of the model training configuration information with the training device.

[0247] In some embodiments, multiple first correspondences may be associated one-to-one with multiple devices, or multiple first correspondences may be associated one-to-one with multiple manufacturers, that is, devices belonging to the same manufacturer have the same first correspondence.

[0248] For example, the first correspondence may be pre-configured (or stored) in the second device, or reported by the first device.

[0249] For example, in cases where multiple first correspondences can be associated one-to-one with multiple devices, the first correspondences can be reported by the first device, such as when the first device needs to perform model training. In this case, the second device does not need to maintain multiple first correspondences associated with multiple first devices, which helps reduce the memory overhead of the second device.

[0250] For example, in cases where multiple primary correspondences are associated one-to-one with multiple manufacturers, these primary correspondences can be stored or pre-configured in a secondary device, which then maintains them. This approach avoids the latency caused by reporting primary correspondences and helps improve the timeliness of primary model training.

[0251] In some embodiments, the first information may include index information, such as any one of Tables 5, 6, or 7, to indicate the function of the first model and the corresponding indicators and / or parameters, thereby helping to save on the overhead of indicating the first information. For example, the first information may include index 1, which indicates that the function of the first model is function 1 and the first indicator list includes indicator 1 and indicator 2. As another example, the first information may include index 5, which indicates that the function of the first model is function 2 and the first indicator list is indicator list 2.

[0252] To address the problem that communication systems in related technologies cannot support the training of reinforcement learning models, embodiments of this application provide a communication method. This method, by reporting the capability information of the model training provider (such as capability information related to reinforcement learning model training), helps the user of model training to determine the configuration information of model training that matches the capability information of the model training provider, thereby helping to improve the feasibility of reinforcement learning model training.

[0253] Figure 6 This is a flowchart illustrating another communication method provided in an embodiment of this application. Figure 6The method shown involves interaction between a first device and a second device. The first device can be a provider of model training, such as the EMS mentioned above, and the second device can be a user of model training, such as the NMS mentioned above.

[0254] Figure 6 The method shown may include step S610. The method provided in the embodiments of this application will be described below from the perspective of the interaction between the first device and the second device.

[0255] S610, the second device obtains the capability information of the first device, and correspondingly, the first device can notify the second device of the capability information of the first device.

[0256] In some embodiments, the capability information of the first device is related to reinforcement learning, or the capability information of the first device may be referred to as information about the reinforcement learning capability of the first device. For example, the capability information of the first device may include one or more of the following: a first function, wherein the first device supports training a model that implements the first function; a second environment, wherein the first device supports an environment for training the model; a third metric list, wherein the third metric list includes metrics that the first device supports using when determining the interaction environment of the model; a second parameter list, wherein the second parameter list includes parameters that the first device supports when determining the behavior of the model; or a correspondence between the metrics in the third metric list and the parameters in the second parameter list (which may be referred to as a first correspondence).

[0257] As an example, the first function in the aforementioned capability information can be associated with and stored in the third indicator list; and / or the first function can be associated with and stored in the second parameter list. In this way, when determining the function of the first model, the indicator used to determine the state of the first environment can be determined based on the correspondence between the first function and the indicators in the third indicator list, or the parameter used to determine the behavior of the first model can be determined based on the correspondence between the first function and the parameters in the second parameter list.

[0258] For example, the capability information of the first device may include the value range of the indicators in the third indicator list. Taking PRB utilization rate as an example, the range of PRB utilization rate supported by different first devices may be the same or different; or, the range of PRB utilization rate supported by the same device in different environments may be the same or different. Determining the state information of the first environment based on the value range of the indicators in the third indicator list helps to improve the feasibility and reliability of training the first model.

[0259] For example, the capability information of the first device may include the value range of the parameters in the second parameter list. Taking antenna settings included in the second parameter list as an example, different first devices may support the same or different ranges of antenna angle adjustment. Determining the behavioral information of the first model based on the value range of the parameters in the second parameter list helps to improve the feasibility and reliability of training the first model.

[0260] For example, the second environment may refer to an available training environment supported by the first device, such as NDT or an available live network environment.

[0261] In some embodiments, the capability information of the first device may be used to determine one or more of the following: indicators for determining the state of the first environment; parameters for determining the behavioral information of the first model; state information of the first environment; or behavioral information of the first model, etc.

[0262] For example, a third indicator list can be used to determine the state and / or state information of the first environment, such as determining the state and / or state information of the first environment based on some or all of the indicators in the third indicator list. As an example, the indicators used to determine the state of the first environment can be determined first, and then the state information of the first environment can be determined based on the value range of the indicators.

[0263] For example, the second parameter list can be used to determine the behavior and / or behavioral information of the first model, such as determining the behavior and / or behavioral information of the first model through some or all of the parameters in the second parameter list. As an example, the parameters used to determine the behavior of the first model can be determined first, and then the behavioral information of the first model can be determined based on the range of values ​​of the parameters.

[0264] The interaction methods for the capability information of the first device (the methods by which the second device obtains the capability information of the first device, or the first device notifies the first device of its capability information) can include a variety of methods, which will be described below.

[0265] In some embodiments, the capability information of the first device may be reported periodically. When a preset period is reached, the first device sends its capability information to the second device.

[0266] In some embodiments, the capability information of the first device is reported in an updated manner. That is, when the capability information of the first device changes, the first device sends its capability information to the second device. This helps to update the capability information of the first device on the second device side in a timely manner, thereby helping to avoid the impact of untimely updates of the first device's capability information on the training performance of the first model. Optionally, the first device can send all of its updated capability information to the second device, or the first device can send only the updated capability information to the second device to save transmission overhead.

[0267] In some embodiments, the capability information of the first device is reported during registration. That is, when the first device registers, such as when registering in the management domain of the second device, the first device sends its capability information to the second device.

[0268] In some embodiments, the second device may request capability information of the first device from the first device, and in response to the request, the first device may send the capability information of the first device to the second device.

[0269] In some embodiments, the second device may subscribe to the capability information of the first device. For example, the timing of the second device acquiring the capability information of the first device varies depending on the subscription method. Optionally, after the second device subscribes to the capability information of the first device, the second device may periodically acquire the capability information of the first device, i.e., the subscription message is triggered periodically. Optionally, after the second device subscribes to the capability information of the first device, if the capability information of the first device changes, the second device may acquire the updated capability information of the first device, i.e., the subscription message is triggered by an event. It should be understood that acquiring the capability information of the first device mentioned herein may include receiving a notification containing the capability information of the first device.

[0270] It should be understood that the two communication methods mentioned above can also be used in combination. This scheme may include, for example, steps 1 through 3. Details not described in detail can be found in the preceding text.

[0271] Step 1: The second device obtains the capability information of the first device. Accordingly, the first device notifies the second device of its capability information.

[0272] Step 2: The second device determines the first information.

[0273] In some embodiments, the second device may determine the first information based on the model training requirements and / or the capability information of the first device. This scheme can be referred to in the preceding description, and for the sake of brevity, will not be repeated here.

[0274] Step 3: The second device sends the first information to the first device, and correspondingly, the first device receives the first information sent by the second device.

[0275] The first information can be used to train a first model. In some embodiments, the first device can determine one or more of the following based on the first information: a first environment, state information of the first environment, behavioral information of the first model, a training environment, or performance metrics. Further, the first device can train the first model based on this information.

[0276] In some embodiments, the capability information of the first device may be encapsulated as a function, that is, the capability information of the first device includes a first function. The first function may include, for example, various RAN domain functions (refer to the use case in 3GPP TS 38.300), such as but not limited to mobile load balancing functions, coverage optimization functions, etc.; functions in various management data analysis functions (MDAF) in domain management functions (refer to the use case in 3GPP TS 28.104), such as but not limited to coverage problem analysis functions, slice coverage analysis functions, etc.; or one or more of the various functions defined in the NWDAF of the CN domain (refer to the use case in 3GPP TS 23.288).

[0277] In this case, the first information may include the function of the first model (but not the first index list, the first parameter list, the state information of the first environment, and the behavior information of the first model, etc.). The first device may determine the training-related information of the first model based on the function of the first model, such as the first index list, the first parameter list, the state information of the first environment, and the behavior information of the first model, etc.

[0278] In this way, the second device only needs to focus on information at the model's functional level, without needing to worry about information at the metric or parameter level, which helps reduce the processing complexity and transmission overhead of the second device. Additionally, the first device can store information related to the first function, showing how the model was trained using reinforcement learning methods, to support the above scheme.

[0279] In some embodiments, the capability information of the first device is unencapsulated functionality, meaning that the capability information of the first device does not include the first function, but includes a third indicator list, a second parameter list, and a first correspondence, etc. In this case, the first information may include the first indicator list, the first parameter list, the state information of the first environment, and the behavioral information of the first model, etc.

[0280] It should be noted that if the capability information of the first device includes the second parameter list, then the first information may include the first parameter list, or the first parameter list and the behavioral information of the first model, or it may not include the first parameter list and the behavioral information of the first model. If the capability information of the first device does not include the second parameter list, then the first information may not include the first parameter list and the behavioral information of the first model. This scheme helps to avoid the situation where the first device does not support determining the behavioral information of the first model from the parameters in the first parameter list, thereby improving the reliability of training the first model.

[0281] Alternatively, if the first information includes a first parameter list, i.e., the parameters or candidate parameters provided by the second device to determine the behavior of the first model, then the first device may report the second parameter list to the second device; if the first information does not include the first parameter list, then the first device may not report the second parameter list to the second device.

[0282] In this embodiment of the application, considering the capability information of the first device during the configuration of the first information helps to improve the feasibility of training the first model.

[0283] In some embodiments, the first device may send the environment used to train the first model to the second device. For example, the first device may select a second training environment from the first training environment and train the first model in the second training environment. As an example, the first device may send the second training environment to the second device. Optionally, the second training environment may be indicated by an identifier of the training environment to save on indication overhead.

[0284] In some embodiments, if the first device uses a subset of indicators from a first indicator list to determine the first environment, then the first device may report the indicators used to determine the state of the first environment to the second device. If the first device uses a subset of parameters from a first parameter list to determine the behavior of the first model, then the first device may report the parameters used to determine the behavior of the first model to the second device. For example, the model training report may include parameters used to determine the behavior of the first model and / or indicators used to determine the state of the first environment.

[0285] In some embodiments, the first device may send performance metrics of the first model to the second device, such as performance metrics during training. Exemplarily, performance metrics may include reward metrics and reward functions. Reward metrics may include, for example, single-time rewards and cumulative rewards. Cumulative rewards may include cumulative rewards within a certain time granularity and / or cumulative rewards throughout the entire training process. The reward function defines the feedback signal that the agent or the ML model comprising the agent can obtain after performing an action.

[0286] For example, performance metrics over a period of time can be reported to the second device, such as performance metrics in the last few training rounds, or all performance metrics during the training process can be reported to the second device. The number of rounds for which performance metrics need to be reported can be determined based on a preset value. For example, with a preset value of 5, the first device can send the performance metrics from the last 5 rounds of the first model's training process to the second device.

[0287] For example, the performance metrics of the first model may include interaction records of the environment, behavior, and reward values. These elements in the interaction records have a temporal relationship, or a sequential relationship. This approach helps the second device understand the training process of the first model. Furthermore, based on these interaction records, the second device can determine model management strategies, such as updating model training parameters, thereby contributing to improved system performance.

[0288] In some embodiments, the first device may send a model training report to the second device, and correspondingly, the second device may receive the model training report sent by the first device. The model training report may include one or more of the following: the environment used to train the first model (such as a second training environment), parameters determining the behavior of the first model, indicators determining the state of the first environment, or performance indicators of the first model.

[0289] For example, the model training report may include a second attribute and / or a third attribute, wherein the second attribute is used to indicate the training environment of the first model, such as a second training environment; and the third attribute is used to indicate the performance metrics of the first model, such as the reward value during the training of the first model.

[0290] By including the training environment and performance of the first model in the model training report, the second device can understand the model training status and provide support for model management on the second device.

[0291] The training process of the first model has been introduced above. The inference process of the first model will be introduced below.

[0292] In some embodiments, the first device may send second information to the second device, and correspondingly, the second device may receive the second information from the first device. The second information may be used to instruct the execution of model inference based on the first model. Optionally, the second information may also be understood as activation information for model inference.

[0293] In some embodiments, the first device may send a model inference report to the second device, and correspondingly, the second device may receive the model inference report sent by the first device. Optionally, the model inference report includes performance metrics of the first model during the model inference process. The performance metrics of the first model included in the inference report may, for example, include one or more of the following: reward value during the entire model inference process, reward value during the model inference process over a period of time, the relationship between the reward value and a first threshold (e.g., whether the reward value is less than the first threshold), reward values ​​less than the first threshold, or interaction records of environment-behavior-reward value.

[0294] In some embodiments, the first device may receive a first threshold sent by the second device, and correspondingly, the second device may send the first threshold to the first device. The first threshold can be used to indicate an acceptable performance metric (or a threshold for an acceptable performance metric) of the first model during model inference, such as the minimum allowed value of the performance metric. Taking the performance metric as a reward value as an example, the acceptable performance metric can refer to the minimum allowed reward value during model inference based on the first model.

[0295] For example, since the reward values ​​in the last few rounds of model training are close to the reward values ​​that can be obtained in the model inference process, the first threshold can be determined based on the reward values ​​in the last few rounds of model training, which helps to improve the accuracy of the first threshold.

[0296] For example, the first threshold can be determined based on the performance metrics of the first model during model training.

[0297] For example, during the inference process of the first model, if the reward value is less than a first threshold, the first device can send a notification message to the second device. This notification message can be used to inform the second device that the reward value is less than the first threshold, or it can be used to inform the second device that the reward value is less than the first threshold, along with the current reward value. Based on this notification message, the second device can manage the inference process of the first model in a timely manner, such as optimizing the first model or changing the inference model, thereby helping to improve system performance.

[0298] To increase the robustness of the system, when the number of times the reward value is less than the first threshold reaches a preset number, the first device can send a notification message to the second device. This notification message can be used to notify the second device that the number of times the reward value is less than the first threshold has reached a preset number, or the notification message can be used to notify the second device that the number of times the reward value is less than the first threshold has reached a preset number, as well as the reward value less than the first threshold during the inference process of the first model.

[0299] For example, the notification message can be sent periodically. For instance, the notification message can be used to notify the second device of the reward value during the inference process of the first model within a certain time granularity, and the relationship between the reward value and a first threshold, such as all reward values ​​being greater than the first threshold, or some reward values ​​being greater than the first threshold. To save overhead, the notification message can be used to notify the second device of reward values ​​less than the first threshold during the inference process of the first model within a certain time granularity.

[0300] As an example, the model training report may include the notification message mentioned above.

[0301] In some embodiments, if the first device is not the provider of model inference (i.e., in a scenario where model inference and model training are deployed separately), the first device can forward the model inference-related information sent by the second device to the provider of model inference (referred to as the third device), and the first device can send the model inference-related information reported by the third device to the second device.

[0302] The model inference information sent by the second device may include, for example, the second information and / or the first threshold. The model inference information reported by the third device may include, for example, the model inference report and / or notification message.

[0303] For example, the second device may send the aforementioned second information and / or first threshold to the first device, and the first device may send the aforementioned second information and / or first threshold to the third device.

[0304] For example, the third device may send the aforementioned model inference report and / or notification message to the first device, and the first device may send the aforementioned model inference report and / or notification message to the second device.

[0305] Reporting performance metrics during the model inference process helps users understand the model's inference status and the accuracy of the inference results, and provides support for the management and monitoring of the model inference process.

[0306] It should be noted that the reward value mentioned in the embodiments of this application may include a single reward value and / or a cumulative reward value. The cumulative reward value may include one or more of the following: the cumulative reward value over a certain period of time, the cumulative reward value during the entire model training process, or the cumulative reward value during the entire model inference process.

[0307] It should be understood that the reward value mentioned in the embodiments of this application can be the reward value during the model training process or the reward value during the model inference process.

[0308] It should be understood that the list mentioned in the embodiments of this application is only an example. The indicators in the indicator list and / or the parameters in the parameter list can also be stored in other forms, such as arrays. This application does not limit this.

[0309] It should be noted that the method provided in this application embodiment can also be applied to the ORAN architecture. In the ORAN architecture, the second device can be an SMO, the first device can be an MLTF, and the third device can be the network element mentioned above, such as a gNB or NWDAF. Under the ORAN architecture, the actions performed by the user of model training (SMO), the provider of model training (MLTF), and the provider of model inference (network element) can be found in the preceding description. For simplicity, they will not be repeated here.

[0310] For ease of understanding, the following text will combine... Figures 7 to 9 Taking the first model as an example, the method provided in the embodiments of this application will be introduced.

[0311] Scenario 1: Co-deployment of training and inference

[0312] Figure 7 This is a flowchart illustrating another communication method provided in an embodiment of this application. Figure 7 The method shown can involve interaction between a first device and a second device, belonging to a scenario where the model training provider and the model training user are co-deployed. Here, the second device is an NMS, and the first device is an EMS.

[0313] Figure 7 The method shown may include steps 1 through 10.

[0314] In step 1, the NMS sends a capability query message to the EMS. This message is used to query capability information related to the training of the reinforcement learning model.

[0315] In step 2, EMS sends capability information to NMS.

[0316] For example, capability information may include a list of adjustable KPIs, i.e., the third indicator list mentioned above. Optionally, capability information may also include a list of adjustable parameters, i.e., the second parameter list mentioned above. In this case, capability information may also include the correspondence between the indicators in the third indicator list and the parameters in the second parameter list.

[0317] For example, an adjustable KPI list can include wireless network performance metrics such as RSRP, PRB utilization, and handover success rate, which can be used to determine the environment for reinforcement learning. Adjustable parameters can be wireless network parameters that affect the aforementioned performance metrics, such as antenna settings and cell settings, which can be used to select reinforcement learning behaviors.

[0318] For example, capability information may include available training environments A supported by EMS, such as NDT or available live network environments.

[0319] In step 3, NMS receives new functional requirement information.

[0320] The function mentioned here refers to the model's functionality. In other words, NMS receives new model training requirements, which include the model's functionalities. Optionally, this functional requirement information may also include a second list of metrics corresponding to that function, where the metrics are expected to be used to determine the model training environment.

[0321] In step 4, NMS determines the first parameter list and the first index list of EMS.

[0322] For example, NMS can determine the first indicator list of EMS based on the third indicator list in step 2 (i.e., carried in the capability information) and the second indicator list in step 3 (i.e., carried in the functional requirement information).

[0323] If the third indicator list in step 2 includes all the indicators in the second indicator list in step 3, then the first indicator list of EMS can be the second indicator list in step 3.

[0324] If the third indicator list in step 2 includes some of the indicators in the second indicator list in step 3, then the first indicator list of EMS may include those partial indicators.

[0325] For example, NMS can determine the first parameter list and the first indicator list based on the correspondence between the parameters in the second parameter list in step 2 and the indicators in the third indicator list. For instance, the first parameter list of EMS may include parameters that correspond to the indicators in the first indicator list of EMS.

[0326] EMS can train the model using reinforcement learning methods based on a first parameter list and / or a first metric list. For example, the metrics in the first metric list are indicators used to determine the state of the environment for model training; the parameters in the first parameter list are parameters used to select the behavior for model training.

[0327] When NMS manages model training for multiple EMS, NMS can determine the first parameter list and the first index list for each of the multiple EMS.

[0328] In step 5, NMS sends a model training request to EMS.

[0329] For example, the model is trained using a reinforcement learning method.

[0330] For example, the model training request may include the first information mentioned above, such as the functionality of the first model, the available training environment (i.e., the first training environment mentioned above, which is a subset of the available training environment A), the first metric list, the first parameter list, and the performance metrics of the first model.

[0331] Among them, the available training environment can support the construction of the environment during the training process of the reinforcement learning model. For example, the available training environment can be identified as NDT or available live network environment.

[0332] The metrics in the first metric list can be used to determine the state of the environment in which the model is trained, and / or state information of that environment.

[0333] The parameters in the first parameter list can be used to determine the behavior of the first model, and / or the first parameter list can be used to determine the behavioral information of the first model.

[0334] The first parameter list is optional. If the training request does not include the first parameter list, then the capability information reported in step 2 may not include the second parameter list.

[0335] In step 6, EMS performs model training of the first model based on the model training request, such as the first information in the model training request.

[0336] In step 7, EMS reports the model training report to NMS.

[0337] For example, the training report may include the training environment used for model training, such as the available live network environment or NDT.

[0338] For example, the training report may include performance metrics for the first model. Performance metrics for the first model may include, for example, a reward metric and / or a reward function. As an example, the performance metrics for the first model may include one or more of the following: single reward value, cumulative reward value, or behavior-environment-reward value interaction records. Optionally, the training report may include behavior-environment-reward value interaction records from the last few rounds.

[0339] In step 8, the NMS sends the threshold of acceptable performance indicators to the EMS, namely the first threshold mentioned above.

[0340] The first threshold could be, for example, a threshold for the acceptable reward value when the agent or the ML model comprising the agent performs an action. Exemplarily, the NMS can configure the first threshold based on performance metrics of the first model obtained in step 6, such as the behavior-environment-reward interaction log. For example, the first threshold could be the average of the reward values ​​obtained in the last few rounds, or a certain percentage of that average, such as 90% of the average.

[0341] In step 9, the NMS sends a second message to the EMS, namely the activation message for performing model inference.

[0342] If the second information is received, the EMS can perform model inference of the first model based on the first threshold.

[0343] For example, during model inference, performance metrics such as whether the reward value exceeds a first threshold can be determined. Optionally, it can be determined whether the reward value exceeds the threshold within a certain time granularity.

[0344] For example, if the second information includes the first threshold, then steps 8 and 9 can be combined into: the NMS sends the second information (i.e., activation information for performing model inference) to the EMS, and the second information includes the first threshold.

[0345] In step 10, EMS sends a model inference report to NMS.

[0346] For example, the inference report may include performance metrics of the first model during the model inference process. For instance, the inference report may include one or more of the following: the model's reward value during the model inference process, the model's reward value within a certain time granularity, the relationship between the reward value and a first threshold (the reward value is greater than, less than, or equal to the first threshold), or a reward value less than the first threshold.

[0347] In this embodiment of the application, the NMS can query the reinforcement learning capabilities of the EMS and configure reinforcement learning training requests according to the reinforcement learning capabilities of the EMS and the required functions. This helps to improve the feasibility and reliability of reinforcement learning model training and enables reinforcement learning functions within the 3GPP management domain.

[0348] Scenario 2: Co-deployment of training and inference

[0349] Figure 8 This is a flowchart illustrating another communication method provided in an embodiment of this application. Figure 8 The method shown can involve interaction between a first device and a second device, belonging to a scenario where the model training provider and the model training user are co-deployed. Here, the second device is an NMS, and the first device is an EMS.

[0350] Figure 8 The method shown is the same as Figure 7 The main difference between the methods shown is: Figure 7 The method shown is to encapsulate reinforcement learning capabilities as a function. Figure 8 The method shown encapsulates reinforcement learning capabilities as a function.

[0351] Figure 8 The method shown may include steps 1 through 9.

[0352] In step 1, the NMS sends a capability query message to the EMS. This message is used to query capability information related to the training of the reinforcement learning model.

[0353] In step 2, EMS sends capability information to NMS.

[0354] For example, the capability information may include reinforcement learning functions supported by EMS, such as mobile load balancing and coverage optimization.

[0355] For example, capability information may include available training environments A supported by EMS, such as NDT or available live network environments.

[0356] In step 3, NMS receives new functional requirement information.

[0357] The function mentioned here refers to the model's functionality. In other words, NMS receives new model training requirements, which include the model's functionalities. Optionally, this functional requirement information may also include a second list of metrics corresponding to that function, where the metrics are expected to be used to determine the model training environment.

[0358] In step 4, NMS sends a model training request to EMS.

[0359] For example, the model is trained using a reinforcement learning method.

[0360] For example, the model training request may include the first information mentioned above, such as the functionality of the first model, the available training environment B (a subset of the available training environment A), and the performance metrics of the first model.

[0361] Optionally, EMS can determine third information based on the functionality of the first model. The third information includes one or more of the following: one or more metrics for determining the state of the first environment; one or more parameters for determining the behavior of the first model; the first environment; the behavior of the first model; state information of the first environment; behavior information of the first model; or the training environment of the first model.

[0362] In step 5, EMS performs model training of the first model based on the model training request.

[0363] For example, EMS determines the third information based on the first information in the model training request, and then performs model training of the first model based on the third information.

[0364] In step 6, EMS reports the model training report to NMS.

[0365] For example, the training report may include the training environment used for model training (a subset of available training environment B or available training environment B), such as the available live network environment or NDT.

[0366] For example, the training report may include performance metrics for the first model. Performance metrics for the first model may include, for example, a reward metric and / or a reward function. As an example, the performance metrics for the first model may include one or more of the following: single reward value, cumulative reward value, or behavior-environment-reward value interaction records. Optionally, the training report may include behavior-environment-reward value interaction records from the last few rounds.

[0367] In step 7, the NMS sends the threshold for the acceptable reward value to the EMS, which is the first threshold mentioned above.

[0368] The first threshold could be, for example, a threshold for the acceptable reward value when the agent or the ML model comprising the agent performs an action. Exemplarily, the NMS can configure the first threshold based on performance metrics of the first model obtained in step 6, such as the behavior-environment-reward interaction log. For example, the first threshold could be the average of the reward values ​​obtained in the last few rounds, or a certain percentage of that average, such as 90% of the average.

[0369] In step 8, the NMS sends a second message to the EMS, namely the activation message for performing model inference.

[0370] If the second information is received, the EMS can perform model inference of the first model based on the first threshold.

[0371] For example, during model inference, performance metrics such as whether the reward value exceeds a first threshold can be determined. Optionally, it can be determined whether the reward value exceeds the threshold within a certain time granularity.

[0372] For example, if the second information includes the first threshold, then steps 7 and 8 can be combined into: the NMS sends the second information (i.e., the activation information for performing model inference) to the EMS, and the second information includes the first threshold.

[0373] In step 9, EMS sends a model inference report to NMS.

[0374] For example, the inference report may include performance metrics of the first model during the model inference process. For instance, the inference report may include one or more of the following: the model's reward value during the model inference process, the model's reward value within a certain time granularity, the relationship between the reward value and a first threshold (the reward value is greater than, less than, or equal to the first threshold), or a reward value less than the first threshold.

[0375] In this embodiment of the application, by encapsulating reinforcement learning capabilities as a function, the second device does not need to care about the underlying information in the reinforcement learning model training process, which helps to reduce the processing complexity of the second device and reduce transmission overhead.

[0376] Scenarios where training and inference are deployed separately (ORAN architecture)

[0377] Figure 9 This is a flowchart illustrating another communication method provided in an embodiment of this application. Figure 9 The method shown can involve interactions between a second device and a first and third device, belonging to a scenario where the model training provider and the model training user are deployed separately. Here, the second device is an SMO, the first device is an MLTF, and the third device is a NE.

[0378] Figure 9 The method shown may include steps 1 through 10.

[0379] In step 1, the SMO sends a capability query message to the MLTF. This message is used to query capability information related to the training of the reinforcement learning model.

[0380] In step 2, the MLTF sends capability information to the SMO.

[0381] For example, capability information may include a list of adjustable KPIs, i.e., the third indicator list mentioned above. Optionally, capability information may also include a list of adjustable parameters, i.e., the second parameter list mentioned above. In this case, capability information may also include the correspondence between the indicators in the third indicator list and the parameters in the second parameter list.

[0382] For example, an adjustable KPI list can include wireless network performance metrics such as RSRP, PRB utilization, and handover success rate, which can be used to determine the environment for reinforcement learning. Adjustable parameters can be wireless network parameters that affect the aforementioned performance metrics, such as antenna settings and cell settings, which can be used to select reinforcement learning behaviors.

[0383] For example, capability information may include available training environments C supported by MLTF, such as NDT or available live network environments.

[0384] In step 3, the SMO receives new functional requirement information.

[0385] The function mentioned here refers to the model's functionality. In other words, the SMO receives new model training requirements, which include the model's functionalities. Optionally, this functional requirement information may also include a second list of metrics corresponding to that function, where the metrics are expected to be used to determine the model training environment.

[0386] In step 4, SMO determines the first parameter list and the first index list of MLTF.

[0387] For example, the SMO can determine the first list of MLTF indicators based on the third list of indicators in step 2 (i.e., carried in the capability information) and the second list of indicators in step 3 (i.e., carried in the functional requirements information).

[0388] If the third indicator list in step 2 includes all the indicators in the second indicator list in step 3, then the first indicator list of MLTF can be the second indicator list in step 3.

[0389] If the third indicator list in step 2 includes some of the indicators in the second indicator list in step 3, then the first indicator list of MLTF may include the aforementioned partial indicators.

[0390] For example, the SMO can determine the first parameter list and the first indicator list based on the correspondence between the parameters in the second parameter list in step 2 and the indicators in the third indicator list. For instance, the first parameter list of the MLTF may include parameters that correspond to the indicators in the first indicator list of the MLTF.

[0391] MLTF can train a model using reinforcement learning methods based on a first parameter list and / or a first metric list. For example, the metrics in the first metric list are indicators used to determine the state of the environment for model training; the parameters in the first parameter list are parameters used to select the behavior for model training.

[0392] When an SMO manages the training of multiple MLTF models, the SMO can determine the first parameter list and the first metric list for each MLTF separately.

[0393] In step 5, SMO sends a model training request to MLTF.

[0394] For example, the model is trained using a reinforcement learning method.

[0395] For example, the model training request may include the first information mentioned above, such as the functionality of the first model, the available training environment D (i.e., the first training environment mentioned above, which is a subset of the available training environment C), the first metric list, the first parameter list, and the performance metrics of the first model.

[0396] Among them, the available training environment can support the construction of the environment during the training process of the reinforcement learning model. For example, the available training environment can be identified as NDT or available live network environment.

[0397] The metrics in the first metric list can be used to determine the state of the environment in which the model is trained, and / or state information of that environment.

[0398] The parameters in the first parameter list can be used to determine the behavior of the first model, and / or the first parameter list can be used to determine the behavioral information of the first model.

[0399] The first parameter list is optional. If the training request does not include the first parameter list, then the capability information reported in step 2 may not include the second parameter list.

[0400] In step 6, MLTF performs model training of the first model based on the model training request, such as the first information in the model training request.

[0401] In step 7, MLTF reports the model training report to SMO.

[0402] For example, the training report may include the training environment used for model training, such as the available live network environment or NDT.

[0403] For example, the training report may include performance metrics for the first model. Performance metrics for the first model may include, for example, a reward metric and / or a reward function. As an example, the performance metrics for the first model may include one or more of the following: single reward value, cumulative reward value, or behavior-environment-reward value interaction records. Optionally, the training report may include behavior-environment-reward value interaction records from the last few rounds.

[0404] In step 8, the SMO sends the threshold of acceptable performance metrics to the NE via the MLTF, namely the first threshold mentioned above.

[0405] For example, the SMO can send a first threshold to the MLTF, and the MLTF can send the first threshold to the NE.

[0406] The first threshold could be, for example, a threshold for the acceptable reward value when the agent or the ML model comprising the agent performs an action. Exemplarily, the NMS can configure the first threshold based on performance metrics of the first model obtained in step 6, such as the action-environment-reward interaction log. For example, the first threshold could be the average of the reward values ​​obtained in the last few rounds, or a certain percentage of that average, such as 80% of the average.

[0407] In step 9, the SMO sends the second information to the NE via the MLTF, namely the activation information for performing model inference.

[0408] For example, the SMO can send a second message to the MLTF, and the MLTF can send the second message to the NE.

[0409] If the second information is received, the NE can perform model inference of the first model based on the first threshold.

[0410] For example, during model inference, performance metrics such as whether the reward value exceeds a first threshold can be determined. Optionally, it can be determined whether the reward value exceeds the threshold within a certain time granularity.

[0411] For example, if the second information includes the first threshold, then steps 8 and 9 can be combined into: the SMO sends the second information (i.e., activation information for performing model inference) to the NE via the MLTF, and the second information includes the first threshold.

[0412] In step 10, NE sends a model inference report to SMO.

[0413] For example, the NE sends a model inference report to the MLTF, and the MLTF can then send the model inference report to the SMO.

[0414] For example, the inference report may include performance metrics of the first model during the model inference process. For instance, the inference report may include one or more of the following: the model's reward value during the model inference process, the model's reward value within a certain time granularity, the relationship between the reward value and a first threshold (the reward value is greater than, less than, or equal to the first threshold), or a reward value less than the first threshold.

[0415] In this embodiment, the interface between SMO and MLTF adds reinforcement learning capability query and reporting functions; the training request supports reinforcement learning information configuration, including available training environments, metric lists, etc.; the training report adds an indication of the training environment used, which helps enable reinforcement learning functions in the ORAN architecture. In addition, performance metric requirements are added to the NE reinforcement learning model inference configuration, which helps improve the reliability of model inference.

[0416] It should be understood that Figures 7 to 9 The methods shown are merely specific examples in a particular scenario and should not be construed as limiting this application.

[0417] The method embodiments provided in this application have been described above. The apparatus embodiments provided in this application will be described below. It should be understood that the description of the apparatus embodiments corresponds to the description of the method embodiments. Therefore, any content not described in detail can be referred to the method embodiments above. For the sake of brevity, it will not be repeated here.

[0418] Figure 10 This is a schematic block diagram of a communication device provided in an embodiment of this application. Figure 10As shown, the communication device 1000 may include a transceiver unit 1010 and / or a processing unit 1020. The transceiver unit 1010 can implement corresponding communication functions, and the processing unit 1020 is used for data processing. The transceiver unit 1010 may also be referred to as a communication interface or communication unit. Optionally, the device 1000 may further include a storage unit, which can be used to store instructions and / or data. The processing unit 1020 can read the instructions and / or data in the storage unit to enable the device to implement the aforementioned method embodiments.

[0419] In one possible design, the device 1000 may be a second device in the above method embodiments, or it may be a chip, processor, or chip system that implements the functions of the second device.

[0420] Specifically, the processing unit 1020 can be used to determine first information, which is used to train a first model, and the training method of the first model is reinforcement learning. The transceiver unit 1010 can be used to send the first information to a first device, which is the device used to train the first model.

[0421] In some embodiments, during model training, the first model interacts with a first environment, and the first information includes one or more of the following: a first metric list, the state of the first environment being determined by one or more metrics in the first metric list; a first parameter list, the behavior of the first model being determined by one or more parameters in the first parameter list; a first training environment, the training environment of the first model being the first training environment, or the training environment of the first model being one of the first training environments; the function of the first model; the state information of the first environment; the behavior information of the first model; or the performance metrics of the first model.

[0422] In some embodiments, the first information includes the first indicator list, and determining the first information includes: determining the first information based on model training requirement information, wherein the model training requirement information includes a second indicator list, and the indicators in the second indicator list are indicators expected to be used to determine the first environment; wherein the first indicator list is the second indicator list.

[0423] In some embodiments, the capability information of the first device includes a third indicator list, which includes indicators that the first device supports using when determining the interaction environment of the model. Determining the first information based on the model training requirement information includes: determining the first indicator list based on the model training requirement information and the capability information of the first device, wherein: if the third indicator list includes the second indicator list, then the first indicator list is the second indicator list; if the third indicator list includes some indicators in the second indicator list, then the first indicator list includes indicators common to both the second and third indicator lists.

[0424] In some embodiments, the first information includes the functionality of the first model, which is determined based on the requirements for model training.

[0425] In some embodiments, the first information includes the first parameter list, and the capability information of the first device includes the correspondence between indicators in the third indicator list and parameters in the second parameter list. The third indicator list includes indicators supported by the first device when determining the interaction environment of the model, and the second parameter list includes parameters supported by the first device when determining the behavior of the model. Determining the first information includes: the first parameter list is determined based on the indicators in the first indicator list and the correspondence, wherein the first parameter list includes parameters that have the correspondence with the indicators in the first indicator list.

[0426] In some embodiments, the processing unit 1020 may be used to: acquire capability information of the first device, the capability information of the first device being related to reinforcement learning, wherein the capability information of the first device includes one or more of the following: a first function, wherein the first device supports training of a model that implements the first function; a second environment, wherein the first device supports an environment for training the model; a third index list, wherein the third index list includes indices that the first device supports using when determining the interaction environment of the model; a second parameter list, wherein the second parameter list includes parameters that the first device supports when determining the behavior of the model; or a correspondence between indices in the third index list and parameters in the second parameter list.

[0427] In some embodiments, sending the first information to the first device includes: sending a first request to the first device, the first request being used to request training of the first model; wherein the first request includes a first attribute, the first attribute being used to indicate the first information.

[0428] In some embodiments, the transceiver unit 1010 may be used to: receive a model training report, the model training report including a second attribute and / or a third attribute, the second attribute being used to indicate the training environment of the first model, and the third attribute being used to indicate the performance metrics of the first model.

[0429] In some embodiments, the transceiver unit 1010 may be used to: send a first threshold to a first device, the first threshold being used to indicate an acceptable performance metric of the first model during model inference.

[0430] In some embodiments, the transceiver unit 1010 may be used to: send second information to a first device, the second information being used to instruct the execution of model inference based on the first model; and receive a model inference report, the model inference report including performance metrics of the first model during the model inference process.

[0431] In one possible design, the device 1000 may be the first device in the above method embodiments, or it may be a chip, processor, or chip system that implements the functions of the first device.

[0432] Specifically, the transceiver unit 1010 can be used to receive first information, which is used to train a first model, and the training method of the first model is reinforcement learning. The processing unit 1020 can be used to train the first model based on the first information.

[0433] In some embodiments, during model training, the first model interacts with a first environment, and the first information includes one or more of the following: a first metric list, the state of the first environment being determined by one or more metrics in the first metric list; a first parameter list, the behavior of the first model being determined by one or more parameters in the first parameter list; a first training environment, the training environment of the first model being the first training environment, or the training environment of the first model being one of the first training environments; the function of the first model; the state information of the first environment; the behavior information of the first model; or the performance metrics of the first model.

[0434] In some embodiments, the first information includes the functionality of the first model, and training the first model based on the first information includes: determining third information based on the functionality of the first model; and training the first model based on the third information, wherein the third information includes one or more of the following: one or more metrics for determining the state of the first environment; one or more parameters for determining the behavior of the first model; the first environment; the behavior of the first model; state information of the first environment; behavioral information of the first model; or the training environment of the first model.

[0435] In some embodiments, the transceiver unit 1010 may be used to: notify the first device of capability information, the capability information of the first device being related to reinforcement learning, wherein the capability information of the first device includes one or more of the following: a first function, wherein the first device supports training a model that implements the first function; a second environment, wherein the first device supports an environment for training the model; a third index list, wherein the third index list includes indices that the first device supports using when determining the interaction environment of the model; a second parameter list, wherein the second parameter list includes parameters that the first device supports when determining the behavior of the model; or a correspondence between indices in the third index list and parameters in the second parameter list.

[0436] In some embodiments, receiving the first information includes: receiving a first request, the first request being used to request training of the first model; wherein the first request includes a first attribute, the first attribute being used to indicate the first information.

[0437] In some embodiments, the transceiver unit 1010 may be used to: send a model training report to a second device, the model training report including a second attribute and / or a third attribute, the second attribute being used to indicate the training environment of the first model, and the third attribute being used to indicate the performance metrics of the first model.

[0438] In some embodiments, the transceiver unit 1010 may be used to: receive a first threshold, the first threshold being used to indicate an acceptable performance metric of the first model during model inference.

[0439] In some embodiments, the transceiver unit 1010 may be used to: receive second information, the second information being used to instruct the execution of model inference based on the first model; and send a model inference report to a second device, the model inference report including performance metrics of the first model during the model inference process.

[0440] It should be understood that the division of units in the above-described device is merely a logical functional division. One function can correspond to one functional unit, or two or more functions can be integrated into one functional unit. In actual implementation, all or some units can be integrated onto a single physical entity, or distributed across different physical entities. Furthermore, the aforementioned functional units can be implemented in hardware, software, or a combination of both. Whether a function is executed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for specific applications, but such implementations should not be considered beyond the scope of this application.

[0441] It should be understood that the "unit" in device 1000 can be implemented in hardware, software, or by hardware executing corresponding software. For example, the "unit" can refer to an application-specific integrated circuit (ASIC), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor, etc.) and memory for executing one or more software or firmware programs, combined logic circuitry, and / or other suitable components supporting the described functions. As another example, transceiver unit 1010 can be replaced by transceiver circuitry (e.g., may include receiving and transmitting circuitry), and processing unit 1020 can be replaced by a processor or processing circuitry.

[0442] Figure 11 A schematic block diagram of another communication device provided in an embodiment of this application is shown. This communication device 1100 may be a first device or a second device, or it may be a chip, chip system, or processor, etc., in the first or second device that implements the above-described methods. This device can be used to implement the methods described in the above-described method embodiments; for details, please refer to the descriptions in the above-described method embodiments.

[0443] The communication device 1100 may include one or more processors 1110, which may also be referred to as processing units, and can implement certain control functions. The processor 1110 may be a general-purpose processor or a dedicated processor, such as a baseband processor or a central processing unit. The baseband processor can be used to process communication protocols and communication data, while the central processing unit can be used to control the communication device, execute software programs, and process data from the software programs.

[0444] In an alternative design, the processor 1110 may also store instructions and / or data that can be executed by the processor 1110 to cause the communication device 1100 to perform the methods described in the above method embodiments.

[0445] In another alternative design, the communication device 1100 may include a communication interface 1120 for implementing receiving and transmitting functions. For example, the communication interface 1120 may be a transceiver circuit, interface, interface circuit, or transceiver. The transceiver circuit, interface, interface circuit, or transceiver for implementing receiving and transmitting functions may be separate or integrated. The aforementioned transceiver circuit, interface, interface circuit, or transceiver may be used for reading and writing code / data, or it may be used for transmitting or relaying signals.

[0446] Optionally, the communication device 1100 may include one or more memories 1130, which may store instructions that can be executed on the processor 1110, causing the communication device 1100 to perform the methods described in the above method embodiments. Optionally, the memories 1130 may also store data. Optionally, the processor 1110 may also store instructions and / or data. The processor 1110 and the memories 1130 may be provided separately or integrated together.

[0447] It should be understood that, in one possible design, the steps in the method embodiments provided in this application can be implemented by integrated logic circuits in the processor's hardware or by instructions in software form. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are not provided here.

[0448] It should be noted that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuitry in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0449] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0450] The above-described device and method embodiments are completely corresponding, with corresponding modules or units performing corresponding steps. For example, a communication unit or communication interface performs the receiving or sending steps in the method embodiment, while other steps besides sending and receiving can be performed by a processing unit or processor.

[0451] This application also provides a computer program product, including computer program instructions that, when executed, cause the various steps or processes performed by the first or second device in any of the above method embodiments to be executed.

[0452] This application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed, cause the various steps or processes performed by the first or second device in any of the above method embodiments to be executed.

[0453] This application also provides a chip, including a processor, for calling and running a computer program or instructions from memory, which, when executed, causes the various steps or processes performed by the first or second device in any of the above method embodiments to be executed.

[0454] This application also provides a communication system that includes at least one of a first device or a second device.

[0455] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0456] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0457] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0458] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0459] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A communication method, characterized in that, include: First information is determined, and the first information is used to train a first model. The training method of the first model is reinforcement learning. Send first information to a first device, which is the device used to train the first model.

2. The method according to claim 1, characterized in that, During model training, the first model interacts with the first environment, and the first information includes one or more of the following: The state of the first environment is determined by one or more indicators in the first indicator list. The first parameter list, wherein the behavior of the first model is determined by one or more parameters in the first parameter list; The first training environment is either the first training environment itself or the training environment of the first model is one of the first training environments. The function of the first model; The status information of the first environment; Behavioral information of the first model; or The performance metrics of the first model.

3. The method according to claim 2, characterized in that, The first information includes the first indicator list, and determining the first information includes: The first information is determined based on the model training requirement information, which includes a second indicator list, wherein the indicators in the second indicator list are the indicators expected to be used to determine the first environment. The first indicator list is the second indicator list.

4. The method according to claim 3, characterized in that, The capability information of the first device includes a third indicator list, which includes indicators that the first device supports using when determining the interaction environment of the model. The requirement information based on model training determines the first information, including: The first index list is determined based on the training requirements of the model and the capability information of the first device, wherein: If the third indicator list includes the second indicator list, then the first indicator list is the second indicator list; If the third indicator list includes some of the indicators in the second indicator list, then the first indicator list includes the indicators that are common to both the second indicator list and the third indicator list.

5. The method according to any one of claims 2-4, characterized in that, The first information indicates the function of the first model, and the function of the first model is determined based on the requirements information for model training.

6. The method according to any one of claims 2-5, characterized in that, The first information includes the first parameter list, and the capability information of the first device includes the correspondence between the indicators in the third indicator list and the parameters in the second parameter list. The third indicator list includes the indicators supported by the first device when determining the interaction environment of the model, and the second parameter list includes the parameters supported by the first device when determining the behavior of the model. The determination of the first information includes: The first parameter list is determined based on the indicators in the first indicator list and the corresponding relationship, wherein the first parameter list includes parameters that have the corresponding relationship with the indicators in the first indicator list.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: Obtain the capability information of the first device, which is related to reinforcement learning, wherein the capability information of the first device includes one or more of the following: First function: The first device supports the training of a model that implements the first function; The second environment is the environment supported by the first device, and the second environment is used for model training; A third indicator list, which includes the indicators that the first device supports using when determining the interaction environment of the model; A second parameter list, comprising parameters supported by the first device when determining the behavior of the model; or The correspondence between the indicators in the third indicator list and the parameters in the second parameter list.

8. The method according to any one of claims 1-7, characterized in that, Sending the first information to the first device includes: Send a first request to the first device, the first request being used to request training of the first model; The first request includes a first attribute, which is used to indicate the first information.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: Receive a model training report, which includes a second attribute and / or a third attribute, wherein the second attribute indicates the training environment of the first model and the third attribute indicates the performance metrics of the first model.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: A first threshold is sent to the first device, the first threshold being used to indicate an acceptable performance metric for the first model during model inference.

11. The method according to any one of claims 1-9, characterized in that, The method further includes: Send a second message to the first device, the second message being used to instruct the execution of model inference based on the first model; Receive a model inference report, which includes the performance metrics of the first model during the model inference process.

12. A communication method, characterized in that, include: Receive first information, the first information is used to train a first model, and the training method of the first model is reinforcement learning; Based on the first information, the first model is trained.

13. The method according to claim 12, characterized in that, During model training, the first model interacts with the first environment, and the first information includes one or more of the following: The state of the first environment is determined by one or more indicators in the first indicator list. The first parameter list, wherein the behavior of the first model is determined by one or more parameters in the first parameter list; The first training environment is either the first training environment itself or the training environment of the first model is one of the first training environments. The function of the first model; The status information of the first environment; Behavioral information of the first model; or The performance metrics of the first model.

14. The method according to claim 13, characterized in that, The first information includes the functionality of the first model, and training the first model based on the first information includes: Based on the functionality of the first model, determine the third information; The first model is trained based on the third information, wherein the third information includes one or more of the following: One or more indicators used to determine the state of the first environment; One or more parameters used to determine the behavior of the first model; The first environment; The behavior of the first model; The status information of the first environment; Behavioral information of the first model; or The training environment of the first model.

15. The method according to any one of claims 12-14, characterized in that, The method further includes: The first device's capability information is notified, and this capability information is related to reinforcement learning. The first device's capability information includes one or more of the following: First function: The first device supports the training of a model that implements the first function; The second environment is the environment supported by the first device, and the second environment is used for model training; A third indicator list, which includes the indicators that the first device supports using when determining the interaction environment of the model; A second parameter list, comprising parameters supported by the first device when determining the behavior of the model; or The correspondence between the indicators in the third indicator list and the parameters in the second parameter list.

16. The method according to any one of claims 12-15, characterized in that, The receiving of the first information includes: Receive a first request, the first request being used to request training of the first model; The first request includes a first attribute, which is used to indicate the first information.

17. The method according to any one of claims 12-16, characterized in that, The method further includes: A model training report is sent to a second device. The model training report includes a second attribute and / or a third attribute, wherein the second attribute indicates the training environment of the first model and the third attribute indicates the performance metrics of the first model.

18. The method according to any one of claims 12-17, characterized in that, The method further includes: A first threshold is received, which is used to indicate an acceptable performance metric for the first model during model inference.

19. The method according to any one of claims 12-18, characterized in that, The method further includes: Receive second information, which is used to instruct the execution of model inference based on the first model; A model inference report is sent to the second device, the model inference report including the performance indicators of the first model during the model inference process.

20. A communication device, characterized in that, Includes units for performing the method as described in any one of claims 1-11.

21. A communication device, characterized in that, Includes units for performing the method as described in any one of claims 12-19.

22. A communication device, characterized in that, Includes a processor, which, when executing a program or instructions, causes the method as described in any one of claims 1-11 to be performed, or causes the method as described in any one of claims 12-19 to be performed.

23. A readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed, they cause the method as described in any one of claims 1-11 to be performed, or cause the method as described in any one of claims 12-19 to be performed.

24. A computer program product, characterized in that, It includes computer program instructions, which, when executed, cause the method as described in any one of claims 1-11 to be performed, or cause the method as described in any one of claims 12-19 to be performed.

25. A chip, characterized in that, Includes a processor for calling and running a computer program from memory, such that the method as claimed in any one of claims 1-11 is executed, or such that the method as claimed in any one of claims 12-19 is executed.

Images