Ai / ML verification method and device in wireless communication system

Abstract

Disclosed is an AI / ML verification method performed by a first communication node including a communication unit and a processor. The disclosed method comprises the steps of: transmitting, to a second communication node, life cycle management (LCM) configuration information for AI / ML full cycle management; in the case of an AI / ML verification environment, applying an LCM lock mode by locking a portion of LCM functions for to-be-verified AI / ML; and verifying the to-be-verified AI / ML in the LCM lock mode.

Description

AI / ML verification method and device in wireless communication systems

[0001] The present disclosure relates to an AI / ML verification method in a wireless communication system, and more specifically, to an AI / ML management method capable of verifying / testing functions, performance, etc. of AI / ML in a wireless communication system supporting AI / ML.

[0002] A communication system may include a core network, base stations (e.g., macro base stations, small base stations, relays, etc.), terminals, etc. Communication between the base stations and terminals may be performed based on various radio access technologies (RATs) (e.g., 4G communication technology, 5G communication technology, 6G communication technology, WiBro (wireless broadband) technology, WLAN (wireless local area network) technology, WPAN (wireless personal area network) technology, etc.).

[0003] 3GPP selected AI / ML as a Study Item in 5G Release 18 and conducted research and discussions. AI / ML technologies are being used to streamline network operations and optimize service quality by leveraging artificial intelligence and machine learning. 3GPP is prioritizing the adoption of AI / ML in scenarios such as 5G beam management, positioning, channel state information prediction, and channel state information compression, expecting these improvements to system throughput, resource utilization, and coverage.

[0004] Furthermore, 3GPP has begun full-scale 6G research in Release 20, considering AI / ML as a core part of 6G's AI-native architecture. Building on the AI / ML framework developed in 5G, the company is researching data management, model deployment, and air interface optimization (e.g., CSI compression, mobility enhancement). Its goal is to enable autonomous AI operation, maximize network efficiency, and support new use cases in 6G systems.

[0005] Standardization organizations such as GPP and O-RAN Alliance are defining a life cycle management (LCM) system for AI / ML functions to apply AI / ML to various network functions and operate them efficiently.

[0006] LCM encompasses a series of management procedures, including deployment, enabling, performance monitoring, disabling, and removal of AI / ML models. If the performance of an AI / ML model falls below a predefined threshold, LCM also includes the ability to replace the model with existing logic-based functions, switch to a different AI / ML model with better performance, or fallback to a legacy model that does not utilize AI / ML.

[0007] AI / ML models can be modified or updated even after deployment. Furthermore, new AI / ML models may be deployed. There is no guarantee that the performance of the device will improve after deployment or update. When applying new or updated AI / ML models to communication nodes, it is essential to conduct AI / ML verification and testing.

[0008] According to at least one embodiment, an AI / ML management method and device supporting verification and / or testing of AI / ML are disclosed.

[0009] In one aspect, an AI / ML verification method performed by a first communication node including a communication unit and a processor is disclosed.

[0010] The disclosed method includes a step of transmitting LCM (Life Cycle Management) configuration information for AI / ML life cycle management to a second communication node; a step of applying an LCM lock mode by locking some of the LCM functions for an AI / ML to be verified in the case of an AI / ML verification environment; and a step of performing verification on the AI / ML to be verified in the LCM lock mode.

[0011] According to at least one embodiment, LCM locking is utilized during the AI / ML verification process to prevent modification, deactivation, or conversion to non-AI / ML before the AI / ML verification is completed. Furthermore, sufficient data on training loss and validation loss for each epoch can be secured in the verification environment to ensure the performance of AI / ML in a mobile communication system. Furthermore, even if performance degradation occurs during the AI / ML verification process, the verification subject can easily determine whether the degradation is due to an inherent problem with the AI / ML model (or a model or training data associated with a function or associated ID) or due to underfitting or overfitting.

[0012] According to at least one embodiment, the LCM operational threshold applied in the AI / ML verification environment and the general operation environment can be set differently. For example, during the verification period, the performance evaluation threshold can be set higher than the threshold used in general operation (so that the threshold corresponds to a higher performance standard, thereby increasing the frequency of situations where the threshold condition is satisfied). This allows for more precise and accurate performance verification of AI / ML performance. While this may result in a slightly increased reporting frequency, it allows for the precise observation of even temporary deterioration of actual AI / ML performance or subtle performance changes.

[0013] Conversely, if you want to reduce the signaling overhead during the verification phase, you can set the threshold to a lower level than in a general environment (so that the threshold corresponds to a lower performance criterion, thereby reducing the frequency with which the second communication node independently evaluates the AI / ML performance status and reports it to the network (the first communication node). This method can effectively increase the verification efficiency of AI / ML models in environments where performance fluctuations are frequent or excessive signaling is a concern. In other words, in the verification environment, the threshold for AI / ML performance evaluation can be set independently from the threshold(s, single or multiple) in the general operating environment, enabling adaptive and flexible LCM management according to various AI / ML operating conditions and environments required by the mobile communication system. This effect provides a technical advantage of simultaneously improving the reliability and performance management efficiency of AI / ML-based mobile communication systems.

[0014] FIG. 1 is a diagram illustrating a wireless communication system according to an exemplary embodiment.

[0015] Figure 2 is a block diagram exemplarily showing the configuration of a communication node constituting a communication system.

[0016] Figure 3 is a diagram illustrating an example of a 5G network architecture.

[0017] Figure 4 is a diagram showing the 6G network architecture.

[0018] Figure 5 is a block diagram showing the entire cycle of AI / ML functions in a mobile communication system.

[0019] Figure 6 is a diagram showing the prediction error according to learning of the AI / ML model.

[0020] Figure 7 illustrates an AI / ML verification environment performed by a first communication node and a second communication node.

[0021] FIG. 8 is a diagram showing the full cycle management (LCM) of each of the first communication node and the second communication node shown in FIG. 7.

[0022] Figure 9 illustrates an AI / ML verification environment performed by a first communication node and a second communication node.

[0023] FIG. 10 is a diagram showing the full cycle management (LCM) of each of the first communication node and the second communication node shown in FIG. 9.

[0024] Fig. 11 is a flowchart illustrating an AI / ML verification method according to an exemplary embodiment.

[0025] Figure 12 is a conceptual diagram illustrating an example of LCM locking based on AI / ML model ID.

[0026] Figure 13 is a diagram illustrating an example of LCM locking based on AI / ML functionality.

[0027] Figure 14 is a diagram illustrating an example of an LCM lock based on an association ID.

[0028] Fig. 15 is a flowchart illustrating a verification method of AI / ML according to an exemplary embodiment.

[0029] Fig. 16 is a flowchart illustrating a verification method of AI / ML according to an exemplary embodiment.

[0030] Figure 17 is a flowchart illustrating an AI / ML verification method according to an exemplary embodiment.

[0031] Fig. 18 is a flowchart illustrating an AI / ML verification method according to an exemplary embodiment.

[0032] Fig. 19 is a flowchart illustrating an AI / ML verification method according to an exemplary embodiment.

[0033] In one aspect, an AI / ML verification method performed by a first communication node including a communication unit and a processor is disclosed.

[0034] The disclosed method includes a step of transmitting LCM (Life Cycle Management) configuration information for AI / ML life cycle management to a second communication node; a step of applying an LCM lock mode by locking some of the LCM functions for an AI / ML to be verified in the case of an AI / ML verification environment; and a step of performing verification on the AI / ML to be verified in the LCM lock mode.

[0035] In the above LCM lock mode, at least one of the LCM functions, including the AI / ML deactivation function, the AI / ML switch function, and the non-AI / ML conversion function, may be locked.

[0036] In the above LCM lock mode, all LCM functions except for the AI / ML deactivation function, AI / ML switch function, and non-AI / ML conversion function can operate normally.

[0037] The above verification target AI / ML can be determined based on at least one AI / ML model ID.

[0038] The above verification target AI / ML can be determined based on at least one AI / ML function.

[0039] The above verification target AI / ML can be determined based on at least one associated ID.

[0040] The step of performing verification on the AI / ML to be verified in the LCM lock mode may include the step of obtaining performance information from the second communication node that the second communication node tested the inference performance using the AI / ML to be verified; and the step of transmitting management instructions for the AI / ML to be verified based on the performance information.

[0041] The above performance information includes a learning history for the AI / ML to be verified, and the learning history includes analysis information on learning loss and prediction loss of the AI / ML to be verified for each epoch, and the management instructions may include instruction information on how many epochs learning will be performed for the AI / ML to be verified.

[0042] An AI / ML verification method performed by a second communication node including a communication unit and a processor is disclosed. The disclosed method includes the steps of: receiving LCM configuration information for AI / ML full-cycle management from a first communication node; applying an LCM lock mode by locking a portion of LCM functions for an AI / ML to be verified, in the case of an AI / ML verification environment; and performing verification on the AI / ML to be verified in the LCM lock mode.

[0043] The above LCM configuration information may include an LCM lock instruction including information of the AI / M to be verified.

[0044] In the above LCM lock mode, at least one of the LCM functions, including the AI / ML deactivation function, the AI / ML switch function, and the non-AI / ML conversion function, may be locked.

[0045] The above verification target AI / ML can be determined based on at least one AI / ML model ID.

[0046] The above verification target AI / ML can be determined based on at least one AI / ML function.

[0047] The above verification target AI / ML can be determined based on at least one associated ID.

[0048] The step of performing verification on the AI / ML to be verified in the LCM lock mode may include the steps of: acquiring a learning history for the AI / ML to be verified; analyzing the verification loss and learning loss for each epoch in the learning history; and transmitting a full-cycle management request to the first communication node based on the analysis results for the verification loss and learning loss for each epoch.

[0049] The step of performing verification on the AI / ML to be verified in the above LCM lock mode may include the steps of: acquiring a learning history for the AI / ML to be verified; analyzing the verification loss and learning loss for each epoch in the learning history; and performing full-cycle management based on the analysis results for the verification loss and learning loss for each epoch, and then transmitting a full-cycle management decision report to the first communication node.

[0050] An AI / ML verification method performed by a first communication node including a communication unit and a processor is disclosed. The disclosed method comprises the steps of: generating LCM configuration information including at least one threshold value set for a verification environment when it is determined that an AI / ML verification environment is applicable; transmitting the LCM configuration information including at least one threshold value set for the verification environment to a second communication node, wherein the at least one threshold value may include a threshold value for determining a condition under which at least one of deactivation of AI / ML, replacement of AI / ML, and conversion to non-AI / ML occurs.

[0051] The thresholds used in AI / ML validation environments and in environments where AI / ML is used can be set differently. In AI / ML validation environments, thresholds for validation and thresholds used in general AI / ML performance environments can be set separately. Similarly, thresholds used for fallback can also be set separately. This means that the functionality provided by LCM Lock settings can be provided through different thresholds.

[0052] The at least one threshold value may include a threshold value for determining a condition under which LCM signaling occurs between the second communication node and the first communication node.

[0053] The at least one threshold value may be set such that the range in which the condition of the at least one threshold value is satisfied is reduced in the AI / ML verification environment compared to a general environment.

[0054] The at least one threshold value may be set such that the range in which the condition of the at least one threshold value is satisfied in the AI / ML verification environment is increased compared to a general environment.

[0055] The present invention is susceptible to various modifications and embodiments. Specific embodiments are illustrated and described in detail in the drawings. However, this is not intended to limit the present invention to specific embodiments, but rather to encompass all modifications, equivalents, and alternatives falling within the spirit and technical scope of the present invention.

[0056] While terms such as "first" and "second" may be used to describe various components, these components should not be limited by these terms. These terms are used solely to distinguish one component from another. For example, without departing from the scope of the present invention, a first component could be referred to as a "second component," and similarly, a second component could also be referred to as a "first component." The term "and / or" includes any combination of multiple related items listed or any one of multiple related items listed.

[0057] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0058] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprise" or "have" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0059] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0060] Hereinafter, with reference to the attached drawings, preferred embodiments of the present invention will be described in more detail. In order to facilitate an overall understanding in describing the present invention, identical reference numerals will be used for identical components in the drawings, and redundant descriptions of identical components will be omitted.

[0061] Table 1 shows the abbreviations used in this disclosure.

[0062] 약어전체 이름AI / MLArtificial Intelligence / Machine Learning3GPP3rd Generation Partnership ProjectACKAcknowledgementAFApplication FunctionAIArtificial IntelligenceAMFAccess and Mobility Management FunctionAUSFAuthentication Server FunctionBWPBandwidth PartC-RNTICell RNTICSIChannel State InformationCSI-RSChannel State Information Reference SignalCLICross link InterferenceCEControl ElementDCIDownlink Control InformationIEInformation elementMACMedium Access ControlMLMachine LearningNSSFNetwork Slicing Selection FunctionNEFNetwork Exposure FunctionNRFNF Repository FunctionPCFPolicy Control FunctionPDCCHPhysical Downlink Control ChannelPDSCHPhysical Downlink Shared ChannelPDUProtocol Data UnitPRACHPhysical Random Access ChannelPT-RSPhase Tracking Reference SignalPUCCHPhysical Uplink Control ChannelPUSCHPhysical Uplink Shared ChannelRARandom AccessRACHRandom Access ChannelRANRadio Access NetworkRBResource BlockRRCRadio Resource ControlRSRPReference Signal Received PowerRSRQReference Signal ReceivedQualityRSSIReceived Signal Strength IndicatorSBFDSub-Band Full DuplexSISelf InterferenceSIBSystem information blockUEUser EquipmentDLDownlinkULUplinkRARRandom Access ResponseRRMRadio Resource ManagementRRCRadio Resource ControlRSRPReference Signal Received PowerQoSQuality of ServiceDRBData Radio BearerDAPSDual Active Protocol StackTDDTime Division DuplexingFDDFrequency Division DuplexingSSBSS BlockCQIChannel Quality IndicatorPMIPrecoding Matrix IndicatorLILayer IndicatorLCMLife Cycle ManagementRIRank IndicatorQCLQuasi coLocation

[0063] Hereinafter, ‘AI / ML verification’ means a procedure for evaluating or testing the performance of AI / ML models identified by at least one of a given AI / ML model ID, an AI / ML function ID, and an associated ID.

[0064] AI / ML validation may include validation procedures conducted in a designated test environment simulating a commercial communications network and validation procedures conducted on a commercial communications network.

[0065] FIG. 1 is a diagram illustrating a wireless communication system (100) according to an exemplary embodiment.

[0066] Referring to FIG. 1, a wireless communication system (100) may be composed of a plurality of communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3, 120-4). Here, a communication node means a node capable of transmitting and receiving signals in the wireless communication system (100), and each of the plurality of communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3, 120-4) may support at least one communication protocol. For example, each of the plurality of communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3, 120-4) may support cellular communication (e.g., long term evolution (LTE), advanced LTE-A, 5G NR, 5G-Advanced, 6G, etc. as defined in the 3rd generation partnership project (3GPP) standard).

[0067] For example, each of the plurality of communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3, 120-4) can support a communication protocol based on CDMA (code division multiple access), a communication protocol based on WCDMA (wideband CDMA), a communication protocol based on TDMA (time division multiple access), a communication protocol based on FDMA (frequency division multiple access), a communication protocol based on OFDM (orthogonal frequency division multiplexing), a communication protocol based on OFDMA (orthogonal frequency division multiple access), a communication protocol based on SC (single carrier)-FDMA, a communication protocol based on NOMA (non-orthogonal multiple access), a communication protocol based on SDMA (space division multiple access), SBFD (sub-band full duplex), AI / ML, etc.

[0068] A plurality of communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3, 120-4) may include a plurality of base stations (110-1, 110-2, 110-3) and a plurality of terminals (120-1, 120-2, 120-3, 120-4). Each of the base stations (110-1, 110-2, 110-3) may form a cell. The cell may include a small cell, a macro cell, a pico cell, a femto cell, etc., but the embodiment is not limited thereto. For example, the first terminal (120-1) and the second terminal (120-2) may belong to the coverage of the first base station (110-1), the third terminal (120-3) may belong to the coverage of the second base station (110-2), and the fourth terminal (120-4) may belong to the coverage of the third base station (110-3).

[0069] The plurality of base stations (110-1, 110-2, 110-3) may be referred to as gNodeB (gNB), NodeB, evolved NodeB, base transceiver station (BTS), radio base station, radio transceiver, access point, access node, road side unit (RSU), radio remote head (RRH), transmission point (TP), transmission and reception point (TRP), relay node, etc. The plurality of base stations (110-1, 110-2, 110-3) may form a radio access network (RAN). The radio access network may be connected to a core network.

[0070] Each of the plurality of terminals (120-1, 120-2, 120-3, 120-4) may be referred to as a terminal, an access terminal, a mobile terminal, a station, a subscriber station, a mobile station, a portable subscriber station, a node, a device, etc.

[0071] The communication protocol support ranges of each of the multiple base stations (110-1, 110-2, 110-3) may differ from each other. For example, some of the multiple base stations (110-1, 110-2, 110-3) may support AI / ML functions while others may not. Similarly, the communication protocol support ranges of each of the terminals (120-1, 120-2, 120-3, 120-4) may also differ from each other. For example, some of the multiple terminals (120-1, 120-2, 120-3, 120-4) may support AI / ML functions while others may not. 3GPP defines 6G research items in Release 20 (2025-2027) and is conducting technology research including 6G air interface, advanced sensing, AI / ML-based automation, and sustainable connectivity.

[0072] Fig. 2 is a block diagram exemplarily showing the configuration of a communication node (200) constituting a communication system. At least some of the communication nodes (110-1, 110-2, 110-3, 120-1, 120-2, 120-3) shown in Fig. 1 may correspond to the communication node (200) shown in Fig. 2.

[0073] Referring to FIG. 2, a communication node (200) may include at least one processor (210), a memory (220), and a transmission / reception device (230) that is connected to a network and performs communication. In addition, the communication node (200) may further include an input interface device (240), an output interface device (250), a storage device (260), etc. Each component included in the communication node (200) may be connected by a bus (270) and communicate with each other.

[0074] The processor (210) can execute program commands stored in at least one of the memory (220) and the storage device (260). The processor (210) may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor in which methods according to embodiments of the present invention are performed. Each of the memory (220) and the storage device (260) may be configured with at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory (220) may be configured with at least one of a read-only memory (ROM) and a random access memory (RAM).

[0075] Figure 3 is an exemplary diagram illustrating the 5G network architecture. 3GPP is conducting 6G research in Release 20 (2025-2027), and the 6G network architecture plans to integrate AI-native design, advanced sensing, and ultra-low-latency communication based on the existing 5G architecture. 6G will optimize the N4 interface between the User Plane Function (UPF) and the Session Management Function (SMF) and the N3 interface between the RAN and the UPF, and strengthen AI / ML-based dynamic resource management and network slicing. Furthermore, 6G is likely to introduce new network functions (NFs) and interfaces to support industrial use cases (e.g., autonomous driving and holographic communications).

[0076] Referring to FIG. 3, the 5G network architecture may include NSSF providing a network slicing control solution, NEF providing a network function opening solution, NRF providing an NF interworking control solution within a 5G network, PCF providing a data packet flow policy control solution, UDM providing a user information and policy management solution, AMF providing a network access and mobility management solution, SMF providing a terminal / network session management solution, UPF providing a base station-to-base station user packet routing and terminal connectivity provision solution, a radio access network (RAN), and user equipment (UE).

[0077] The N1 reference point is defined to transmit signaling between the UE and the AMF, the N2 reference point is defined to connect between the RAN's access node and the AMF, and the N3 reference point is defined to connect between the RAN's access node and the UPF.

[0078] Figure 4 is a diagram showing the 6G network architecture.

[0079] Referring to Figure 4, the 6G network architecture provides AI-based end-to-end network management and automation services, as well as exposure of analytical information, and provides support functions based on cloud infrastructure, transport networks, data pipelines, and common platform functions. The 6G network architecture includes a 6G terminal (610), a 6G wireless access network (RAN) (620), and an AI-native cloud-based 6G core network (CN) (630), and can also be interconnected with existing 5G networks as needed.

[0080] Here, AI-based service and analysis information disclosure provides AI-processed network status information and analysis results to service providers or external systems, supporting improved network efficiency and service quality. Furthermore, 6G core networks operating in a cloud-native environment enable dynamic network resource management, autonomous service orchestration, and efficient user service quality assurance through network functions that incorporate artificial intelligence (AI).

[0081] This architecture strengthens connectivity with the data network, providing flexibility in data flow management and defining efficient communication paths and management interfaces between 6G terminals (610) and the data network. In particular, AI-based end-to-end (E2E) end-to-end management capabilities enable autonomous and intelligent network control across the entire network (terminal-RAN-CN-data network).

[0082]

[0083] Figure 5 is a block diagram illustrating the entire cycle of AI / ML functions in a mobile communication system. The AI / ML function blocks illustrated in Figure 6 may be implemented by at least one of the communication nodes of the mobile communication system. For example, the AI / ML function blocks illustrated in Figure 4 may be implemented by at least one of a base station and a terminal.

[0084] When the first terminal (120-1) is connected to the service area of ​​the first base station (110-1) that supports AI / ML functions, the first terminal (120-1) or the first base station (110-1) can activate at least some of the function blocks (410, 412, 414, 416, 418) of FIG. 4.

[0085] The terminal (120-1) can collect data related to a wireless channel by activating the data collection block (410). The terminal (120-1) can collect data related to wireless channel information from reference signals (CSI-RS, SSB, PT-RS, PRS, downlink reference signal, etc.). When the base station (110-1) activates the data collection block (610), data related to wireless channel information can be collected from reference signals (SRS, PRS, PT-RS) and channel status reports.

[0086] The data collected by the data collection block (410) can be used as data for learning / monitoring / inference. The model learning block (412) can train an AI / ML model using the data collected by the data collection block (410). The model storage block (418) can store the trained AI / ML model. The stored model can be used by the inference block (416). The inference block (416) can infer information necessary for operating a wireless communication system based on the data collected by the data collection block (410).

[0087] The inference block (416) can report the inference result to the AI / ML management block (414). The AI / ML management block (414) can perform overall functions related to training of the AI / ML model, selection of the AI / ML model, activation / deactivation of the AI / ML model, and control of other function blocks. The AI / ML management block (414) can monitor the performance of the AI / ML model to maintain the performance of the AI / ML and provide feedback on the training result to the model training block (412). If the performance of the monitored AI / ML model is low and retraining is required, the AI / ML management block (414) can request retraining from the model training block (412). The AI / ML management block (414) can determine activation / deactivation of the AI / ML model and model selection / change. The AI / ML management block (414) performs management instructions such as fallback to a conventional communication method that does not use the AI / ML function.

[0088] AI / ML models can be classified in various ways. For example, each AI / ML model can be assigned a model ID. Alternatively, AI / ML models can be classified by function, and each function can be assigned a functionality ID. Associated IDs can also be used to distinguish AI / ML models.

[0089] An Associated ID can indicate a configuration for data collection for training an AI / ML model. The Associated ID can be used to ensure the consistency of the AI / ML model's learning and inference environments. Terminal (120-1) can verify the configuration for data collection from the Associated ID and collect data according to the configuration. Terminal (120-1) can use the data collected according to the Associated ID to train or develop an AI / ML model.

[0090] AI / ML models have characteristics called overfitting and generalization. Overfitting occurs when a model performs well on a specific dataset and deteriorates on other datasets. Conversely, generalization refers to the ability to maintain performance across diverse datasets. There are limitations to the generalization of AI / ML models. In other words, AI / ML model performance can only be guaranteed for data collected within a relatively limited data collection configuration. Therefore, wireless communication networks can define data collection configurations that guarantee AI / ML model performance and assign Associated IDs to each data collection configuration.

[0091] A wireless communication network can define a set (range) of generalizable datasets that guarantee the performance of AI / ML models and determine a data collection configuration to match them. The wireless communication network can define various data collection configurations and assign an Associated ID to each data collection configuration. The wireless communication network can train or develop AI / ML models based on the data collection configuration corresponding to each Associated ID.

[0092] The method for assigning Associated IDs may vary depending on the use case of the AI / ML model. For example, for AI / ML models used for beam management, data collection configurations may be determined based on the same downlink spatial domain transmission filter, a reference signal in a quasi-co-location (QCL) state, or the antenna panels used for transmission / reception. A different Associated ID may be assigned to each data collection configuration. In the example of CSI compression and prediction, data collection configurations may be determined based on SNR, CQI, PMI, RI, which indicate channel environment conditions, and the antenna panels used for transmission / reception. Associated IDs are uniquely defined in the network and can be classified into global association IDs that indicate the same environment and local association IDs that are applicable only to certain cell groups / cells / sites / regions or equipment vendors. In the following, an associated ID may correspond to at least one of a global association ID and a local association ID.

[0093] Each AI / ML model can be assigned a Model ID. An Associated ID can correspond to a single Model ID. Alternatively, an Associated ID can correspond to multiple Model IDs. Associated IDs and Model IDs can have a many-to-many relationship.

[0094] For example, if the downlink spatial domain transmission filter is configured with multiple (N) numbers, the Associated ID for beam management can also be configured with multiple (N) numbers, and the AI / ML model used for the corresponding AI / ML beam management can be configured with N numbers corresponding thereto. Similarly, if the base station (110-1) configures multiple (N) sets of reference signals / resources that are not in the QCL state, the AI / ML model and associated ID used for the corresponding AI / ML beam management can be configured with N numbers corresponding thereto. On the other hand, if the AI / ML model has excellent generalization characteristics and its performance is guaranteed in all datasets, it can be configured and operated with a small number (less than N) of AI / ML models.

[0095] AI / ML model operation can be broadly categorized into standalone operation (one-side AI / ML model) and joint operation (two-side AI / ML model). In the one-side AI / ML model, the AI / ML model operates only on either the base station or the terminal. In the two-side AI / ML model, the AI / ML model can operate simultaneously on both the base station and the terminal.

[0096] AI / ML lifecycle management (LCM) can be performed in various ways. First, the scope of LCM application can be determined by the AI / ML model ID. A base station or terminal can determine the scope of LCM application based on at least one AI / ML model ID. Based on at least one AI / ML model ID, the base station or terminal can determine at least one AI / ML model to which LCM is applied.

[0097] Second, the scope of AI / ML application can be determined by a function ID or function identifier. Function-based LCM can be applied to AI / ML model(s) associated with at least one function without requiring precise AI / ML model designation between the base station and the terminal. The base station or terminal can perform LCM-based operations for AI / ML model(s) associated with a specific function.

[0098] Third, the scope of AI / ML application can be determined by the associated ID. Associated ID-based LCM can be applied to AI / ML models corresponding to at least one associated ID without requiring precise AI / ML model designation between the base station and the terminal. The base station or terminal can perform LCM-based operations for the AI / ML model(s) corresponding to a specific associated ID.

[0099] Figure 6 is a diagram showing the prediction error according to learning of the AI / ML model.

[0100] In Figure 6, the horizontal axis represents the number of training cycles or epochs, and the vertical axis represents the prediction error of the AI / ML model. The L1 curve in Figure 6 represents the validation error of the AI / ML model, and the L2 curve represents the training error (training data). Here, the training error represents the error between the value predicted by AI / ML for the data used for training and the correct answer, and the validation error represents the error between the value predicted by AI / ML for validation data not used for training and the correct answer.

[0101] Referring to Figure 6, as the number of training cycles for an AI / ML model increases, the learning error tends to decrease, but the validation error tends to increase after a certain point. This phenomenon is called overfitting. In other words, insufficient training cycles can lead to underfitting, while excessive training cycles can lead to overfitting.

[0102] In mobile communication environments, training data is often a subset of the environmental data acquired by communication nodes. If an AI / ML model overfits or underfits to this training data, the reliability of the AI / ML model's inference results may decline and performance may degrade.

[0103] To prevent such performance degradation, AI / ML verification is necessary. During the verification process, AI / ML performance can be monitored and tested. Verification can be conducted in an actual wireless communication environment or in a test environment that simulates the actual wireless communication environment.

[0104] AI / ML verification can be performed by a first communication node and a second communication node. Hereinafter, the first communication node may include at least one of a base station and an emulator simulating the base station. The second communication node may include at least one of a terminal and an emulator simulating the terminal.

[0105] Fig. 7 illustrates an AI / ML verification environment performed by a first communication node (110) and a second communication node (120). Fig. 8 is a diagram illustrating the life cycle management (LCM) of each of the first communication node (110) and the second communication node (120) illustrated in Fig. 7.

[0106] Referring to FIGS. 7 and 8, an inference function using AI / ML can be performed by a second communication node (120). The second communication node (120) can perform an inference function using an AI / ML model. In FIGS. 6 and 7, the first communication node (110) functions as test equipment, and the second communication node (120) can perform a test / verification function. Verification can be performed on an AI / ML model driven by the second communication node (120). Although FIGS. 7 and 8 illustrate that inference using an AI / ML model is performed in the second communication node (120), the embodiment is not limited thereto. For example, inference using an AI / ML model can be performed in the first communication node (110). In this case, the first communication node (110) can be a device that performs a test / verification function, and the second communication node (120) can be a test equipment.

[0107] Fig. 9 illustrates an AI / ML verification environment performed by a first communication node (110) and a second communication node (120). Fig. 10 is a diagram illustrating the life cycle management (LCM) of each of the first communication node (110) and the second communication node (120) illustrated in Fig. 9.

[0108] Referring to FIGS. 9 and 10 , AI / ML functions may be applied to each of the first communication node (110) and the second communication node (120). Each of the first communication node (110) and the second communication node (120) may perform an inference function using an AI / ML model. The first communication node (110) may also utilize AI / ML to verify the inference performance of the AI / ML model used by the second communication node (120). This verification environment may be suitable for situations where AI / ML is applied to both terminals and base stations in a mobile communication system.

[0109] Fig. 11 is a flowchart illustrating an AI / ML verification method according to an exemplary embodiment.

[0110] Referring to Figure 11, the network side performs key decisions regarding LCM, and the network side may also initiate procedures associated with LCM. According to the embodiment of Figure 11, LCM can be performed in a network-driven manner.

[0111] At step S110, the first communication node (110) may transmit LCM configuration information for AI / ML verification. The second communication node (120) may receive the LCM configuration information. The LCM configuration information may include information necessary for performing LCM. For example, the LCM configuration information may include configuration information for the second communication node (120) to measure wireless communication quality, configuration information for the second communication node (120) to report measurement results, etc.

[0112] At step S112, the first communication node (110) may lock some of the LCM functions. If the first communication node (110) determines that the environment is an AI / ML verification environment, it may decide to lock the LCM. If the LCM is locked, some of the LCM functions may be locked.

[0113] The first communication node (110) can lock the switching function of the AI / ML model (or the function, or the model or learning data associated with the associated ID), the deactivation of AI / ML, and the fallback function to non-AI / ML among the LCM functions. In addition to the switching function of the AI / ML model, the deactivation of AI / ML, and the fallback function to non-AI / ML, the first communication node (110) can still maintain other LCM functions. For example, the first communication node (110) can normally maintain all other functions except the LCM function that triggers the discontinuation of the AI / ML model under verification, such as LCM performance monitoring, status reporting, learning data collection, and model feedback collection. Through this, the first communication node (110) can continuously observe the performance changes or stability of the target AI / ML model (or the function, or the model or learning data associated with the associated ID) while minimizing the impact on the network operating environment or the AI / ML verification environment.

[0114] By locking the LCM, the first communication node (110) can maintain the AI / ML model under verification. This can prevent the use of the AI / ML model from being interrupted during the verification process. Even if the performance of the AI / ML model deteriorates while the AI / ML model is in operation, the LCM lock can distinguish whether the performance deterioration is due to a problem of the AI / ML model itself or a problem caused by overfitting or underfitting. While the LCM lock is in effect, the first communication node (110) and the second communication node (120) can analyze the learning loss and the verification loss by repeating the training cycle (or epoch). In addition, the first communication node (110) or the second communication node (120) can utilize the learning results only up to the epoch where overfitting does not occur.

[0115] The first communication node (110) can set the LCM lock target in various ways. Here, the LCM lock target may correspond to the AI / ML to be verified. For example, the first communication node (110) may determine the LCM lock target based on any one of at least one AI / ML model ID, at least one AI / ML function, and at least one associated ID.

[0116] Figure 12 is a conceptual diagram illustrating an example of LCM locking based on AI / ML model ID.

[0117] Referring to FIG. 12, the first communication node (110) can determine an LCM lock target based on at least one AI / ML model ID. For example, the first communication node (110) can apply an LCM lock to AI / ML model ID A, and can normally operate all LCM functions for other AI / ML models without an LCM lock. The first communication node (110) can also apply an LCM lock to multiple AI / ML models based on multiple AI / ML model IDs.

[0118] Figure 13 is a diagram illustrating an example of LCM locking based on AI / ML functionality.

[0119] Referring to FIG. 13, the first communication node (110) can determine an LCM lock target based on at least one AI / ML function. For example, the first communication node (110) can apply an LCM lock to AI / ML function A, and can normally operate all LCM functions without an LCM lock for other AI / ML functions. The first communication node (110) can also apply an LCM lock to multiple AI / ML functions based on multiple AI / ML function identifiers.

[0120] Figure 14 is a diagram illustrating an example of an LCM lock based on an association ID.

[0121] Referring to FIG. 14, the first communication node (110) can determine an LCM lock target based on at least one associated ID. For example, the first communication node (110) can apply an LCM lock to associated ID a, and can normally operate all LCM functions without an LCM lock for other associated IDs. The first communication node (110) can also apply an LCM lock to multiple AI / ML functions based on multiple associated IDs.

[0122] By identifying AI / ML models to be verified based on their associated IDs, the verification entity can verify whether the learning and inference environments are truly identical or within a generalizable range. This allows verification that AI / ML models are trained on datasets that guarantee expected performance.

[0123] Referring back to FIG. 11, at step S114, the second communication node (120) may record the learning history of the AI / ML model (or the model or learning data associated with the function or associated ID) and record the learning loss and validation loss for each epoch. As another example, the second communication node (120) may also record the performance history after the already learned AI / ML model (or the model or learning data associated with the function or associated ID) is applied to the mobile communication environment.

[0124] The second communication node (120) can transmit AI / ML performance information and assistance information. The performance information and assistance information may include a measurement report generated by the second communication node (120). The performance information and assistance information may include training loss and validation loss recorded for each learning cycle or epoch. Based on the training loss and validation loss, the AI / ML model can be prevented from being overfitted or underfitted throughout the entire operation cycle.

[0125] In step S116, the first communication node (110) may transmit a management instruction. The management instruction may include management instructions necessary for AI / ML verification. For example, the first communication node (110) may include information on the number of epochs to prevent overfitting or underfitting and maximize the performance of the AI / ML model (or model or training data associated with a function or associated ID) based on the training loss and validation loss per epoch obtained in step S114 in the management instruction. The second communication node (120) may utilize an AI / ML model (or model or training data associated with a function or associated ID) for which learning has been repeated up to a specific epoch based on the management instruction.

[0126] The management instructions may also include information regarding the use of the AI / ML to be verified. For example, based on the performance information obtained in step S114, the first communication node (110) can determine whether to use the AI / ML to be verified.

[0127] If it is determined that AI / ML verification has been completed at step S118, the first communication node (110) may release the LCM lock. Once the LCM lock is released, the functions of changing the AI / ML model (or function, or model or learning data associated with the associated ID), deactivating the AI / ML, and switching to non-AI / ML (Fallback) may be permitted.

[0128] In step S119, the first communication node (110) can perform full-cycle management with the LCM unlocked. The second communication node (120) can use the AI / ML model (or the function, or the model or learning data associated with the associated ID) for which verification has been completed. If the performance degradation of the AI / ML model (or the function, or the model or learning data associated with the associated ID) for which verification has been completed is reported by the second communication node (120), the first communication node (110) can use at least one of the switching function of the AI / ML model (or the function, or the model or learning data associated with the associated ID), the deactivation of AI / ML, and the fallback function to non-AI / ML.

[0129] As described above, the use of LCM locking during the AI / ML verification process prevents changes, deactivation, or conversion to non-AI / ML before the AI / ML verification is complete. Furthermore, sufficient data on training and validation losses for each epoch in the verification environment can be secured to ensure the performance of AI / ML in mobile communication systems. Furthermore, even if performance degradation occurs during the AI / ML verification process, the verification subject can easily determine whether this degradation is due to an inherent problem with the AI / ML model (or the model or training data associated with the function or associated ID) or due to underfitting or overfitting.

[0130] The verification process described above can be implemented in a pre-configured verification environment before the AI / ML model (or the model or training data associated with the function or associated ID) is applied to an actual mobile communication environment. Furthermore, the verification process can also be performed by the base station or terminal after the AI / ML model (or the model or training data associated with the function or associated ID) is applied to an actual mobile communication environment. This allows for confirmation of whether the AI / ML model (or the model or training data associated with the function or associated ID) verified in the pre-verification process overfits only to a specific test environment. Furthermore, it allows for confirmation of how the AI / ML model (or the model or training data associated with the function or associated ID) is affected by various environmental variables that could not be implemented in the verification / test environment, such as variations in the mobile communication radio channel, the base station's multi-antenna technology, the terminal's moving speed, and the influence of unspecified interference.

[0131] By identifying AI / ML models to be verified based on their associated IDs, the verification subject can verify whether the learning and inference environments are truly identical or within a generalizable range. This allows verification that AI / ML models are trained on datasets that guarantee expected performance.

[0132] Fig. 15 is a flowchart illustrating a verification method of AI / ML according to an exemplary embodiment.

[0133] In explaining the embodiment of Fig. 15, any content overlapping with Fig. 10 is omitted.

[0134] Referring to Figure 15, the network side performs key decisions regarding LCM, but the terminal may initiate the procedures involved in LCM. According to the embodiment of Figure 15, the terminal may request LCM from the network, and the network side may perform the necessary judgments for LCM and respond.

[0135] Referring to FIG. 15, at step S120, the first communication node (110) may transmit LCM configuration information for AI / ML verification. The second communication node (120) may receive the LCM configuration information. The LCM configuration information may include information necessary for performing LCM. For example, the LCM configuration information may include configuration information for the second communication node (120) to measure wireless communication quality, configuration information for the second communication node (120) to report measurement results, etc.

[0136] In a verification environment, the first communication node (110) may include an LCM locking instruction in the LCM configuration information. As described above, the LCM locking instruction may include information about an LCM locking target. Here, the LCM locking target may correspond to an AI / ML to be verified. The LCM locking target may be determined by at least one of an AI / ML model ID, an AI / ML function identifier, and an association ID. The first communication node (110) may include the LCM locking instruction in the LCM configuration information when pre-verification of the AI / ML model is performed or when verification of the AI / ML is initiated in an actual mobile communication environment.

[0137] The LCM lock instruction may also be transmitted via separate signaling rather than the LCM configuration information. For example, the first communication node (110) may transmit the LCM lock instruction via separate signaling before or after transmitting the LCM configuration information.

[0138] The embodiment is not limited to this. The LCM configuration information may not include an LCM locking instruction. In this case, the second communication node (120) may determine the LCM locking on its own after recognizing that it is an AI / ML verification environment.

[0139] At step S122, the second communication node (120) may lock some of the LCM functions. The second communication node (120) may lock the switching function of the AI / ML model (or function, or model or learning data associated with the associated ID), the deactivation of AI / ML, and the fallback function to non-AI / ML.

[0140] In step S123, the second communication node (120) may record the learning history of the AI / ML model (or the model or training data associated with the function or associated ID) and record the learning loss and validation loss for each epoch. As another example, the second communication node (120) may also record the performance history after the already trained AI / ML model (or the model or training data associated with the function or associated ID) is applied to the mobile communication environment.

[0141] The second communication node (120) can analyze the training loss and the verification loss for each epoch. The second communication node (120) can analyze the difference between the verification loss and the training loss to determine up to which epoch the trained AI / ML model (or the model or training data associated with the function or associated ID) will be used. The second communication node (120) can independently determine an allowable value for the difference between the verification loss and the training loss and repeat the training until the allowable value is exceeded. As another example, the second communication node (120) can obtain information on an allowable value for the difference between the verification loss and the training loss from the first communication node (110) and repeat the training until the allowable value is exceeded. As another example, the second communication node (120) can report the difference between the validation loss and the learning loss per epoch to the first communication node (110) and obtain information from the first communication node (110) on which epoch to use the trained AI / ML model (or model or learning data associated with a function or associated ID).

[0142] In step S124, the second communication node (120) may transmit a management request to the first communication node (110). The management request message transmitted by the second communication node (120) may include at least one of information regarding availability of the verified AI / ML model (or function, or model or learning data associated with the associated ID), information regarding performance indicators, and measurement reports related to the performance indicators.

[0143] At step S126, the first communication node (110) may transmit a management instruction. The management instruction of the first communication node (110) may include an LCM unlock instruction. If the use of the AI / ML model (or function, or model or training data associated with the associated ID) for which verification has been performed is appropriate, the management instruction of the first communication node (110) may include an explicit or implicit instruction to use the AI / ML model (or function, or model or training data associated with the associated ID).

[0144] At step S128, the second communication node (120) may unlock the LCM. Once the LCM is unlocked, the functions of switching the AI / ML model (or function, or model or learning data associated with the associated ID), deactivating the AI / ML, and switching to non-AI / ML (Fallback) may be permitted.

[0145] At step S129, the second communication node (120) can perform full-cycle management with the LCM lock released.

[0146] As described above, an LCM lock is applied to the second communication node (120) performing the terminal function, so that the second communication node (120) can be prevented from stopping the use of the AI / ML model (or function, or model or learning data associated with the associated ID) being verified during AI / ML verification. During the verification period, even if the performance of the AI / ML model (or function, or model or learning data associated with the associated ID) is somewhat reduced, the second communication node (120) may not request the first communication node (110) to change, deactivate, or switch the AI / ML model.

[0147] Fig. 16 is a flowchart illustrating an AI / ML verification method according to an exemplary embodiment. In describing the embodiment of Fig. 16, any descriptions that overlap with those of Figs. 14 and 15 will be omitted.

[0148] According to the embodiment of Fig. 16, LCM can be performed under the initiative of the terminal side.

[0149] The terminal-side leadership method can be divided into two types.

[0150] First, LCM decisions can be initiated by the terminal. The LCM decision can determine the actions or operations required for LCM. The initiation of LCM actions or action procedures can also be initiated by the terminal. However, the base station can set trigger conditions for the LCM procedure and notify the terminal. The terminal can initiate the LCM procedure when an event corresponding to the trigger condition occurs and proactively perform the necessary LCM decisions.

[0151] Secondly, LCM decisions are made under the initiative of the terminal side, and the actions or operations required for LCM can be initiated autonomously by the terminal rather than through event triggering.

[0152] At step S130, the first communication node (110) may transmit LCM configuration information. For the first method (event-triggered method) mentioned above, the LCM configuration information may include event-triggering configuration information. For the second method (autonomous terminal LCM procedure initiation), the LCM configuration information may not include event-triggering configuration information.

[0153] As described in step S120 of FIG. 15, if necessary, the LCM configuration information may include an LCM lock instruction. If the first communication node (110) determines that the environment is a verification environment, it may transmit LCM configuration information including an LCM lock instruction. Even if the first communication node (110) does not transmit an LCM lock instruction, the second communication node (120) may recognize that the environment is a verification environment and apply an LCM lock.

[0154] At step S132, the second communication node (120) can apply an LCM lock. The second communication node (120) can lock the switching function of the AI / ML model (or function, or model or learning data associated with the associated ID), the deactivation of AI / ML, and the fallback function to non-AI / ML.

[0155] In step S134, the second communication node (120) may record the learning history of the AI / ML model (or the model or training data associated with the function or associated ID) and record the learning loss and validation loss for each epoch. As another example, the second communication node (120) may also record the performance history after the already trained AI / ML model (or the model or training data associated with the function or associated ID) is applied to the mobile communication environment.

[0156] In step S136, the second communication node (120) may transmit a lifecycle management decision report (LCM management report). The second communication node (120) may report management decisions made based on an event trigger or its own judgment. The second communication node (120) may transmit information about the learning history. The second communication node (120) may determine up to which epoch the trained AI / ML model (or the model or learning data associated with the function or associated ID) will be used based on the verification loss and the learning loss. The second communication node (120) may determine whether to use the AI / ML model (or the model or learning data associated with the function or associated ID) to be verified based on the verification result. The second communication node (120) may report whether to continue to use the AI / ML to be verified.

[0157] At step S138, the second communication node (120) can unlock the LCM.

[0158] At step S139, the second communication node (120) can perform full-cycle management (LCM) with the LCM lock released.

[0159] The above describes embodiments that utilize the LCM locking function for AI / ML verification.

[0160] At least one of the first and second communication nodes (110, 120) can adjust a threshold value at which a predetermined function is triggered in the LCM in the AI / ML verification environment.

[0161] A threshold value may be used to trigger at least one of a switch function of an AI / ML model (or a feature, or a model or training data associated with an associated ID), deactivation of AI / ML, and fallback to non-AI / ML. For example, when a threshold condition is satisfied, at least one of a switch function of an AI / ML model (or a feature, or a model or training data associated with an associated ID), deactivation of AI / ML, and fallback to non-AI / ML may be induced.

[0162] The threshold value may be used to determine the conditions under which at least one of the first and second communication nodes (110, 120) performs a management request, a management instruction, and a performance report. When the conditions of the threshold value are satisfied, at least one of the first and second communication nodes (110, 120) may perform at least one of the management request, the management instruction, and the performance report.

[0163] Figure 17 is a flowchart illustrating an AI / ML verification method according to an exemplary embodiment.

[0164] Referring to Figure 17, the network side performs key decisions regarding LCM, and the network side may also initiate procedures associated with LCM. According to the embodiment of Figure 16, LCM can be performed in a network-driven manner.

[0165] At step S140, the first communication node (110) can generate LCM configuration information by applying a threshold value for AI / ML verification.

[0166] The threshold for AI / ML verification may include one or more threshold values. The threshold for AI / ML verification may include a threshold value for determining a condition under which at least one of deactivation, replacement, and transition (fallback to non-AI / ML) of the AI / ML occurs. The threshold for AI / ML verification may also include a threshold value for determining a condition under which LCM signaling occurs between the first and second communication nodes (110, 120). For example, the threshold for AI / ML verification may include a threshold value for determining a condition under which the second communication node (120) transmits performance and auxiliary information of the AI / ML.

[0167] The first communication node (110) may reduce the range in which the threshold value condition is satisfied during the verification process. For example, if the threshold value is a parameter that directly indicates the performance of AI / ML, the first communication node (110) may set the threshold value low to suppress deactivation, replacement, and conversion (fallback to non-AI / ML) of AI / ML even with relatively low performance, thereby allowing AI / ML verification to proceed. However, the embodiment is not limited thereto. Depending on which metric the parameter indicated by the threshold value corresponds to, the threshold value may be set higher in the verification environment compared to normal operation.

[0168] The first communication node (110) can set multiple threshold values ​​for the same parameter. For example, the first threshold value may be a threshold value used in a verification environment. The second threshold value may be a threshold value used in a general environment other than a verification environment (e.g., when used for performing general communication functions other than verification in a commercial communication network).

[0169] The first threshold value can be set to exhibit higher performance than the second threshold value. For example, the first threshold value can be set higher than the second threshold value. This can increase the frequency with which the first threshold condition (e.g., AI / ML performance deteriorating below the first threshold value) occurs. While this may result in a slightly increased reporting frequency, it can provide the effect of enabling precise observation of temporary AI / ML performance degradation or even subtle performance changes.

[0170] The first threshold value can be set to exhibit lower performance than the second threshold value. For example, the first threshold value can be set lower than the second threshold value. This can reduce the frequency with which the first threshold condition (e.g., AI / ML performance falling below the first threshold value) occurs. This method can effectively increase the validation efficiency of AI / ML models in environments with frequent performance fluctuations or concerns about excessive signaling.

[0171] At step S142, the first communication node (110) can transmit LCM configuration information with threshold values ​​applied for the AI / ML verification environment. The second communication node (120) can receive the LCM configuration information.

[0172] The LCM configuration information may include information regarding the scope to which the threshold value for AI / ML verification is applied. The scope to which the threshold value for AI / ML verification is applied may be determined based on at least one AI / ML model ID. The scope to which the threshold value for AI / ML verification is applied may be determined based on at least one AI / ML function identifier. The scope to which the threshold value for AI / ML verification is applied may be determined based on at least one associated ID.

[0173] At step S144, the second communication node (120) may apply a threshold value based on the LCM configuration information. Even if the performance of the AI / ML is somewhat degraded in the verification environment compared to the normal network operation environment, the second communication node (120) may not deactivate, replace, or switch (fallback to non-AI / ML) the AI / ML.

[0174] In step S146, the second communication node (120) can transmit performance information and auxiliary information reports to the first communication node (110) when the conditions of threshold values ​​related to performance information and auxiliary information reports are satisfied.

[0175] The second communication node (120) can reduce the number of reports of performance information and auxiliary information by applying threshold values ​​for AI / ML verification. This can reduce signaling overhead.

[0176] At step S148, the second communication node (120) can receive management instructions from the first communication node (110).

[0177] Fig. 18 is a flowchart illustrating an AI / ML verification method according to an exemplary embodiment.

[0178] Referring to Figure 18, the network side performs key decisions regarding LCM, but the terminal may initiate the procedures involved in LCM. According to the embodiment of Figure 17, the terminal requests LCM from the network, and the network side performs the necessary judgments for LCM and responds. In describing the embodiment of Figure 17, any overlapping content from Figure 16 will be omitted.

[0179] At step S150, the first communication node (110) can generate LCM configuration information by applying a threshold value for AI / ML verification.

[0180] The threshold for AI / ML verification may include one or more threshold values. The threshold for AI / ML verification may include a threshold value for determining a condition under which at least one of AI / ML deactivation, replacement, and transition (fallback to non-AI / ML) occurs. The threshold for AI / ML verification may also include a threshold value for determining a condition under which LCM signaling occurs between the first and second communication nodes (110, 120). For example, the threshold for AI / ML verification may include a threshold value for determining a condition under which the second communication node (120) transmits a full-cycle management request.

[0181] The first communication node (110) can set multiple threshold values ​​for the same parameter. For example, the first threshold value may be a threshold value used in a verification environment. The second threshold value may be a threshold value used in a general environment other than a verification environment (e.g., when used for performing general communication functions other than verification in a commercial communication network).

[0182] The first threshold value can be set to exhibit higher performance than the second threshold value. For example, the first threshold value can be set higher than the second threshold value. This can increase the frequency with which the first threshold condition (e.g., AI / ML performance deteriorating below the first threshold value) occurs. While this may result in a slightly increased reporting frequency, it can provide the effect of enabling precise observation of temporary AI / ML performance degradation or even subtle performance changes.

[0183] The first threshold value can be set to exhibit lower performance than the second threshold value. For example, the first threshold value can be set lower than the second threshold value. This can reduce the frequency with which the first threshold condition (e.g., AI / ML performance falling below the first threshold value) occurs. This method can effectively increase the validation efficiency of AI / ML models in environments with frequent performance fluctuations or concerns about excessive signaling.

[0184]

[0185] At step S154, the second communication node (120) can perform operations required for LCM by applying a threshold value for verification of AI / ML.

[0186] In step S156, the second communication node (120) can transmit a full-cycle management request to the first communication node (110) if the condition of the threshold value related to the full-cycle management request is satisfied.

[0187] At step S158, the second communication node (120) can receive management instructions from the first communication node (110).

[0188] Fig. 19 is a flowchart illustrating an AI / ML verification method according to an exemplary embodiment.

[0189] According to the embodiment of Fig. 19, LCM can be performed under the initiative of the terminal side.

[0190] The terminal-side leadership method can be divided into two types.

[0191] First, LCM decisions can be initiated by the terminal. The LCM decision can determine the actions or operations required for LCM. The initiation of LCM actions or action procedures can also be initiated by the terminal. However, the base station can set trigger conditions for the LCM procedure and notify the terminal. The terminal can initiate the LCM procedure when an event corresponding to the trigger condition occurs and proactively perform the necessary LCM decisions.

[0192] Second, LCM decisions are made under the initiative of the terminal, and the initiation of actions or operations required for LCM can be autonomously initiated by the terminal rather than through event triggering. In describing the embodiment of Fig. 19, any overlap with Figs. 17 and 18 will be omitted.

[0193] At step S160, the first communication node (110) can generate LCM configuration information by applying a threshold value for AI / ML verification. At step S162, the first communication node (110) can transmit the LCM configuration information.

[0194] The first communication node (110) can set multiple threshold values ​​for the same parameter. For example, the first threshold value may be a threshold value used in a verification environment. The second threshold value may be a threshold value used in a general environment other than a verification environment (e.g., when used for performing general communication functions other than verification in a commercial communication network).

[0195] The first threshold value can be set to exhibit higher performance than the second threshold value. For example, the first threshold value can be set higher than the second threshold value. This can increase the frequency with which the first threshold condition (e.g., AI / ML performance deteriorating below the first threshold value) occurs. While this may result in a slightly increased reporting frequency, it can provide the effect of enabling precise observation of temporary AI / ML performance degradation or even subtle performance changes.

[0196] The first threshold value can be set to exhibit lower performance than the second threshold value. For example, the first threshold value can be set lower than the second threshold value. This can reduce the frequency with which the first threshold condition (e.g., AI / ML performance falling below the first threshold value) occurs. This method can effectively increase the validation efficiency of AI / ML models in environments with frequent performance fluctuations or concerns about excessive signaling.

[0197] The threshold for AI / ML verification may include one or more threshold values. The threshold for AI / ML verification may include a threshold value for determining a condition under which at least one of AI / ML deactivation, replacement, and transition (fallback to non-AI / ML) occurs. The threshold for AI / ML verification may also include a threshold value for determining a condition under which LCM signaling occurs between the first and second communication nodes (110, 120). For example, the threshold for AI / ML verification may include a threshold value for determining a condition under which the second communication node (120) transmits a full-cycle management decision report.

[0198] At step S166, the second communication node (120) can perform the operations required for LCM by applying a threshold value for verification of AI / ML.

[0199] At step S167, the second communication node (120) can manage and verify AI / ML. For example, the second communication node (120) can analyze verification loss and learning loss for each epoch. The second communication node (120) can also monitor the performance of a pre-trained AI / ML model.

[0200] At step S168, the second communication node (120) can report a management decision made by an event trigger or by its own judgment.

[0201] The AI / ML verification method according to exemplary embodiments has been described with reference to FIGS. 1 to 18.

[0202] According to at least one embodiment, LCM locking is utilized during the AI / ML verification process to prevent modification, deactivation, or conversion to non-AI / ML before the AI / ML verification is completed. Furthermore, sufficient data on training loss and validation loss for each epoch can be secured in the verification environment to ensure the performance of AI / ML in a mobile communication system. Furthermore, even if performance degradation occurs during the AI / ML verification process, the verification subject can easily determine whether the degradation is due to an inherent problem with the AI / ML model (or a model or training data associated with a function or associated ID) or due to underfitting or overfitting.

[0203] The verification process described above can be implemented in a pre-configured verification environment before the AI / ML model (or the model or training data associated with the function or associated ID) is applied to an actual mobile communication environment. Furthermore, the verification process can also be performed by the base station or terminal after the AI / ML model (or the model or training data associated with the function or associated ID) is applied to an actual mobile communication environment. This allows for confirmation of whether the AI / ML model (or the model or training data associated with the function or associated ID) verified in the pre-verification process overfits only to a specific test environment. Furthermore, it allows for confirmation of how the AI / ML model (or the model or training data associated with the function or associated ID) is affected by various environmental variables that could not be implemented in the verification / test environment, such as variations in the mobile communication radio channel, the base station's multi-antenna technology, the terminal's moving speed, and the influence of unspecified interference.

[0204] By identifying AI / ML models to be verified based on their associated IDs, the verification subject can verify whether the learning and inference environments are truly identical or within a generalizable range. This allows verification that AI / ML models are trained on datasets that guarantee expected performance.

[0205] According to at least one embodiment, by adjusting the threshold values ​​used in LCM operation, the use of the AI / ML under verification can be maintained during the AI / ML verification process. Furthermore, the signaling overhead for LCM can be reduced during the AI / ML verification process.

[0206] Based on the description of the various embodiments of the present disclosure above, it will be apparent to those skilled in the art that the methods and / or processes of the present invention, and their steps, can be implemented by hardware, software, or any combination of hardware and software suitable for a specific application. The hardware may include a general-purpose computer and / or a dedicated computing device, or a specific computing device or a specific aspect or component of a specific computing device. The processes may be implemented by one or more processors, such as a microprocessor, a controller, such as a microcontroller, an embedded microcontroller, a microcomputer, an arithmetic logic unit (ALU), a digital signal processor, such as a programmable digital signal processor, or other programmable devices, having internal and / or external memory. Additionally, or alternatively, the processes may be implemented in an application specific integrated circuit (ASIC), a programmable gate array, such as a field programmable gate array (FPGA), a programmable logic unit (PLU), or Programmable Array Logic (PAL), or any other device capable of executing instructions and responding thereto, or any other device or combination of devices that may be configured to process electronic signals. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. Furthermore, the processing device may access, store, manipulate, process, and generate data in response to the execution of the software.For ease of understanding, the processing unit is sometimes described as being used alone. However, those skilled in the art will appreciate that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors, or a processor and a controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0207] Software may include computer programs, codes, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage media or devices, or transmitted signal waves, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on one or more machine-readable recording media.

[0208] Furthermore, the objects of the technical solution of the present invention or the parts contributing to prior arts may be implemented in the form of program instructions that can be executed through various computer components and recorded on a machine-readable medium. The machine-readable medium may include program instructions, data files, data structures, etc., alone or in combination. The program instructions recorded on the machine-readable recording medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the machine-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs, DVDs, and Blu-rays, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program instructions such as ROMs, RAMs, and flash memories. Examples of program instructions include, but are not limited to, machine code, byte code, as well as high-level language code that can be executed by a computer using an interpreter, etc., which can be created using a structured programming language such as C, an object-oriented programming language such as C++, or a high-level or low-level programming language (assembler, hardware description languages, and database programming languages ​​and technologies), which can be stored and compiled or interpreted for execution on any of the aforementioned devices, as well as a heterogeneous combination of processors, processor architectures, or combinations of different hardware and software, or any other machine capable of executing program instructions.

[0209] Accordingly, in one aspect of the present invention, when the methods and combinations thereof described above are performed by one or more computing devices, the methods and combinations thereof may be implemented as executable code that performs each step. In another aspect, the methods may be implemented as systems that perform the steps, and the methods may be distributed in various ways across devices, or all functions may be integrated into a single dedicated, standalone device or other hardware. In yet another aspect, the means for performing the steps associated with the processes described above may comprise any of the hardware and / or software described above. All such sequential combinations and arrangements are intended to fall within the scope of the present disclosure.

[0210] For example, the hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa. The hardware device may include a processor, such as an MPU, a CPU, a GPU, or a TPU, coupled with a memory, such as a ROM / RAM, for storing program instructions and configured to execute the instructions stored in the memory, and may include a communication unit capable of transmitting and receiving signals with an external device. In addition, the hardware device may include a keyboard, a mouse, or other external input devices for receiving instructions written by developers.

[0211] Although the present invention has been described above with specific details such as specific components and limited embodiments and drawings, these are provided only to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments, and a person having ordinary knowledge in the technical field to which the present invention pertains can make various modifications and variations from this description.

[0212] Therefore, the spirit of the present invention should not be limited to the embodiments described above, and all modifications equivalent to or equivalent to the appended claims of this disclosure, as well as those equivalent to or equivalent to the claims, are considered to fall within the scope of the spirit of the present invention. For example, appropriate results may be achieved even if the described techniques are performed in a different order from the described method, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different form from the described method, or are replaced or substituted by other components or equivalents.

[0213] Such equivalent or equivalent modifications would include, for example, logically equivalent methods capable of producing the same results as those obtained by carrying out the method according to the present invention, and the spirit and scope of the present invention should not be limited by the examples set forth above, but should be understood in the broadest sense permissible by law.

Claims

1. In an AI / ML verification method performed by a first communication node including a communication unit and a processor, A step of transmitting LCM (Life Cycle Management) configuration information for AI / ML full-cycle management to a second communication node; In the case of an AI / ML verification environment, a step of applying an LCM lock mode by locking some of the LCM functions for the AI / ML to be verified; An AI / ML verification method comprising a step of performing verification on an AI / ML to be verified in the above LCM lock mode.

2. In paragraph 1, An AI / ML verification method in which at least one of the AI / ML deactivation function, the AI / ML switch function, and the non-AI / ML conversion function among the LCM functions is locked in the above LCM lock mode.

3. In paragraph 2, An AI / ML verification method in which, in the above LCM lock mode, all functions except for the AI / ML deactivation function, the AI / ML switch function, and the non-AI / ML conversion function operate normally.

4. In paragraph 1, An AI / ML verification method in which the AI / ML to be verified is determined based on at least one AI / ML model ID.

5. In paragraph 1, An AI / ML verification method in which the above-mentioned AI / ML to be verified is determined based on at least one AI / ML function.

6. In paragraph 1, An AI / ML verification method in which the AI / ML to be verified is determined based on at least one associated ID.

7. In paragraph 1, The steps for performing verification on the AI / ML to be verified in the above LCM lock mode are: A step of obtaining performance information from the second communication node to test the inference performance of the second communication node using the AI / ML to be verified; and An AI / ML verification method comprising a step of transmitting management instructions for the AI / ML to be verified based on the performance information.

8. In paragraph 7, The above performance information includes a learning history for the AI / ML to be verified, and the learning history includes analysis information on the learning loss and prediction loss of the AI / ML to be verified for each epoch. The above management instructions are an AI / ML verification method including instruction information on how many epochs to proceed with learning for the AI / ML to be verified.

9. In an AI / ML verification method performed by a second communication node including a communication unit and a processor, A step of receiving LCM configuration information for AI / ML full-cycle management from a first communication node; In the case of an AI / ML verification environment, a step of applying an LCM lock mode by locking some of the LCM functions for the AI / ML to be verified; An AI / ML verification method comprising a step of performing verification on an AI / ML to be verified in the above LCM lock mode.

10. In paragraph 9, An AI / ML verification method in which the above LCM configuration information includes an LCM lock instruction including information of the AI / M to be verified.

11. In paragraph 9, An AI / ML verification method in which at least one of the AI / ML deactivation function, the AI / ML switch function, and the non-AI / ML conversion function among the LCM functions is locked in the above LCM lock mode.

12. In paragraph 9, An AI / ML verification method in which the AI / ML to be verified is determined based on at least one AI / ML model ID.

13. In paragraph 9, An AI / ML verification method in which the above-mentioned AI / ML to be verified is determined based on at least one AI / ML function.

14. In paragraph 9, An AI / ML verification method in which the AI / ML to be verified is determined based on at least one associated ID.

15. In paragraph 9, The steps for performing verification on the AI / ML to be verified in the above LCM lock mode are: A step of acquiring learning history for the AI / ML to be verified; A step of analyzing the validation loss and training loss for each epoch in the above learning history; and An AI / ML verification method comprising a step of transmitting a full-cycle management request to the first communication node based on the analysis results for the verification loss and learning loss per epoch.

16. In paragraph 9, The steps for performing verification on the AI / ML to be verified in the above LCM lock mode are: A step of acquiring learning history for the AI / ML to be verified; A step of analyzing the validation loss and training loss for each epoch in the above learning history; and An AI / ML verification method comprising a step of performing full-cycle management based on the analysis results for the verification loss and learning loss for each epoch, and then transmitting a full-cycle management decision report to the first communication node.

17. In an AI / ML verification method performed by a first communication node including a communication unit and a processor, When it is determined that the AI / ML verification environment is applicable, a step of generating LCM configuration information including at least one threshold value set for the verification environment; A step of transmitting LCM configuration information including at least one threshold value set for the above verification environment to a second communication node, An AI / ML verification method wherein at least one of the above thresholds comprises a threshold value for determining a condition under which at least one of AI / ML deactivation, AI / ML replacement, and non-AI / ML conversion occurs.

18. In paragraph 17, An AI / ML verification method wherein said at least one threshold value comprises a threshold value for determining a condition under which LCM signaling occurs between said second communication node and said first communication node.

19. In paragraph 17, An AI / ML verification method wherein the at least one threshold value is set such that the range in which the condition of the at least one threshold value is satisfied is reduced in the AI / ML verification environment compared to a general environment.

20. In paragraph 17, An AI / ML verification method in which at least one threshold value is set such that the range in which the condition of at least one threshold value is satisfied increases in the AI / ML verification environment compared to a general environment.

Images