Method and apparatus for providing ai and ml media services in wireless communication system

Abstract

The present disclosure relates to a 5G or 6G communication system for supporting higher data rates. According to various embodiments of the present disclosure, a method performed by a user equipment (UE) in a wireless communication system, the method comprising: receiving information about at least one artificial intelligence (AI) model from a first network entity; determining an AI model based on the information about the at least one AI model; determining whether the AI model is to be used for an AI segmentation inference service; requesting an AI split reasoning service from the first network entity; establishing an AI model transfer assembly line for the AI model; and establishing a media transfer pipeline for transferring media data used in the AI model.

Description

Technical Field

[0001] The present disclosure relates generally to wireless communication systems, and more particularly to methods and apparatus for providing artificial intelligence (AI) and machine learning (ML) media services in wireless communication systems. Background Art

[0002] The fifth generation (5G) mobile communication technology defines a wide frequency band, making high transmission rates and new services possible, and can be implemented not only in "6 GHz or less" frequency bands such as 3.5 GHz, but also in "6 GHz or more" frequency bands called millimeter waves (mmWave) including 28 GHz and 39 GHz. In addition, the implementation of the sixth generation (6G) mobile communication technology (referred to as a super 5G system) in the terahertz (THz) frequency band (e.g., 95 GHz to 3 THz frequency band) has been considered to achieve a transmission rate fifty times faster than that of 5G mobile communication technology and an ultra-low latency one-tenth of that of 5G mobile communication technology.

[0003] At the beginning of the development of 5G mobile communication technology, in order to support services and meet performance requirements for enhanced mobile broadband (eMBB), ultra-reliable low-latency communication (URLLC) and massive machine type communication (mMTC), standardization has been ongoing on the following items: beamforming and massive multiple-input multiple-output (MIMO) for mitigating radio wave path loss and increasing radio wave transmission distance in mmWave; dynamic operation of supporting parameter sets (e.g., operating multiple subcarrier intervals) to efficiently utilize mmWave resources and time slot formats; initial access technology for supporting multi-beam transmission and broadband; definition and operation of bandwidth parts (BWP); new channel coding methods such as low-density parity-check (LDPC) codes for large-scale data transmission and polarization codes for highly reliable transmission of control information; L2 preprocessing; and network slicing for providing dedicated networks dedicated to specific services.

[0004] Currently, discussions are underway regarding improvements and performance enhancements to initial 5G mobile communication technologies in consideration of services to be supported by the 5G mobile communication technologies, and physical layer standardization is ongoing regarding technologies such as: vehicle-to-everything (V2X) for helping autonomous vehicles make driving decisions based on information about the location and status of the vehicle sent by the vehicle and enhancing user convenience; unlicensed new radio (NR-U) for the purpose of making system operations compliant with various regulatory requirements in unlicensed frequency bands; new radio (NR) user equipment (UE) power saving; non-terrestrial network (NTN) for UE-satellite direct communication for providing coverage in areas where communication with terrestrial networks is unavailable; and positioning.

[0005] In addition, standardization has been carried out in terms of air interface architecture / protocols for technologies such as: Industrial Internet of Things (IIoT) for supporting new services through interworking and integration with other industries; Integrated Access and Backhaul (IAB) for providing nodes for network service area expansion by supporting wireless backhaul links and access links in an integrated manner; Mobility enhancement including conditional switching and dual active protocol stack (DAPS) switching; and two-step random access (2-step random access channel (RACH) for NR) for simplifying the random access process. Standardization of system architecture / services for a 5G baseline architecture (e.g., a service-based architecture or a service-based interface) for combining network function virtualization (NFV) and software-defined network (SDN) technologies, and mobile edge computing (MEC) for receiving services based on UE location is also underway.

[0006] With the commercialization of 5G mobile communication systems, the already exponentially increasing number of connected devices will be connected to the communication network, so it can be expected that enhanced functions and performance of the 5G mobile communication system and integrated operation of the connected devices will be necessary. To this end, new research is planned on the following items: extended reality (XR) for efficiently supporting augmented reality (AR), virtual reality (VR), mixed reality (MR), etc.; 5G performance improvement and complexity reduction by utilizing artificial intelligence (AI) and machine learning (ML); AI service support; metaverse service support, and drone communication.

[0007] In addition, such development of 5G mobile communication systems will serve as the basis for not only developing new waveforms for providing coverage in the terahertz band of 6G mobile communication technology, multi-antenna transmission technologies such as full-dimensional MIMO (FD-MIMO), array antennas and massive antennas, metamaterial-based lenses and antennas for improving the coverage of terahertz band signals, and high-dimensional spatial multiplexing technologies using orbital angular momentum (OAM) and reconfigurable smart surfaces (RIS), but also developing full-duplex technology for improving the frequency efficiency of 6G mobile communication technology and improving system networks, AI-based communication technology for implementing system optimization from the design stage by utilizing satellites and artificial intelligence (AI) and internalizing end-to-end AI support functions, and next-generation distributed computing technology for implementing services at a complexity level that exceeds the limits of UE operating capabilities by utilizing ultra-high performance communication and computing resources.

[0008] The above information is presented as background information only to assist with an understanding of the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above might be applicable as prior art with respect to the present disclosure. Summary of the invention

[0009] Technical issues

[0010] Based on the above discussion, various embodiments of the present disclosure are provided to propose a method and apparatus for providing artificial intelligence (AI) and machine learning (ML) media services in a wireless communication system.

[0011] Technical Solution

[0012] According to an aspect of the exemplary embodiments, there is provided a communication method in wireless communication.

[0013] Beneficial Effects

[0014] Various embodiments of the present disclosure may provide a method and apparatus for providing artificial intelligence (AI) and machine learning (ML) media services in a wireless communication system.

[0015] Effects obtainable from the present disclosure are not limited to the above-mentioned effects, and other advantageous effects not mentioned will be clearly understood by those skilled in the art from the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A wireless communication system according to various embodiments of the present disclosure is shown.

[0017] Figure 2 The configuration of a base station in a wireless communication system according to various embodiments of the present disclosure is shown.

[0018] Figure 3 The configuration of a terminal in a wireless communication system according to various embodiments of the present disclosure is shown.

[0019] Figure 4 The overall 5G media flow architecture of TS26.501 according to various embodiments of the present disclosure is shown.

[0020] Figure 5 The 5G media flow general architecture of TS26.501 according to various embodiments of the present disclosure is shown.

[0021] Figure 6 An example of a process of a media downlink flow specified in TS26.501 according to various embodiments of the present disclosure is shown.

[0022] Figure 7 A baseline procedure describing the establishment of a unicast media downlink streaming session defined in TS 26.501 according to various embodiments of the present disclosure is shown.

[0023] Figure 8 An example of an AI / ML media service scenario in which an AI / ML model must be transmitted from a network to a UE (terminal device) according to various embodiments of the present disclosure is shown.

[0024] Fig. 9 Examples of scenarios in which an AI model is transmitted to a UE and a media (e.g., video) stream is transmitted to the UE according to various embodiments of the present disclosure are shown.

[0025] Fig.10 An example of a scenario of splitting the reasoning required for AI media services between the network and the UE according to various embodiments of the present disclosure is shown.

[0026] Fig.11 An example of an AI for media (AI4Media) architecture according to various embodiments of the present disclosure is shown, which identifies various functional entities and interfaces that implement AI model transmission for media services in a wireless communication system.

[0027] Fig.12 The wireless communication system according to various embodiments of the present disclosure is shown. Figure 8 An example of the AL model downlink transmission process corresponding to the scenario.

[0028] Fig.13 The wireless communication system according to various embodiments of the present disclosure is shown. Figure 8 The basic workflow process of the scene.

[0029] Fig.14A The wireless communication system according to various embodiments of the present disclosure is shown. Fig.13 Examples of various AI model delivery pipelines obtained in step 4.

[0030] Fig. 14B The wireless communication system according to various embodiments of the present disclosure is shown. Fig.13 Examples of various AI model delivery pipelines obtained in step 4.

[0031] Fig.15 The wireless communication system according to various embodiments of the present disclosure is shown. Fig. 9 An example of the process of AL model downlink transmission and media content streaming transmission corresponding to the scenario.

[0032] Fig.16 The wireless communication system according to various embodiments of the present disclosure is shown as follows Fig.13 An extension of the basic workflow process described in Fig. 9 The basic workflow process of the scene.

[0033] Fig.17 According to various embodiments of the present disclosure, the present invention is shown in a wireless communication system. Fig.10 An example of the process of segmenting reasoning AI media services corresponding to the scenario.

[0034] Fig.18 The wireless communication system according to various embodiments of the present disclosure is shown. Fig. 9 The basic workflow process of the scene.

[0035] Best Mode for Carrying Out the Invention

[0036] According to various embodiments of the present disclosure, a method performed by a user equipment (UE) in a wireless communication system is provided, the method comprising: receiving information about at least one artificial intelligence (AI) model from a first network entity; determining an AI model based on the information about at least one AI model; determining whether to use the AI ​​model for an AI segmentation inference service; requesting an AI segmentation inference service from the first network entity; establishing an AI model transmission pipeline for the AI ​​model; and establishing a media transmission pipeline for transmitting media data used in the AI ​​model.

[0037] The information about the at least one AI model includes a uniform resource locator (URL) for obtaining a list of the at least one AI model.

[0038] The method may further include receiving intermediate data from a second network entity over the media delivery pipeline.

[0039] The method may also include sending a status report of information about the AI ​​segmentation inference service to the first network function.

[0040] The method may also include updating the AI ​​model and the AI ​​model delivery pipeline based on the information about the AI ​​segmentation inference service and the information about a network status.

[0041] According to various embodiments of the present disclosure, a method performed by a first network entity in a wireless communication system is provided, the method comprising: sending information about at least one artificial intelligence (AI) model to a user equipment (UE), identifying an AI model determined based on the information about the at least one AI model, receiving a request for an AI segmentation inference service using the AI ​​model, establishing an AI model transmission pipeline for the AI ​​model, and establishing a media transmission pipeline for transmitting media data used in the AI ​​model.

[0042] The information about the at least one AI model includes a uniform resource locator (URL) for obtaining a list of the at least one AI model.

[0043] The method may also include receiving a status report from the UE regarding information regarding the AI ​​segmentation inference service.

[0044] The method may also include updating the AI ​​model and the AI ​​model delivery pipeline based on the information about the AI ​​segmentation inference service and the information about a network status.

[0045] The method may further include triggering an inference process in the second network entity.

[0046] According to another embodiment of the present disclosure, a user equipment (UE) in a wireless communication system includes: at least one transceiver; and at least one processor, the at least one processor being operably coupled to the at least one transceiver, wherein the at least one processor is configured to: receive information about at least one artificial intelligence (AI) model from a first network entity, determine the AI ​​model based on the information about the at least one AI model, determine whether to use the AI ​​model for an AI segmentation reasoning service, request the AI ​​segmentation reasoning service from the first network entity, establish an AI model transmission pipeline for the AI ​​model, and establish a media transmission pipeline for transmitting media data used in the AI ​​model.

[0047] The information about the at least one AI model includes a uniform resource locator (URL) for obtaining a list of the at least one AI model.

[0048] The at least one processor is further configured to receive intermediate data from a second network entity on the media delivery pipeline.

[0049] The at least one processor is further configured to send a status report of information about the AI ​​segmentation inference service to the first network function.

[0050] The at least one processor is further configured to update the AI ​​model and the AI ​​model delivery pipeline based on information about the AI ​​segmentation inference service and information about a network status.

[0051] According to another embodiment of the present disclosure, a first network entity in a wireless communication system includes: at least one transceiver; and at least one processor, the at least one processor being operably coupled to the at least one transceiver, wherein the at least one processor is configured to: send information about at least one artificial intelligence (AI) model to a user equipment (UE), identify an AI model determined based on the information about the at least one AI model, receive a request for an AI segmentation inference service using the AI ​​model, establish an AI model transmission pipeline for the AI ​​model, and establish a media transmission pipeline for transmitting media data used in the AI ​​model.

[0052] The information about the at least one AI model includes a uniform resource locator (URL) for obtaining a list of the at least one AI model.

[0053] The at least one processor is further configured to receive a status report from the UE regarding information regarding the AI ​​segmentation inference service.

[0054] The at least one processor is further configured to update the AI ​​model and the AI ​​model delivery pipeline based on information about the AI ​​segmentation inference service and information about a network status.

[0055] The at least one processor is further configured to trigger an inference process in the second network entity. DETAILED DESCRIPTION

[0056] The terms used in this disclosure are only used to describe specific embodiments, rather than to limit the scope of other embodiments. Singular expressions may include plural expressions, unless the context clearly indicates otherwise. The terms used herein, including technical terms or scientific terms, may have the same meaning as the meanings generally understood by those of ordinary skill in the art described in this disclosure. Among the terms used in this disclosure, the terms defined in the general dictionary may be interpreted as having the same or similar meanings as those in the context of the relevant technology, and unless clearly defined in this disclosure, will not be interpreted as having ideal or overly formal meanings. In some cases, even the terms defined in this disclosure may not be interpreted as excluding embodiments of the present disclosure.

[0057] In various embodiments of the present disclosure described below, a hardware access method is described as an example. However, since various embodiments of the present disclosure include technologies using both hardware and software, various embodiments of the present disclosure do not exclude software-based access methods.

[0058] Figure 1 A wireless communication system according to various embodiments of the present disclosure is shown. Figure 1 Base station 110, terminal 120, and terminal 130 are shown as some nodes using wireless channels in a wireless communication system. Figure 1 Only one base station is shown, but other base stations that are the same as or similar to base station 110 may also be included.

[0059] The base station 110 is a network infrastructure that provides wireless access to the terminals 120 and 130. The coverage of the base station 110 is defined as a specific geographical area based on the distance over which a signal can be transmitted. In addition to being called a "base station", the base station 110 may also be called an "access point (AP)", "eNode B (eNB)", "5th generation node (5G node)", "next generation node B (gNB)", "radio point", "transmit / receive point (TRP)" or other terms with equivalent technical meanings.

[0060] Each of the terminal 120 and the terminal 130 is a device for use by a user and communicates with the base station 110 through a wireless channel. In some cases, at least one of the terminal 120 and the terminal 130 may work without user participation. That is, at least one of the terminal 120 and the terminal 130 is a device that performs machine type communication (MTC) and may not be carried by a user. Each of the terminal 120 and the terminal 130, in addition to the "terminal", may also be referred to as a "user equipment (UE)", "mobile station", "user station", "remote terminal", "wireless terminal", "user device", or other terms with equivalent technical meanings.

[0061] The base station 110, the terminal 120, and the terminal 130 can transmit and receive wireless signals in the mmWave frequency band (e.g., 28 GHz, 30 GHz, 38 GHz, and 60 GHz). At this time, in order to improve the channel gain, the base station 110, the terminal 120, and the terminal 130 can perform beamforming. Here, beamforming may include transmit beamforming and receive beamforming. That is, the base station 110, the terminal 120, and the terminal 130 can give directionality to the transmitted signal or the received signal. For this purpose, the base station 110 and the terminals 120 and 130 can select the service beams 112, 113, 121, and 131 through a beam search or a beam management process. After selecting the service beams 112, 113, 121, and 131, communication can be performed through resources that are in a quasi-co-location (QCL) relationship with the resources that transmit the service beams 112, 113, 121, and 131.

[0062] The first antenna port and the second antenna port may be evaluated as being QCL related when a large-scale characteristic of a channel over which symbols are transmitted on a first antenna port can be inferred from a channel over which symbols are transmitted on a second antenna port. For example, the large-scale characteristic may include at least one of delay spread, Doppler spread, Doppler shift, average gain, average delay, and spatial reception parameters.

[0063] Figure 2 The configuration of a base station in a wireless communication system according to various embodiments of the present disclosure is shown. Figure 2The configuration shown in can be understood as the configuration of the base station 110. The terms such as "-unit", "-part" and the like used below refer to a unit that processes at least one function or operation, and can be implemented by hardware or software, or by a combination of hardware and software.

[0064] Reference Figure 2 The base station includes a wireless communication unit 210, a backhaul communication unit 220, a storage unit 230, and a control unit 240.

[0065] The wireless communication unit 210 performs functions for transmitting and receiving signals through a wireless channel. For example, the wireless communication unit 210 performs a conversion function between a baseband signal and a bit stream according to the physical layer standard of the system. For example, when sending data, the wireless communication unit 210 provides a composite symbol by encoding and modulating the transmission bit stream. Moreover, when receiving data, the wireless communication unit 210 recovers the received bit stream by demodulating and decoding the baseband signal.

[0066] Moreover, the wireless communication unit 210 up-converts the baseband signal into a radio frequency (RF) band signal, transmits the signal through an antenna, and down-converts the RF band signal received through the antenna into a baseband signal. For this purpose, the wireless communication unit 210 may include a transmission filter, a reception filter, an amplifier, a mixer, an oscillator, a digital-to-analog converter (DAC), an analog-to-digital converter (ADC), etc. Moreover, the wireless communication unit 210 may include a plurality of transmission paths and reception paths. In addition, the wireless communication unit 210 may include at least one antenna array consisting of a plurality of antenna elements.

[0067] In terms of hardware, the wireless communication unit 210 may include a digital unit and an analog unit, and the analog unit may include a plurality of subunits according to operating power and operating frequency, etc. The digital unit may be implemented as at least one processor (eg, a digital signal processor (DSP)).

[0068] The wireless communication unit 210 transmits and receives signals as described above. Therefore, all or part of the wireless communication unit 210 may be referred to as a "transmitter", "receiver" or "transceiver". Moreover, in the following description, transmission and reception performed through a wireless channel are used as meanings including that the above-mentioned processing is performed by the wireless communication unit 210.

[0069] The backhaul communication unit 220 provides an interface for communicating with other nodes in the network. That is, the backhaul communication unit 220 converts a bit stream transmitted from the base station to another node, such as another access node, another base station, an upper node, a core network, etc., into a physical signal, and converts a physical signal received from another node into a bit stream.

[0070] The storage unit 230 stores data such as basic programs, applications, setting information, etc. for base station operation. The storage unit 230 may include a volatile memory, a nonvolatile memory, or a combination of a volatile memory and a nonvolatile memory. And, the storage unit 230 provides the stored data according to a request of the control unit 240.

[0071] The control unit 240 controls the overall operation of the base station. For example, the control unit 240 sends and receives signals through the wireless communication unit 210 or the backhaul communication unit 220. Moreover, the control unit 240 writes data in the storage unit 230 and reads data in the storage unit 230. And, the control unit 240 can perform the functions of the protocol stack required by the communication standard. According to another implementation example, the protocol stack can be included in the wireless communication unit 210. To this end, the control unit 240 may include at least one processor.

[0072] Figure 3 The configuration of a terminal in a wireless communication system according to various embodiments of the present disclosure is shown. Figure 3 The configuration shown in can be understood as the configuration of the terminal 120. Terms such as "-unit", "-part" and the like used below refer to a unit that processes at least one function or operation, and can be implemented by hardware or software, or by a combination of hardware and software.

[0073] Reference Figure 3 The terminal includes a communication unit 310, a storage unit 320 and a control unit 330.

[0074] The communication unit 310 performs functions for sending and receiving signals through a wireless channel. For example, the communication unit 310 performs a conversion function between a baseband signal and a bit stream according to the physical layer standard of the system. For example, when sending data, the communication unit 310 provides a composite symbol by encoding and modulating the transmission bit stream. Moreover, when receiving data, the communication unit 310 recovers the received bit stream by demodulating and decoding the baseband signal. Moreover, the communication unit 310 up-converts the baseband signal to an RF band signal and transmits the signal through an antenna, and down-converts the RF band signal received through the antenna to a baseband signal. For example, the communication unit 310 may include a transmission filter, a reception filter, an amplifier, a mixer, an oscillator, a DAC, an ADC, etc.

[0075] Moreover, the communication unit 310 may include multiple transmit / receive paths. In addition, the communication unit 310 may include at least one antenna array consisting of multiple antenna elements. In terms of hardware, the communication unit 310 may include digital circuits and analog circuits (e.g., radio frequency integrated circuits (RFIC)). Here, the digital circuits and analog circuits may be implemented in one package. Moreover, the communication unit 310 may include multiple RF chains. In addition, the communication unit 310 may perform beamforming.

[0076] The communication unit 310 transmits and receives signals as described above. Therefore, all or part of the communication unit 310 may be referred to as a "transmitter", "receiver" or "transceiver". Moreover, in the following description, transmission and reception performed through a wireless channel are used as meanings including that the above-mentioned processing is performed by the wireless communication unit 210.

[0077] The storage unit 320 stores data such as basic programs, applications, setting information, etc. for terminal operation. The storage unit 320 may include a volatile memory, a nonvolatile memory, or a combination of a volatile memory and a nonvolatile memory. And, the storage unit 320 provides the stored data according to a request of the control unit 330.

[0078] The control unit 330 controls the overall operation of the terminal. For example, the control unit 330 sends and receives signals through the communication unit 310. Moreover, the control unit 330 writes data in the storage unit 320 and reads data in the storage unit 320. Moreover, the control unit 330 can perform the functions of the protocol stack required by the communication standard. To this end, the control unit 330 may include at least one processor or microprocessor, or may be a part of a processor. Moreover, the communication unit 310 and a part of the control unit 330 may be referred to as a communication processor (CP).

[0079] Various embodiments of the present disclosure relate to the following topics:

[0080] Mechanism for Segmenting AI (Artificial Intelligence) / ML (Machine Learning) Reasoning Media Services The technical fields of various embodiments of the present disclosure include:

[0081] 5G network system for multimedia, architecture and process for AI / ML model delivery and transmission over 5G, AI / ML model delivery and transmission over 5G for AI-enhanced multimedia services.

[0082] The background technologies of various embodiments of the present disclosure include the following:

[0083] Artificial Intelligence (AI) is a general concept that defines the ability of a system to take actions based on 2 main conditions:

[0084] -The situation in which the task must be completed, referring to the values ​​or states of different input parameters.

[0085] - Past experience of completing the same task with different parameter values, and the potential success record with each parameter value.

[0086] Machine learning (ML) is often described as a subset of AI in which applications have the ability to learn from past experience. This learning feature usually begins with an initial training phase to ensure a minimum level of performance when it is put into use.

[0087] Recently, AI / ML has been introduced and promoted to media-related applications, ranging from traditional applications such as image classification, voice / face recognition, etc. to more recent applications such as video quality enhancement. As research in this field matures, it can be expected that there will be an increasing number of complex AI / ML-based applications that require higher computational processing; such processing involves processing large amounts of data, not only for the input and output to the AI / ML models, but also for the increased data size and complexity of the AI / ML models themselves. The growing amount of AI / ML-related data and the need to support processing-intensive mobile applications (such as VR, AR / MR, games, etc.) highlight the importance of handling certain aspects of AI / ML processing by servers over 5G systems in order to meet the latency requirements required for various applications.

[0088] Problems of the prior art in various embodiments of the present disclosure include the following:

[0089] The current implementation of AI / ML is mainly proprietary solutions, which are implemented through applications and are not compatible with other market solutions. In order to support AI / ML for multimedia applications through 5G, AI / ML models should support compatibility between UE devices and application providers from different MNOs. Not only that, but the transmission of AI / ML models for AI / ML media services should support the selection and transmission of AI / ML models based on media context, UE status and network status. The processing power of UE devices is also a limitation for AI / ML media services, because next-generation media services such as AR are typically consumed on lightweight, low-processing power devices such as AR glasses, for which long battery life is also a major design obstacle / limitation.

[0090] The invention objectives as solutions to the problems in various embodiments of the present disclosure are as follows:

[0091] The present invention extends the current framework and architecture of 5G Media Streaming (5GMS) to support AI / ML media services, namely:

[0092] -Transmit AI / ML models from the network to the UE for multimedia services.

[0093] - Selection, configuration and management of the AI / ML models and their delivery by newly defined network and UE entities, which may consider 5G network state, UE processing / runtime state and / or capabilities and media characteristics as inputs to these decisions related to AI / ML model delivery.

[0094] -AI profile, which contains the information necessary for the selection, configuration and management of the AI / ML models and their transmission by entities in the network or in the UE. The AI ​​profile is created by the network and sent to the UE, which can also be updated and send requests back to the network.

[0095] - The AI ​​configuration file may also include information on the possibilities of different delivery method pipelines for delivering AI / ML models, namely: full download, partial download in streaming mode, topology information download / update, and parameter information download / update.

[0096] - The AI ​​profile may also contain information indicating optional support for split reasoning between the network and the UE, which may be selected by the UE or the network.

[0097] The invention results of various embodiments of the present disclosure are as follows:

[0098] The present invention can achieve the following:

[0099] - AI / ML model selection, delivery and management between network and UE for multimedia services driven by network status, UE status and multimedia context

[0100] Figure 4 The overall 5G media flow architecture in TS26.501 is shown, representing the specified 5GMS functions within the 5GS defined in TS23.501.

[0101] Figure 5 The general architecture of 5G media flow in TS26.501 is shown, identifying which media flow functional entities and interfaces are specified in the specification.

[0102] Figure 6 The high level process for media downlink flow specified in TS 26.501 is shown.

[0103] For reference, Figure 7 A baseline procedure describing the establishment of a unicast media downlink streaming session defined in TS 26.501 is shown.

[0104] Figure 8A simple AI / ML media service scenario is shown that requires an AI / ML model to be transmitted from the network to the UE (terminal equipment). Upon receiving the AI ​​model, the UE device performs inference of the model, feeding the relevant media into the AI ​​model as input.

[0105] Typical examples:

[0106] - John was in Seoul for his summer vacation and he wanted to go sightseeing at Lotte Tower in Jamsil. John couldn't read Korean and found it difficult to find the way to Lotte Tower.

[0107] - John takes out his mobile phone (UE) and opens the augmented reality navigation service on it. His network operator provides this service over 5G and transmits the appropriate AI model to his mobile phone by analyzing different information. Such information includes information available from the network, such as John's UE location, his billing policy, network availability and conditions (bandwidth, latency), etc., the processing power and status of his UE, and media attributes that will be used as input to the AI ​​model.

[0108] -Once the AI ​​model is transferred to John’s phone, the AR navigation service activates the camera on the phone to capture John’s surroundings.

[0109] -The video captured from the phone’s camera is fed into the AI ​​model as input and the AI ​​model inference is initiated.

[0110] -The output of the AI ​​model provides direction labels (such as navigation arrows) superimposed on the phone screen live camera to guide John to the Lotte Tower. The Korean road signs are also overlaid with English labels output from the AI ​​model.

[0111] Fig. 9 A scenario is shown where an AI model is transmitted to a UE and media (such as video) is also streamed to the UE. In the UE, the streamed video is fed as input into the received AI model for processing.

[0112] AI models can perform any media related processing, such as: video upscaling, video quality enhancement, vision applications such as object recognition, facial recognition, etc.

[0113] A brief description of the steps required is (see Fig.15 and Fig.16 ):

[0114] *Service Announcement

[0115] * Request / selection made by UE or network (which task UE wants to perform, considering media requirements, network state parameters, UE state parameters, network, or UE selects appropriate AI model)

[0116] * Provisioning and ingestion of models in the network

[0117] * Serving media in the network

[0118] *Session establishment

[0119] *Transmit AI model from network to UE

[0120] *Configure media session downlink

[0121] * Stream media from the network

[0122] *AI media reasoning in UE

[0123] Fig.10 A scenario is shown where the reasoning required for an AI media service is split between the network and the UE. A portion of the AI ​​model to be reasoned on the UE is transmitted from the network to the UE. Another portion of the AI ​​model to be reasoned in the network is provided by the network to an entity that performs reasoning in the network. When the media for reasoning is fed as input into the network portion of the AI ​​model, the media for reasoning is first supplied and ingested by the network to the network reasoning entity. The output (intermediate data) of the network-side reasoning is then sent to the UE, which receives the intermediate data and feeds it as input into the UE-side portion of the AI ​​model, thereby completing the reasoning of the entire model.

[0124] In this scenario, the split decision and configuration is negotiated between the UE and the network. A brief description of the steps required is as follows (see Fig.17 and Fig.18 ):

[0125] *Service Announcement

[0126] * Request / selection made by UE (which task it wants to perform, given media requirements, AF selects appropriate model header)

[0127] * Provisioning UE task model header and core model in the network

[0128] * Serving media in the network

[0129] * Split configuration settings and build

[0130] *Session establishment

[0131] * Configure intermediate data session downlink

[0132] * Download / stream model headers from the network

[0133] *Execute network core model inference

[0134] * Stream intermediate data from the network

[0135] *Task model reasoning in UE

[0136] In one example of a split configuration, an AI model service may include a core part and a task-specific part (e.g., a traffic sign recognition task or a facial recognition task), where the core part of the AI ​​model is common to multiple possible tasks. In this case, the split configuration may overlap the core part with the task part in a way that the network performs reasoning on the core part of the model and the UE (receives and) performs reasoning on the task part of the model.

[0137] Fig.11 The AI4Media architecture in the present invention is shown, which identifies various functional entities and interfaces for implementing the AI ​​model transmission of media services. The "M" interface corresponds to the interface specified in TS26.501, while the "A" interface is defined by the content of the present invention. The "AM" interface corresponds to the interface within the scope of 5GMS (TS26.501) and within the scope of the content of the present invention.

[0138] AI Media AF: An application function dedicated to AI media services similar to the application function defined in TS23.501 clause 6.2.10. Various control functions are usually provided to the AI ​​media session handler on the UE and / or to the AI ​​media application provider. It may interact with other 5GC network functions, such as the Data Collection Proxy (DCP) functional entity (which interacts with the AI / ML endpoints and / or the 3GPP core network to collect the information required by the AI ​​Media AF). The DCP may or may not include the NWDAF function / functionality.

[0139] AI AS: An application server dedicated to AI media services that hosts 5G AI media (sub)functions such as Fig.13 The AI ​​delivery function and AI model provider function shown in . The AI ​​AS typically supports AI model hosting, such as a media AS, by ingesting AI models from AI media application providers and egesting the models to other network functions for network inference.

[0140] Media AS: corresponds to the Media AS in TS26.501, but may also include AI engine functionality that performs AI model reasoning.

[0141] AI Media Application Provider: External application with content-specific media capabilities and / or AI-specific media capabilities (AI model creation, segmentation, updates, etc.).

[0142] The AI ​​media client in UE contains three sub-functions:

[0143] AI Media Session Handler: A function on the UE that communicates with the AI ​​Media AF to establish, control and support the delivery of AI model sessions and / or media sessions, and may perform additional functions such as consumption and QoE metrics collection and reporting. The AI ​​Media Session Handler may expose an API that can be used by AI media applications.

[0144] AI Client: A function on the UE that communicates with the AI ​​AS to download / stream (or even upload) AI model data, and can provide an API in the UE to the AI ​​media application for AI model inference ( Fig.13 The AI ​​engine runtime in the AI ​​​​Media Session Handler provides AI model session control (AI model manager for managing AI models (e.g., ONNX files)), and AI access functions for accessing AI model data such as topology data and / or AI model parameters (weights, biases).

[0145] Media player: The same function on the UE that corresponds to the function specified in TS26.501. The media player can play media content to the AI ​​client as data input for AI model reasoning in the AI ​​client.

[0146] In another embodiment of the present invention, the AI ​​reasoning engine in the UE may exist in the media player instead of the AI ​​client. It may also exist in another function of the UE.

[0147] In another embodiment of the present invention, the AI ​​engine in the network may exist in the AI ​​AS instead of the media AS.

[0148] Fig.12 Shown with Figure 8 The high-level process of AL model downlink transmission corresponding to the scenario in .

[0149] step:

[0150] 1. The AI ​​Media Application Provider creates a provisioning session with the AI ​​Media AF and starts provisioning the use of the AI4Media system. During the setup phase, the features used are negotiated and detailed configurations are exchanged. The AI ​​Media AF receives service access information for A5 (AI Model Session Handling), where AI Model data hosting is negotiated, and also receives service access information for A2 (Ingestion) and A4 (AI Model Data Delivery). The AI ​​Media Client requires this information to access the service. Depending on the provisioning, only a reference to the service access information may be provided.

[0151] 2. When AI model data hosting is provided and selected, there may be interactions between the AI ​​Media AF and the AI ​​AS, such as for allocating AI model data ingestion and distribution resources. The AI ​​AS provides the resource identifiers of the allocated resources to the AI ​​Media AF, which then provides this information to the AI ​​Media Application Provider.

[0152] 3. The AI ​​media application provider starts the ingestion session by ingesting the AI ​​model data. In the case of a dynamic AI model AI media service, the AI ​​model data may be ingested continuously or may be uploaded once and then updated later.

[0153] 4. The AI ​​media application provider provides service announcement information to the AI ​​media application. The service announcement includes overall service access information (i.e., details of AI model session processing (A5) and AI model data transfer access (A4), such as including an AI profile) or a reference to the service access information or pre-configured information. When only a reference is included, the AI ​​media client (in step 6) retrieves the service access information (including, for example, an AI profile) when necessary.

[0154] 5. When the AI ​​media application decides to start receiving AI model data, service access information (full or reference) is provided to the AI ​​media client. The AI ​​media client activates a unicast downlink delivery session.

[0155] 6. Optional) In case the AI ​​Media Client only receives a reference to the service access information, it then obtains the service access information from the AI ​​Media AF.

[0156] 7. The AI ​​Media Client uses the AI ​​Model Data Session Processing API exposed by the AI ​​Media AF at A5. The AI ​​Model Data Session Processing API is used to configure consumption measurement, logging, collection, and reporting; configure QoE metric measurement, logging, collection, and reporting; request different policies and billing processing; or network assistance based on the AI ​​Media AF. The actual time of API usage may depend on the features and interactions used during media content reception.

[0157] 8. The AI ​​media client activates the reception of AI model data.

[0158] For all embodiments of the present invention, the service announcement includes any of the following:

[0159] 1) Overall service access information (i.e., details of AI model session processing (A5) and / or media session processing (M5) and details of AI model data delivery access (A4) and / or media content data access (M5), including AI profiles), or

[0160] 2) A reference to service access information or pre-configured information.

[0161] The AI ​​Media Client requires this information to access the service.

[0162] Fig.13 Shows Figure 8 The basic workflow process of the scene in .

[0163] step:

[0164] 1. The application contacts the application provider to retrieve the entry point of the AI ​​media service. The acquisition of the entry point can be performed in different ways. The entry point can be, for example, a URL to an AI configuration file.

[0165] 2. Session settings:

[0166] 2a. In the case where the entry point is the URL of the AI ​​configuration file, the application uses the obtained entry point to initialize the AI ​​model manager.

[0167] 2b. The AI ​​model manager retrieves the AI ​​configuration file from the AI ​​model provider based on the entry point information.

[0168] The AI ​​configuration file contains:

[0169] *General description of AI media services

[0170] * A list of selected AI models (indicated by identifiers) that can be used for AI media services.

[0171] *For each AI model in the list:

[0172] - One or more identifiers (e.g., one or more URLs) pointing to the location of the actual AI model data

[0173] - A general description of the AI ​​model, including metadata describing different parameters of the AI ​​model, such as the model neural network type (CNN, DNN, etc.), topology information, number of layers, whether the model contains full or partial layer data, number of filters, kernels, biases, quantized weights, tensors, etc.

[0174] - The AI ​​configuration file may also contain information related to the delivery of available AI models. Specifically, whether AI model data can be downloaded as a whole file (or similar), whether AI models can be delivered as partial models (and whether they can be inferred separately), whether AI model data can be delivered according to different data types (such as topological data, parameter data (weights, biases, etc.), etc.).

[0175] - The AI ​​profile may also contain information about possible inference configurations that are possible for the AI ​​media service, such as possible network-UE split inference configurations, which may or may not depend on the selected AI model.

[0176] In one embodiment of the present invention, the AI ​​Media AF may receive information about the capabilities, status, and requirements of the UE from the UE (such as processing capabilities, media requirements for AI model processing, model inference buffer capacity, etc.). The AI ​​Media AF may also receive information about the UE / network status from the 5GS, such as network conditions (e.g., congestion, bandwidth, latency), UE location, UE charging policy, etc., and other relevant information. The AI ​​Media AF then utilizes this information (from the UE and 5GS) to select an AI model for the AI ​​media service and provide the UE with relevant AI model configuration information (e.g., model location URL, model general description, etc.).

[0177] In another embodiment, the AI ​​Media AF uses this information to pre-select an appropriate AI model set from the AI ​​models supplied by the AI ​​media application provider. The AI ​​Media AF then updates the AI ​​configuration file to include configuration information related to the selected AI model (e.g., AI model data location URL, AI model general description attributes such as model type, number of layers, etc.), and sends the updated AI configuration file to the UE. Using the updated configuration file, the AI ​​model manager in the UE selects an appropriate AI model for the service.

[0178] In another embodiment, the AI ​​Media AF does not perform pre-selection of the AI ​​model, but instead sends relevant network status / condition information (obtained from the 5GS) to the UE. The UE then uses all available information (including information available within the UE) upon receiving both the AI ​​profile and the network information in order to decide and select the appropriate AI model for the service.

[0179] 2c. The AI ​​model manager parses the AI ​​configuration file and processes the AI ​​configuration file. The AI ​​model manager may also select an appropriate AI model for the AI ​​media service based on the AI ​​configuration file. It may also determine the required AI model data delivery pipeline required for the selected AI model during this step.

[0180] 2d. The AI ​​Model Manager requests the creation of an AI Runtime Session from the AI ​​Engine Runtime.

[0181] 2e. The AI ​​Engine Runtime initializes the AI ​​Runtime session to prepare for the ingestion of the AI ​​model for inference.

[0182] 2f. The AI ​​Engine runtime notifies the AI ​​Model Manager of its readiness status.

[0183] 3. The AI ​​Model Manager may select an appropriate AI model for the AI ​​media service based on the AI ​​Profile (if it was not done during step 2c). The AI ​​Model Manager also uses information from the AI ​​Profile during this step to decide the required AI model data delivery pipeline (including metadata) needed for the selected AI model.

[0184] 4. AI Model Delivery Pipelines: Possible pipelines for AI model data delivery, as determined in step 3. Details are given in Figures 14a and 14b. Once established, these AI model delivery pipelines enable delivery of AI model data from the AI ​​AS to the UE via A4.

[0185] 5. For AI media services, UE uses relevant media as input to infer the AI ​​model.

[0186] 6. The UE may send a status report about its AI media service status (e.g., AI reasoning status, latency, resource status, capability status, dynamic media attributes, etc.) to the AI ​​media AF. This step may occur before step 3 or even before step 2c (i.e., before selecting the AI ​​model and its delivery pipeline), so that this information can be shared and used for the decision processing in step 3 and step 2c.

[0187] 7. The AI ​​Media AF may send network-related status / condition reports to the UE. This step may occur before step 3 or even before step 2c (i.e., before selecting the AI ​​model and its delivery pipeline), so that this information can be shared and used for the decision processing in steps 3 and 2c.

[0188] Steps 6 and 7 enable sharing of UE and network state information between the two endpoints. Such information can be used by the UE or the network to make decisions during AI media services regarding the selection of an AI model according to the AI ​​profile, the selection of multiple required delivery pipelines, any required split inference configuration (if necessary), and any updates to the AI ​​profile itself, or any of the decisions listed above.

[0189] 8. Based on the internal functions, conditions and information from step 6 and / or step 7, the UE may reselect the AI ​​model for the AI ​​media service, and / or may also reselect / redetermine / update the transmission pipeline for AI model data transmission.

[0190] Figures 14a and 14b show Fig.13 Various AI model delivery pipelines in step 4 of .

[0191] The UE or the network may decide to set up and use various different pipelines to transmit AI model data from the network to the UE.

[0192] Download the overall AI model:

[0193] The overall AI model may be transmitted to the UE in a download manner (eg, download of a file), wherein inference on the UE is performed only after all data of the overall AI model is received.

[0194] Part of the AI ​​model loop:

[0195] The AI ​​model can be transmitted in a partial download manner. For transmission in a partial manner, the AI ​​model is divided into multiple partial models in the network. Depending on the implementation, such partial models may or may not be individually inferable. The partial models can be transmitted to the UE in a streaming-like manner, where each partial model is transmitted continuously. For individually inferable partial models, reasoning on the UE can occur before receiving the overall AI model data (for example, UE reasoning of the first partial model can occur after receiving the first partial model data and before receiving the data of the second partial model).

[0196] AI topology loop:

[0197] The AI ​​topology data of the AI ​​model may be transmitted separately from the rest of the data associated with the AI ​​model.

[0198] AI parameter loop:

[0199] Data for AI parameters (eg, node parameters such as weights, biases, etc.) may be transmitted separately from other data associated with the AI ​​model (ie, separately from topology data).

[0200] One or more possible delivery pipelines may be selected and set for AI model data delivery, and the loop frequency of each pipeline may be different depending on the AI ​​model characteristics and properties specified by the data in the AI ​​configuration file (e.g., a parameter pipeline may loop at a frequency corresponding to the required parameter updates over the duration of the AI ​​media service).

[0201] Fig.15 Shown with Fig. 9 The high-level process of AL model downlink transmission and media content stream transmission corresponding to the scenario in the figure, where the AI ​​media service involves both AI model transmission and media stream transmission to the UE.

[0202] step:

[0203] 1. The AI ​​media application provider creates a provisioning session with the AI ​​Media AF and starts provisioning the use of the AI4Media system (and the 5GMS system for media streaming). During the establishment phase, the features used are negotiated and detailed configurations are exchanged. The AI ​​Media AF receives service access information for A5 and M5 (AI model session processing and media session processing), where AI model data and media content hosting are negotiated, and also receives service access information for A2 / M2 (ingestion) and A4 / M4 (AI model data delivery and media content delivery). The AI ​​media client needs this information to access the service. Depending on the provisioning, only a reference to the service access information may be provided.

[0204] 2. When AI model data hosting is provided and selected, there may be interaction between the AI ​​Media AF and the AI ​​AS / Media AS, for example, for allocating both AI model data and media content ingestion and distribution resources. The AI ​​AS and Media AS provide resource identifiers of the allocated resources to the AI ​​Media AF, which then provides this information to the AI ​​Media Application Provider.

[0205] 3. The AI ​​media application provider starts an AI model ingestion session by ingesting AI model data. In the case of dynamic AI model AI media services, AI model data can be ingested continuously or can be uploaded once and then updated later.

[0206] 4. The AI ​​media application provider starts a media ingestion session by ingesting media content. In the case of real-time media services, the media content can be ingested continuously or can be uploaded once and then updated later.

[0207] 5. The AI ​​media application provider provides service announcement information to the AI ​​media application. The service announcement includes overall service access information (i.e., details of AI model session processing (A5), media session processing (M5), and details of AI model data delivery access (A4), media content delivery access (M4), such as including AI configuration files) or references to service access information or pre-configured information. When only a reference is included, the AI ​​media client (in step 6) retrieves the service access information (including, for example, an AI configuration file) when necessary.

[0208] 6. When the AI ​​media application decides to start receiving AI model data and media content data, the service access information (all or reference) is provided to the AI ​​media client. The AI ​​media client activates a unicast downlink delivery session.

[0209] 7. Optional) In case the AI ​​Media Client only receives a reference to the service access information, it then obtains the service access information from the AI ​​Media AF.

[0210] 8. The AI ​​Media Client uses the AI ​​Model Data Session Processing API exposed by the AI ​​Media AF at A5. The AI ​​Model Data Session Processing API is used to configure consumption measurement, logging, collection, and reporting; configure QoE metric measurement, logging, collection, and reporting; request different policies and billing processing; or network assistance based on the AI ​​Media AF. The actual time of API usage may depend on the features and interactions used during the reception of media content.

[0211] 9. The AI ​​media client activates the reception of AI model data.

[0212] 10. The AI ​​Media Client uses the Media Session Handling API exposed by the AI ​​Media AF at M5. The Media Session Handling API is used to configure consumption measurement, logging, collection, and reporting; configure QoE metric measurement, logging, collection, and reporting; request different policies and billing processing; or network assistance based on the AI ​​Media AF. The actual time of API usage may depend on the features and interactions used during the reception of media content.

[0213] 11. The AI ​​media client activates reception of content data.

[0214] 12. The media player passes the media content of the received stream to the AI ​​client.

[0215] 13. The AI ​​client performs AI inference on the received AI model using the streamed media content as input to the model.

[0216] Fig.16 Shows Fig. 9 The basic workflow process of the scene in is Fig.13 An extension of the basic workflow process described in .

[0217] Fig.16 Steps 1 to 3 correspond to Fig.13 Steps 1 to 3, Fig.16 Steps 7 to 9 correspond to Fig.13 Follow steps 5 to 7. Fig.16 Step 11 corresponds to Fig.13 Step 8.

[0218] The AI ​​configuration file contains:

[0219] *General description of AI media services

[0220] * A list of selected AI models (indicated by identifiers) that can be used for AI media services.

[0221] *For each AI model in the list:

[0222] - One or more identifiers (e.g., one or more URLs) pointing to the location of the actual AI model data

[0223] - A general description of the AI ​​model, including metadata describing different parameters of the AI ​​model, such as the model neural network type (CNN, DNN, etc.), topology information, number of layers, whether the model contains full or partial layer data, number of filters, kernels, biases, quantized weights, tensors, etc.

[0224] - The AI ​​configuration file may also contain information related to the delivery of available AI models. Specifically, whether AI model data can be downloaded as a whole file (or similar), whether AI models can be delivered as partial models (and whether they can be inferred separately), whether AI model data can be delivered according to different data types (such as topological data, parameter data (weights, biases, etc.), etc.).

[0225] - The AI ​​profile may also contain information about possible inference configurations that are possible for the AI ​​media service, such as possible network-UE split inference configurations, which may or may not depend on the selected AI model.

[0226] In one embodiment of the present invention, the AI ​​Media AF may receive information about the capability status and requirements of the UE from the UE (such as processing capability, media requirements for AI model processing, model inference buffer capacity, etc.). The AI ​​Media AF may also receive information about the UE / network status from the 5GS, such as network conditions (e.g., congestion, bandwidth, latency), UE location, UE charging policy, etc., and other relevant information. The AI ​​Media AF then utilizes this information (from the UE and 5GS) to select an AI model for the AI ​​media service and provide the UE with relevant AI model configuration information (e.g., model location URL, model general description, etc.).

[0227] In another embodiment, the AI ​​Media AF uses this information to pre-select an appropriate AI model set from the AI ​​models supplied by the AI ​​media application provider. The AI ​​Media AF then updates the AI ​​configuration file to include configuration information related to the selected AI model (e.g., AI model data location URL, AI model general description attributes such as model type, number of layers, etc.), and sends the updated AI configuration file to the UE. Using the updated configuration file, the AI ​​model manager in the UE selects an appropriate AI model for the service.

[0228] In another embodiment, the AI ​​Media AF does not perform pre-selection of the AI ​​model, but instead sends relevant network status / condition information (obtained from the 5GS) to the UE. The UE then uses all available information (including information available within the UE) upon receiving both the AI ​​profile and the network information in order to decide and select the appropriate AI model for the service.

[0229] Fig.13 All the different embodiments described in the Fig.16 Details in.

[0230] At the most basic level of using the media delivery pipeline defined in 5GMS (TS26.501), in addition to delivering AI model data through the AI ​​model delivery pipeline, Fig.16 It also includes transmitting media content data from the network to the UE through the media transmission pipeline.

[0231] In step 3, the AI ​​client may also determine the media delivery pipeline required for the AI ​​media session.

[0232] step:

[0233] 4. See Figure 14a and Figure 14b.

[0234] 5. Media delivery pipeline: A possible pipeline for delivering media content data from a media AS to an AI media client. The delivery can be in the form of media streaming, or download delivery, or any other form of delivery mechanism required by the AI ​​media service. One such typical service is defined by 5GMS (TS 26.501).

[0235] 6. Upon receiving the media data, the media player sends the media data to the AI ​​client to be used as input for AI reasoning. Any media processing required before being used as input to the AI ​​model is also performed at this stage in the media player or in any other related entity (e.g., media decoding, downscaling / upscaling, etc.).

[0236] 10. The AI ​​client may receive media status reports from the network or from within the media player (via the AI ​​media session handler) for use in updating AI model selection decisions and / or also for updating decisions on the model delivery pipeline for AI model data and also for media content data.

[0237] Fig.17 Shown with Fig.10 A high-level process of split reasoning AI media service corresponding to the scenario in , where the AI ​​media service involves AI model reasoning in the network and also involves AI model reasoning in the UE (where the UE receives the IE AI model and also receives intermediate data from the network).

[0238] step:

[0239] 1. The AI ​​media application provider creates a provisioning session with the AI ​​Media AF and starts provisioning the use of the AI4Media system (and the 5GMS system for media streaming). During the establishment phase, the features used are negotiated and detailed configurations are exchanged. The AI ​​Media AF receives service access information for A5 and M5 (AI model session processing and media session processing), where AI model data and media content hosting are negotiated, and also receives service access information for A2 / M2 (ingestion) and A4 / M4 (AI model data delivery and media content delivery). The AI ​​media client needs this information to access the service. Depending on the provisioning, only a reference to the service access information may be provided.

[0240] 2. When AI model data hosting is provided and selected, there may be interaction between the AI ​​Media AF and the AI ​​AS / Media AS, for example, for allocating both AI model data and media content ingestion and distribution resources. The AI ​​AS and Media AS provide resource identifiers of the allocated resources to the AI ​​Media AF, which then provides this information to the AI ​​Media Application Provider.

[0241] 3. The AI ​​media application provider starts the AI ​​model ingestion session by ingesting the AI ​​model data. In the case of dynamic AI model AI media services, the AI ​​model data can be ingested continuously or can be uploaded once and then updated later. For segmented reasoning between the network and the UE, the UE AI model and the network AI model required by the AI ​​media service are ingested.

[0242] 4. The AI ​​media application provider starts a media ingestion session by ingesting media content. In the case of real-time media services, the media content can be ingested continuously or can be uploaded once and then updated later.

[0243] 5. The AI ​​media application provider provides service announcement information to the AI ​​media application. The service announcement includes overall service access information (i.e., details of AI model session processing (A5), media session processing (M5), and details of AI model data delivery access (A4), media content delivery access (M4), such as including AI configuration files) or a reference to service access information or pre-configured information. When only a reference is included, the AI ​​media client (in step 6) retrieves the service access information (including, for example, an AI configuration file) when necessary.

[0244] 6. When the AI ​​media application decides to start receiving AI model data and media content data, the service access information (all or reference) is provided to the AI ​​media client. The AI ​​media client activates a unicast downlink delivery session.

[0245] 7. Optional) In case the AI ​​Media Client receives a reference to the service access information only, it then obtains the service access information from the AI ​​Media AF.

[0246] 8. The AI ​​Media Client uses the AI ​​Model Data Session Processing API exposed by the AI ​​Media AF at A5. The AI ​​Model Data Session Processing API is used to configure consumption measurement, logging, collection, and reporting; configure QoE metric measurement, logging, collection, and reporting; request different policies and billing processing; or network assistance based on the AI ​​Media AF. The actual time of API usage may depend on the features and interactions used during the reception of media content.

[0247] 9. The AI ​​media client activates the reception of UE AI model data.

[0248] 10. For intermediate data to be sent from the network to the UE, the AI ​​media client uses the media session handling API exposed by the AI ​​media AF at M5. The media session handling API is used to configure consumption measurement, logging, collection, and reporting; configure QoE metric measurement, logging, collection, and reporting; request different policies and billing processing; or network assistance based on the AI ​​media AF. The actual time of API usage may depend on the features and interactions used during the reception of media content.

[0249] 11. The media AS passes the ingested media data to the AI ​​engine in the network as input for network AI reasoning. In the case where the network AI model has not been previously passed to the AI ​​engine (e.g., during ingestion), the AI ​​AS or media AS may also pass the network AI model to the network AI engine at this stage.

[0250] 12. Upon receiving the media data, the network AI engine uses the media data in step 11 as input to perform network AI reasoning.

[0251] 13. The network AI model inference output data (intermediate data) from step 12 is sent to the media AS to be ready for transmission to the UE.

[0252] 14. The AI ​​media client and / or media AS activates the transmission of intermediate data. Depending on the characteristics of the AI ​​media service, the intermediate data may also be media data.

[0253] 15. The media player passes the intermediate data / media data to the AI ​​client.

[0254] 16. The UE AI client performs UE-side device AI inference on the received UE AI model (received in step 9) using the received intermediate data from step 14 as input to the model.

[0255] Fig.18 Shows Fig. 9 The basic workflow process of the scene in is Fig.16 An extension of the basic workflow process described in .

[0256] Fig.18 Steps 1 to 3 correspond to Fig.16 Steps 1 to 3, Fig.18 Step 5 corresponds to Fig.16 Step 4, Fig.18 Step 10 corresponds to Fig.16 Step 5, Fig.18 Step 11 corresponds to Fig.16 Step 6, Fig.18 Step 12 corresponds to Fig.16 Step 6, and so on.

[0257] The AI ​​configuration file contains:

[0258] *General description of AI media services

[0259] * A list of selected AI models (indicated by identifiers) that can be used for AI media services.

[0260] *For each AI model in the list:

[0261] - One or more identifiers (e.g., one or more URLs) pointing to the location of the actual AI model data

[0262] - A general description of the AI ​​model, including metadata describing different parameters of the AI ​​model, such as the model neural network type (CNN, DNN, etc.), topology information, number of layers, whether the model contains full or partial layer data, number of filters, kernels, biases, quantized weights, tensors, etc.

[0263] - The AI ​​configuration file may also contain information related to the delivery of available AI models. Specifically, whether AI model data can be downloaded as a whole file (or similar), whether AI models can be delivered as partial models (and whether they can be inferred separately), whether AI model data can be delivered according to different data types (such as topological data, parameter data (weights, biases, etc.), etc.).

[0264] - The AI ​​profile may also contain information about possible inference configurations that are possible for the AI ​​media service, such as possible network-UE split inference configurations, which may or may not depend on the selected AI model.

[0265] In one embodiment of the present invention, the AI ​​Media AF may receive information about the capability status and requirements of the UE from the UE (such as processing capability, media requirements for AI model processing, model inference buffer capacity, etc.). The AI ​​Media AF may also receive information about the UE / network status from the 5GS, such as network conditions (e.g., congestion, bandwidth, latency), UE location, UE charging policy, etc., and other relevant information. The AI ​​Media AF then utilizes this information (from the UE and 5GS) to select an AI model for the AI ​​media service and provide the UE with relevant AI model configuration information (e.g., model location URL, model general description, etc.).

[0266] In another embodiment, the AI ​​Media AF uses this information to pre-select an appropriate AI model set from the AI ​​models supplied by the AI ​​media application provider. The AI ​​Media AF then updates the AI ​​configuration file to include configuration information related to the selected AI model (e.g., AI model data location URL, AI model general description attributes such as model type, number of layers, etc.), and sends the updated AI configuration file to the UE. Using the updated configuration file, the AI ​​model manager in the UE selects an appropriate AI model for the service.

[0267] In another embodiment, the AI ​​Media AF does not perform pre-selection of the AI ​​model, but instead sends relevant network status / condition information (obtained from the 5GS) to the UE. The UE then uses all available information (including information available within the UE) upon receiving both the AI ​​profile and the network information in order to decide and select the appropriate AI model for the service.

[0268] Fig.13 All the different embodiments described in the Fig.18 Details in.

[0269] At the most basic level of using the media delivery pipeline defined in 5GMS (TS26.501), in addition to delivering AI model data through the AI ​​model delivery pipeline, Fig.18 It also includes the transmission of intermediate data from the network to the UE through the media transmission pipeline.

[0270] In step 3, the AI ​​client may also determine the media delivery pipeline required for the AI ​​media session.

[0271] Step 4: The AI ​​client notifies the network of the requested desired split inference configuration, as decided in step 3.

[0272] Steps 6-9: The required network AI model data (e.g., core model) is sent to the network AI engine, and the ingested media data is also sent to the AI ​​engine. The network AI engine then uses this data to perform AI model inference, and sends the output intermediate data to the UE in step 10 via the intermediate data transmission pipeline.

[0273] The methods of the embodiments described in the claims and specification of the present disclosure may be implemented in the form of hardware, software, or a combination of hardware and software.

[0274] When it is implemented by software, a computer-readable storage medium for storing one or more programs (software modules) may be provided. One or more programs stored in the computer-readable storage medium are configured to be run by one or more processors in the electronic device. One or more programs include instructions that cause the electronic device to execute the methods of the embodiments described in the claims and specification of the present disclosure.

[0275] Such a program (software module, software) may be stored in a random access memory, a non-volatile memory including flash memory, a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a magnetic disk storage device, a compact disc ROM (CD-ROM), a digital versatile disk (DVDs) or other optical storage device, a magnetic tape cartridge. Or, it may be stored in a memory including a combination of some or all of these. Moreover, each constituent memory may include a plurality.

[0276] Furthermore, the program may be stored in an attachable storage device that can be accessed through a communication network such as the Internet, an intranet, a local area network (LAN), a wide area network (WAN), or a storage area network (SAN), or a combination thereof. Such a storage device may be connected to the apparatus for executing the embodiments of the present disclosure via an external port. Furthermore, a separate storage device on a communication network may be connected to the apparatus for executing the embodiments of the present disclosure.

[0277] In the specific embodiments of the present disclosure as described above, the components included in the present disclosure are expressed in the singular or plural, depending on the specific embodiment provided. However, for the convenience of description, the expression of the singular or plural is appropriately selected for the provided situation, and the present disclosure is not limited to singular or plural components, even components expressed in the plural are composed of the singular, or even components expressed in the singular can be composed of the plural.

[0278] Meanwhile, in the detailed description of the present disclosure, specific embodiments have been described, but various modifications may be made without departing from the scope of the present disclosure. Therefore, the protection scope of the present disclosure should not be limited to and defined by the described embodiments, but should be defined by the scope of the attached claims and the equivalents of the scope of these claims.

Claims

1. A method performed by a user equipment UE in a wireless communication system, the method comprising: receiving information about at least one artificial intelligence (AI) model from a first network entity; determining an AI model based on the information about the at least one AI model; Determining whether to use the AI ​​model for AI segmentation reasoning service; Requesting the AI ​​segmentation inference service from the first network entity; establishing an AI model delivery pipeline for the AI ​​model; and A media delivery pipeline is established for delivering media data used in the AI ​​model.

2. The method according to claim 1, in, The information about the at least one AI model includes a uniform resource locator (URL) for obtaining a list of the at least one AI model.

3. The method according to claim 1, further comprising: Intermediate data is received from a second network entity on the media delivery pipeline.

4. The method according to claim 1, further comprising: Send a status report of information about the AI ​​segmentation inference service to the first network function.

5. The method according to claim 4, further comprising: The AI ​​model and the AI ​​model delivery pipeline are updated based on the information about the AI ​​segmentation inference service and the information about the network status.

6. A method performed by a first network entity in a wireless communication system, the method comprising: Sending information about at least one artificial intelligence AI model to a user equipment UE; identifying an AI model determined based on the information about the at least one AI model; receiving a request for an AI segmentation inference service using the AI ​​model; establishing an AI model delivery pipeline for the AI ​​model; and A media delivery pipeline is established for delivering media data used in the AI ​​model.

7. The method according to claim 6, in, The information about the at least one AI model includes a uniform resource locator (URL) for obtaining a list of the at least one AI model.

8. The method according to claim 6, further comprising: A status report is received from the UE regarding information about the AI ​​segmentation inference service.

9. The method according to claim 8, further comprising: The AI ​​model and the AI ​​model delivery pipeline are updated based on the information about the AI ​​segmentation inference service and the information about the network status.

10. The method according to claim 6, further comprising: An inference process in the second network entity is triggered.

11. A user equipment UE in a wireless communication system, the UE comprising: at least one transceiver; as well as at least one processor operatively coupled to the at least one transceiver, Wherein, the at least one processor is configured to: receiving information about at least one artificial intelligence AI model from a first network entity, determining an AI model based on the information about the at least one AI model, determining whether to use the AI ​​model for AI segmentation inference services, requesting the AI ​​segmentation inference service from the first network entity, establishing an AI model delivery pipeline for the AI ​​model, and A media delivery pipeline is established for delivering media data used in the AI ​​model.

12. The UE according to claim 11, in, The information about the at least one AI model includes a uniform resource locator (URL) for obtaining a list of the at least one AI model.

13. The UE according to claim 11, wherein: The at least one processor is further configured to: Intermediate data is received from a second network entity on the media delivery pipeline.

14. The UE according to claim 11, wherein: The at least one processor is further configured to: Send a status report of information about the AI ​​segmentation inference service to the first network function.

15. A first network entity in a wireless communication system, the first network entity comprising: at least one transceiver; as well as at least one processor operatively coupled to the at least one transceiver, Wherein, the at least one processor is configured to: Sending information about at least one artificial intelligence AI model to a user equipment UE, identifying an AI model determined based on the information about the at least one AI model, receiving a request for an AI segmentation inference service using the AI ​​model, establishing an AI model delivery pipeline for the AI ​​model, and A media delivery pipeline is established for delivering media data used in the AI ​​model.

Images