Method and apparatus for transmitting and receiving avatar model

Abstract

The present disclosure relates to a 5G communication system for supporting a higher data transmission rate. An operation method of a transmission UE that transmits an avatar model, according to an embodiment of the present disclosure, may comprise the steps of: partitioning the avatar model into at least one piece of avatar data and at least one avatar manifest; acquiring at least one transmission manifest on the basis of the at least one avatar manifest; transmitting the at least one transmission manifest to a reception UE; and on the basis of reception information of the reception UE, transmitting the at least one piece of avatar data to the reception UE.

Description

Avatar model transmission and reception method and device

[0001] The present disclosure relates to a method and device for avatar communication. Specifically, the present disclosure relates to a method and device for configuring an avatar manifest and transmitting and receiving an avatar model between terminals based on the avatar manifest.

[0002] 5G mobile communication technology defines a wide frequency band to enable fast transmission speeds and new services, and can be implemented not only in the sub-6GHz frequency band such as 3.5 gigahertz (3.5GHz), but also in the ultra-high frequency band called millimeter wave (mmWave) such as 28GHz and 39GHz ('Above 6GHz'). In addition, for 6G mobile communication technology, which is called the system after 5G communication (Beyond 5G), implementation in the terahertz band (for example, the 3 terahertz (3THz) band at 95GHz) is being considered to achieve a transmission speed that is 50 times faster than 5G mobile communication technology and an ultra-low latency time that is reduced to one-tenth.

[0003] In the early stages of 5G mobile communication technology, the goal is to support services and satisfy performance requirements for enhanced Mobile Broadband (eMBB), Ultra-Reliable Low-Latency Communications (URLLC), and massive Machine-Type Communications (mMTC). These include beamforming and massive MIMO to mitigate path loss of radio waves in ultra-high frequency bands and increase the transmission distance of radio waves, support for various numerologies (such as operation of multiple subcarrier intervals) and dynamic operation of slot formats for efficient use of ultra-high frequency resources, initial access technology to support multi-beam transmission and wideband, definition and operation of BWP (Bidth Part), new channel coding methods such as LDPC (Low Density Parity Check) codes for large-capacity data transmission and Polar Code for reliable transmission of control information, and L2 pre-processing (L2). Standardization has been made for network slicing, which provides dedicated networks specialized for specific services, and pre-processing.

[0004] Currently, discussions are underway to improve and enhance the initial 5G mobile communication technology in consideration of the services that 5G mobile communication technology was intended to support, and physical layer standardization is in progress for technologies such as V2X (Vehicle-to-Everything) to help autonomous vehicles make driving decisions and increase user convenience based on their own location and status information transmitted by vehicles, NR-U (New Radio Unlicensed) for the purpose of system operation that complies with various regulatory requirements in unlicensed bands, NR terminal low power consumption technology (UE Power Saving), Non-Terrestrial Network (NTN), which is direct terminal-satellite communication to secure coverage in areas where communication with terrestrial networks is impossible, and Positioning.

[0005] In addition, standardization of wireless interface architecture / protocols is in progress for technologies such as intelligent factories (Industrial Internet of Things, IIoT) to support new services through linkage and convergence with other industries, Integrated Access and Backhaul (IAB) that provides nodes for expanding network service areas by integrating wireless backhaul links and access links, Mobility Enhancement technology including Conditional Handover and Dual Active Protocol Stack (DAPS) handover, and 2-step random access (2-step RACH for NR) that simplifies random access procedures. Standardization is also in progress for system architecture / services such as 5G baseline architecture (e.g., Service-based Architecture, Service-based Interface) for grafting Network Functions Virtualization (NFV) and Software-Defined Networking (SDN) technologies, and Mobile Edge Computing (MEC) that provides services based on the location of the terminal.

[0006] Once these 5G mobile communication systems are commercialized, an explosive increase in connected devices will be connected to the communication network, necessitating enhanced functionality and performance of 5G mobile communication systems and integrated operation of these connected devices. To this end, new research will be conducted on improving 5G performance and reducing complexity, supporting AI services, supporting metaverse services, and drone communications by utilizing eXtended Reality (XR), Artificial Intelligence (AI), and Machine Learning (ML) to efficiently support Augmented Reality (AR), Virtual Reality (VR), and Mixed Reality (MR).

[0007] In addition, the development of these 5G mobile communication systems includes new waveforms to ensure coverage in the terahertz band of 6G mobile communication technology, multi-antenna transmission technologies such as Full Dimensional MIMO (FD-MIMO), Array Antenna, and Large Scale Antenna, metamaterial-based lenses and antennas to improve the coverage of terahertz band signals, high-dimensional spatial multiplexing technology using Orbital Angular Momentum (OAM), Reconfigurable Intelligent Surface (RIS) technology, as well as full duplex technology to improve the frequency efficiency and system network of 6G mobile communication technology, satellite, AI (Artificial Intelligence) from the design stage and AI-based communication technology that realizes system optimization by internalizing end-to-end AI support functions, and ultra-high-performance communication and computing resources to provide services with complexity that exceeds the limits of terminal computing capabilities. It can serve as a basis for the development of next-generation distributed computing technologies that can be realized by utilizing them.

[0008] As described above and with the development of wireless communication systems, various services have become available, and methods for providing these services smoothly are required.

[0009] An operation method of a transmitting terminal for transmitting an avatar model according to one embodiment of the present disclosure may include a step of dividing the avatar model into at least one avatar data and at least one avatar manifest.

[0010] A method of operating a transmitting terminal for transmitting an avatar model according to one embodiment of the present disclosure may include a step of obtaining at least one transmission manifest based on at least one avatar manifest.

[0011] An operation method of a transmitting terminal for transmitting an avatar model according to one embodiment of the present disclosure may include a step of transmitting at least one transmission manifest to a receiving terminal.

[0012] An operation method of a transmitting terminal for transmitting an avatar model according to one embodiment of the present disclosure may include a step of transmitting at least one avatar data to the receiving terminal based on reception information of the receiving terminal.

[0013] A transmitting terminal for transmitting an avatar model according to one embodiment of the present disclosure may include a transceiver and at least one processor.

[0014] According to one embodiment of the present disclosure, the at least one processor may divide the avatar model into at least one avatar data and at least one avatar manifest.

[0015] According to one embodiment of the present disclosure, the at least one processor can obtain at least one transmission manifest based on the at least one avatar manifest.

[0016] According to one embodiment of the present disclosure, the at least one processor can transmit the at least one transmission manifest to a receiving terminal.

[0017] According to one embodiment of the present disclosure, the at least one processor may transmit the at least one avatar data to the receiving terminal based on the reception information of the receiving terminal.

[0018] FIG. 1 is a diagram illustrating the structure of a terminal that transmits and receives an avatar model according to one embodiment of the present disclosure.

[0019] FIG. 2 is a diagram illustrating a transmission path of an avatar transmission manifest and avatar data according to a connection method according to one embodiment of the present disclosure.

[0020] FIG. 3 is a diagram illustrating a transmission path of an avatar transmission manifest and avatar data according to a connection method according to one embodiment of the present disclosure.

[0021] FIG. 4 is a diagram illustrating a signal flow for media transmission according to one embodiment of the present disclosure.

[0022] FIG. 5 is a diagram illustrating a signal flow for media transmission according to one embodiment of the present disclosure.

[0023] The terms used in this specification will be briefly explained, and the present invention will be described in detail.

[0024] The terms used in this invention have been selected from widely used, current terms, taking into account their functions. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, in which case their meanings will be described in detail in the relevant description of the invention. Therefore, the terms used in this invention should not be defined simply as names, but rather based on their inherent meanings and the overall content of the invention.

[0025] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the cases where it is "directly connected" but also the cases where it is "electrically connected" with another element in between.

[0026] When a part of the specification is said to "include" a component, this does not exclude other components, but rather implies the inclusion of other components, unless otherwise specifically stated. Furthermore, terms such as "part," "module," etc., used throughout the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software.

[0027] Additionally, the description 'at least one of A, B, and C' means that it can be any one of 'A', 'B', 'C', 'A and B', 'A and C', 'B and C', and 'A, B, and C'.

[0028] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, for the purpose of clearly explaining the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar parts are designated with similar reference numerals throughout the specification.

[0029] XR (eXtended Reality) is a technology for 3D graphic elements that are combined and played back in 3D space with the user's reality. Examples of XR content include applications that provide information based on the user's location or interaction in an airport or museum. When the user searches for an object (e.g., an art collection), a description of the object is displayed in a speech bubble in the space, an additional 3D video is played, or a corresponding content control loop is executed based on the user's interaction (e.g., an exit button).

[0030] In the above example, the animation, the execution of control loops by user interaction, etc. are all planned in advance and built into the XR content. When the user searches and selects, the set animation is expressed according to the planned time flow and then ends, and any new interaction events that are not described are processed as errors or ignored.

[0031] The technical background of the present invention is avatar communication, which combines the above-mentioned 3D content, mobile communication network, and user interaction. The basic scenario is that a first terminal captures the movements (gestures, facial expressions) of user 1, substitutes them with a series of commands, and transmits an avatar model, which is prepared in advance or created through continuous user filming and artificial intelligence learning, to a second terminal of user 2, the target of the avatar communication. The second terminal then manipulates the avatar model based on the received avatar model and commands, renders the modified model, and displays it on a display, so that user 2 can see user 1's avatar combined with the real world before his eyes as an XR object. The second terminal also performs the same task, causing the first terminal to display user 2's avatar, so that user 1 can also see user 2's avatar expressed in XR.

[0032] An avatar model can be composed of a mesh, a texture, and a skeleton. The mesh has points (vertexes), faces connecting three or more points, and vertex-to-texture mapping information. The texture has point color and face surface information. The skeleton expresses the structure of the human body in the form of a scene graph, where the connections between lines are joint connections between bones.

[0033] These avatar models can be generated based on various photos, videos, or other data acquired from the user's terminal or other sensor devices, and the device performing the generation can be the user's terminal or a separate specialized processing device provided by a service provider that provides the service to which the user has subscribed.

[0034] The above avatar model usually has different sizes depending on the user, and may also vary depending on the resolution or expressive capability of the display of the terminal that outputs the avatar, and may also vary depending on the technological maturity of the service provider.

[0035] In order for avatar model information to be used in avatar calls, especially real-time avatar calls, a step must first be taken where it is transmitted from the terminal of the user who owns the avatar model to the terminal of the other party. A size of 20MB can be considered very large data even from the perspective of 5G uplink traffic.

[0036] A similar service in the past was video calling. When attempting a video call, the other party accepts the call, and the connection is established. The video is immediately displayed. Therefore, the initial latency of avatar calls can be perceived as significantly lower than that of video calls, degrading QoS. Therefore, a method to reduce this initial latency is needed.

[0037] The ProSe layer discussed in the present disclosure may refer to the entire layer that performs control operations for device-to-device (or D2D, or ProSe: Proximity Service) communication services. This layer exists above the AS layer, and refers to a layer that exchanges signals with the AS layer for device-to-device communication connections. In addition, the layer may provide a data transmission function on a link established for direct communication between terminals. That is, it may include an IP protocol for message transmission, a non-IP protocol, a transport protocol (e.g., TCP or UDP), etc.

[0038] In addition, the terminal (UE) connects to the 5G Radio Access Network (5G-RAN) and accesses the core network devices of 5G. The core network of 5G can be composed of the following network functions. AMF (Access and Mobility Management Function) can refer to a network function that manages the mobility of the terminal. SMF (Session Management Function) can refer to a network function that manages the Packet Data Network connection provided to the terminal. This Packet Data Network connection can be referred to as a PDU (protocol data unit) session. PCF (Policy and Charging Function) can refer to a network function that applies the mobile communication service provider's service policy, charging policy, and policy for PDU sessions to the terminal. UDM is an abbreviation for Unified Data Management and can refer to a network function that stores and manages information about subscribers. NEF (Network Exposure Function) can access information that manages terminals in the 5G network, and can subscribe to mobility management events for the terminal, subscribe to session management events for the terminal, request session-related information, set charging information for the terminal, request change of PDU session policy for the terminal, etc. It can be connected to NF (Network Function) of the 5G core network and transmit information about the terminal to the NF or report information about the terminal to the outside. 5G-RAN can refer to a base station that provides wireless communication functions to the terminal. UPF is an abbreviation for User Plane Function and can act as a gateway that transmits packets transmitted and received by the terminal.UPF is connected to the Data Network and plays a role in transmitting data packets generated in the 5G system to an external Data Network. For example, it is connected to the Data Network connected to the Internet and can route data packets sent by terminals to the Internet.

[0039] ProSe (Proximity-based Service) Function refers to the logical function that performs the network operation required to provide Proximity Service. There can be one or more ProSe Functions in a PLMN (Public Land Mobile Network) and can be deployed to serve a specific area or serve specific terminals. ProSe Function can perform the Direct Provisioning function that provides the information required for a terminal to use the ProSe Service. This means provisioning the information required for the Direct Discovery function used for direct terminal-to-terminal discovery or the Direct Communication function used for direct terminal-to-terminal communication. In addition, ProSe Function can perform the function of managing the name or code used for Direct Discovery to support the Direct Discovery function used for direct terminal-to-terminal discovery. Additionally, the ProSe Function can trigger the terminal to perform Direct Discovery, and can also provide billing and security features for ProSe service usage. The terminal connects to the ProSe Function via the PC3 Interface defined by 3GPP.

[0040] One embodiment of the present disclosure proposes a method of configuring an avatar model into multiple independently executable steps to reduce initial latency.

[0041] One embodiment of the present disclosure proposes a system architecture of a transmitting terminal and a receiving terminal that can selectively receive and execute a step of the avatar model to reduce the initial delay time.

[0042] #1

[0043] One embodiment of the present disclosure proposes a method of configuring an avatar model into multiple independently executable steps to reduce initial latency.

[0044] An avatar model according to one embodiment of the present disclosure is composed of various types of avatar model data for expressing the avatar model, and an avatar model manifest, which is metadata that describes the relationship between each type of data.

[0045]

[0046] Types of avatar model data according to one embodiment of the present disclosure may include mesh, texture, skeleton, scene description, etc.

[0047]

[0048] Mesh is information for expressing a three-dimensional model in three-dimensional space using points (vertexes) and faces.

[0049] Texture is information for expressing color information of points and surfaces of the mesh, such as color or image.

[0050] Skeleton is information such as bones, perspectives, and facial features to express the structure of the human body.

[0051] Scene description is information for expressing the relationship between mesh, texture, and skeleton.

[0052] According to one embodiment of the present disclosure, multiple data items included in an avatar model may have alternative relationships with other data items for one data item. For example, for low-resolution texture data, medium-resolution texture data may have an alternative relationship.

[0053]

[0054] According to one embodiment of the present disclosure, a receiving terminal receives texture data of the lowest resolution for immediate avatar expression, and while displaying it, if an intermediate resolution is received using spare bandwidth, the avatar can be displayed using the intermediate resolution texture data instead of the previous low resolution texture. Another relationship among multiple data included in an avatar model according to one embodiment of the present disclosure is that one data may be supplementary to another data. For example, medium density mesh data may be supplementary to low density mesh data.

[0055]

[0056] According to one embodiment of the present disclosure, if a receiving terminal receives mesh data with the lowest density (e.g., 100 vertex information per unit space) for immediate avatar expression, and while displaying it, receives mesh data with medium density (e.g., 10,000 vertex information per unit space) using spare bandwidth, the space occupied by the previous low-density mesh data can be filled using both the low-density and medium-density mesh data. (For example, 10,100 vertex information per the same unit space is filled.) An avatar model manifest according to one embodiment of the present disclosure is characterized in that it includes one or more stages that a receiving terminal can select and receive, and includes information about each stage. The receiving terminal receives information about the stages, determines a stage to be received based on the transmission time required for each stage and the attribute information of each stage, and receives data necessary for avatar expression based on the determination. The data transmission information in the avatar model manifest is appropriately converted according to the mobile communication network protocol connected between the transmitting terminal and the receiving terminal, and is created and exchanged as an avatar transmission manifest. For this purpose, hints in the manifest can be provided for each transmission protocol.

[0057] An avatar model manifest according to one embodiment of the present disclosure may be configured as follows.

[0058]

[0059] Avatar model attributes may include sub-attribute information for distinguishing avatar models, such as model identifier, owner identifier, number of steps, and policy.

[0060]

[0061] Model identifiers are unique within a user, service provider, or across providers.

[0062] The owner identifier represents the object represented by the avatar model or the owner who owns the avatar model file. The owner identifier is unique within the service provider or across providers.

[0063] The number of steps indicates the number of steps provided by the manifest.

[0064] Model policies indicate the policies that the receiving terminal must follow when using the avatar model. Policies according to one embodiment of the present disclosure include delete_after_close and preserve_after_close. delete_after_close indicates that the avatar model must be deleted after the avatar call ends. preserve_after_close indicates that the avatar model is retained after the avatar call ends and can be used in the next call.

[0065] Step attributes are information that must be considered when transmitting data corresponding to each step, including step identifiers, size, semantics, quality, and step policies. Each step assumes that correct avatar representation is impossible if only a portion of the data is received. In other words, artifacts that occur when representing an avatar when only a portion of a step has been received are considered errors on the receiving terminal. The transmitting terminal determines whether to allow or prohibit avatar representation when only a portion of the data has been received through policy.

[0066]

[0067] A step identifier is a unique identifier for a step within an avatar model.

[0068] The step size is the sum total of the data sizes included in the step. The receiving terminal can determine reception based on the sum total of data, regardless of the number or type of data to be received. Specifically, sizes can be provided as lossless size and loss size. Lossless size refers to data that does not tolerate errors. It represents the sum total of data sizes that must be corrected through retransmission if an error occurs during the reception process. Loss size refers to data that tolerates errors. It represents the sum total of data sizes that do not require retransmission even if an error occurs during the reception process.

[0069] Semantics is a list of semantics included in a given step. Additionally, metadata_type can be specified to specify the scope of expression of the semantics. Depending on the transmission scope, one embodiment of the present disclosure proposes face, head, upper body with arms, and full body as basic semantics.

[0070] When a step attribute includes semantics and semantics, the receiving terminal can make a step reception decision based on the scope expressed by the step. For example, from the perspective of a receiving terminal that wishes to express a full body, if a certain step only contains a face, the terminal selects a step that contains more data and full-body semantics than that step.

[0071] Quality (qualities) indicates the quality information included in the corresponding step. Additionally, quality_type and quality_value can be specified to specify the range of quality expressions. Quality_type can include PSNR, resolution, and num_vertex.

[0072] PSNR indicates the lossy compression quality of texture data in image or video format compared to the original.

[0073] Resolution indicates the resolution of texture data in image or video format.

[0074] num_vertex indicates the number of vertices included in the mesh data.

[0075] The step policy indicates the policies that the receiving terminal must follow when receiving the steps of the avatar model. Policies according to one embodiment of the present disclosure may include partial_playback_allowed and partial_playback_disallowed. Partial_playback_allowed means that the receiving terminal can attempt to play the avatar model after receiving a portion of the step data. Partial_playback_disallowed means that the receiving terminal can attempt to play the avatar model only after receiving the step data.

[0076] A transmission hint can provide information for generating an SDP offer that a transmitting terminal provides to a receiving terminal to transmit the corresponding data for each step.

[0077]

[0078] A transmission hint can contain information for transmitting data belonging to each stage, for each stage included in the avatar model manifest. Each stage can include a stage identifier and data transmission information for one or more pieces of data belonging to the stage. The data transmission information can provide attribute information used when creating a message corresponding to the transmission protocol for each piece of data. Using this, the avatar model sender of the transmitting terminal can create a message suitable for the transmission protocol and transmit it to the network client.

[0079] The step identifier specifies the step to which the data information belongs.

[0080] Data information has m-line, a-line, access location and access size.

[0081] The m-line indicates the media type, protocol, and format of the data to be transmitted, referring to the syntax of the Session Description Protocol (hereinafter referred to as SDP) defined by the IETF. According to SDP, attributes are described starting from "m=" and in a set order, with attributes separated by spaces. Therefore, in one embodiment of the present disclosure, "m=media type protocol format" is described.

[0082] a-line describes additional properties for the data indicated by m-line. One embodiment of the present invention can use mid to indicate the media identifier property of data, and the data has a unique identifier for each different data within the avatar model. According to SDP, "a=mid: <value>"It is written in the same format as "a=mid:media identifier" in one embodiment of the present disclosure.

[0083] Media types defined in SDP include "audio", "video", "text", "application", and "message". For example, for Texture, the media type can be video, and for Mesh, the media type can be application.

[0084] The protocol is the recommended value for the protocol through which the data is transmitted. If the network to which the transmitting and receiving terminals belong does not support it, it can be replaced with another value. If supported, the protocol and format values ​​are used. For example, for mesh, UDP / DTLS / SCTP can be specified as the protocol and webrtc-datachannel as the format. For texture, for example, RTP / AVPF can be specified as the protocol and 99 as the format.

[0085] The access location indicates the path the avatar model transmitter uses to access and transmit data over the network. This path can be a location within the avatar model file, a location within the transmitter's storage, or a URL address for a network storage or Internet server outside the transmitter.

[0086] The access size indicates the start and end positions of the data in the above access location.

[0087] In the case of data outside the transmitting terminal, the access location part of the file of the external server is negotiated to be transmitted to a specific port of the external server address through communication between the avatar model transmitting unit and the external server, and the avatar model transmitting unit can transmit it to the receiving terminal by writing the address and port of the external server in the SDP.

[0088] After a transmission channel is established between a transmitting terminal and a receiving terminal, the avatar model transmitting unit can transmit the corresponding file to each transmission channel at the request of the receiving terminal.

[0089] An avatar model transmitter according to one embodiment of the present disclosure is characterized in that it constructs an SDP offer for each step using a transmission hint, the SDP offer includes data included in a step identifier described in the transmission hint, each data describes an m-line using information of an m-line, and in the format of an m-line, media, proto, and fmt are written using media type, protocol, and format information, respectively, and an a-line is described using mid information.

[0090] Other contents of the above-mentioned generated SDP offer message, such as port information of the m line or connection information of the c line, are not provided in the transmission hint because they use the available port and IP address information of the transmitting terminal. However, the avatar model transmitting unit according to one embodiment of the present disclosure is characterized by designating different unique ports for each different piece of data.

[0091] By using different ports for each data, a receiving terminal receiving multiple or integrated SDP offers described below has the advantage of being able to initiate data reception simultaneously regardless of whether reception of other data belonging to one or multiple stages has been completed.

[0092] A transmitting terminal can transmit a transmission protocol message including an avatar transmission manifest to a receiving terminal.

[0093] An avatar transfer manifest can consist of step information and an SDP offer corresponding to each step.

[0094] Depending on the communication architecture, multiple SDP offers may be allowed, or only a single SDP offer may be allowed.

[0095] According to one embodiment of the present disclosure, transmission manifest type 1 transmits model attributes and step information. A receiving terminal that receives the transmission manifest can select a desired step from the step information and request transmission for that step using the step identifier. The transmitting terminal transmits an SDP offer for the requested step, and the receiving terminal transmits an SDP answer, after which transmission is initiated from the transmitting terminal.

[0096]

[0097] According to one embodiment of the present disclosure, a transmission manifest type 2 can transmit model attributes, step information, and multiple SDP offers together. In this case, each SDP offer is distinguished by the field value of the session_id field in the o line of the SDP, and the information necessary to identify and transmit the included data can be provided for each piece of data.

[0098] Each step in the step information of the transmission manifest specifies the session_id value of the corresponding SDP as the step identifier field value to specify the corresponding SDP.

[0099] Transmission manifest-type2 allows multiple SDP offers to be distinguished by the session_id value in the o line.

[0100]

[0101] According to one embodiment of the present disclosure, a transmission manifest type 3 may be transmitted together with an integrated SDP in which model attributes, step information, and SDP offers are integrated into one. In this case, data included in each step may additionally have a step_id attribute and have a step identifier of the corresponding step as the value of the step_id. A transmission manifest type 3 is composed of two steps and one integrated SDP, and the step identifier 1 of the first step and the step identifier 2 of the second step may be indicated as the step_id field among the attributes in the data within the SDP offer.

[0102]

[0103] #2

[0104] One embodiment of the present disclosure proposes a system architecture of a transmitting terminal and a receiving terminal that can selectively receive and execute a step of the avatar model to reduce the initial delay time.

[0105] FIG. 1 is a diagram illustrating the structure of a terminal that transmits and receives an avatar model according to one embodiment of the present disclosure.

[0106] A transmitting terminal (110) according to one embodiment of the present disclosure may include an app (111), a capture device (112) for capturing a user's gestures and facial expressions, an animation command generator (113) for moving an avatar from the captured information, an avatar model generator (114) for generating an avatar model, an avatar model storage (115) for storing the generated avatar model, an avatar model transmitter (116) for transmitting the avatar model based on manifest information of the avatar model, and a service provider or network provider app and a network connection device (117).

[0107] A receiving terminal (120) according to one embodiment of the present disclosure may include an app (121), an avatar model receiver (126) for selecting and receiving steps of an avatar model from a transmitting terminal (110), an avatar model storage (125) for storing the received avatar model, a skeletal animation generator (124) for moving the avatar model based on an animation command transmitted from the transmitting terminal (110), an avatar renderer (122) for rendering a 3D model generated from the skeletal animation generator, a display device (123), a service provider or network provider app, and a network connection device (127).

[0108] Between the avatar model sender and receiver, an avatar model is transmitted, the steps to be received for the transmission of the avatar model are determined, and an avatar transmission manifest is transmitted to determine the steps to be received. The avatar transmission manifest is generated from the avatar model manifest, and the structure of the SDP offer within the transmission manifest may vary depending on the type of communication architecture and the protocol message format.

[0109] The transmitting terminal (110) or the receiving terminal (120) may include a processor (not shown) that includes one or more processors or other processing devices that control functions, processes, and / or methods proposed above or to be proposed below. The operations of the transmitting terminal or the receiving terminal proposed above or to be proposed below may be implemented by the processor (not shown). The terms "transmitting" and "receiving" are used herein for illustrative purposes only and may be used interchangeably without departing from the scope of the embodiments.

[0110] #3

[0111] <Operation in Communication Architecture>

[0112] Connections for step-by-step transmission of an avatar model according to one embodiment of the present disclosure can be divided into the following three types of connections, depending on the type and service type of the service provider providing the connection between terminals:

[0113] - IMS connection that ensures compatibility between network operators (hereinafter referred to as network operator compatible connection)

[0114] - IMS connection for OTT or network operator proprietary services (hereinafter referred to as network operator / OTT proprietary connection) using IMS

[0115] - WebRTC connection for OTT service provider-specific services (hereinafter referred to as OTT-specific connection)

[0116] In one embodiment of the present disclosure, depending on the connection method differences described above, the transmission manifest is created differently, and the request procedure for step information between the transmitting and receiving terminals is different. One embodiment of the present disclosure explains the differences in how the avatar transmission manifest is transmitted in each connection method.

[0117] <ims>

[0118] FIG. 2 is a diagram illustrating a transmission path of an avatar transmission manifest and avatar data according to a connection method according to one embodiment of the present disclosure.

[0119] In an IMS-based network operator compatible connection or network operator-specific connection, a transmitting terminal can generate a SIP INVITE message, which is a first message that includes the phone number of the receiving terminal, etc., for connection with the receiving terminal.

[0120] The payload of a SIP INVITE message according to one embodiment of the present disclosure includes an avatar transmission manifest, which includes all SDPs for each stage generated by the avatar model transmitter, along with avatar model attributes and stage attributes. For this purpose, the aforementioned transmission manifest-type2 or transmission manifest-type3 may be used.

[0121] The first message is transmitted to the IMS AS after reviewing the terminal's eligibility for communication services through the P-CSCF, S-CSCF, HSS, etc. Depending on the type of data, the data transmission channels within the SDP in the first message are categorized as media channels, data channels, etc., and again, depending on the needs such as NAT (Network Address Translation), the IP address and port information are replaced and converted into a second message, which is relayed to the next server or terminal.

[0122] If a data channel is included in the SDP, upon receipt of the message, the IMS AS requests the DCSF to open a data channel, and the DCSF instructs the DCMF to open a data channel, thereby opening the data channel.

[0123] The receiving terminal receives a transmission manifest including an SDP composed of an IP address and port converted through the above processes, and performs the receiving terminal operation part described below.

[0124] <Receiving terminal operation>

[0125] In one embodiment of the present disclosure, an initial delay time, a target initial delay time, and an actual delay time are defined.

[0126] The time it takes to completely receive each step is called the Estimated Initial Service Delay (EISD) for each step, and EISD varies depending on the level of network resources, such as bandwidth and delay time, that the network where the receiving terminal is located can provide.

[0127] Users or service providers can specify a 'Targeted Initial Service Delay (TISD)' value, which is the allowable initial delay time, in the app operation settings of the receiving terminal.

[0128] The actual service delay time required to receive data for service initiation is called 'Measured Initial Service Delay (MISD).

[0129] A receiving terminal according to one embodiment of the present disclosure is characterized by calculating an EISD for each step and determining whether to receive a step within the TISD. The receiving terminal calculates an EISD for each step, and if there is a step with an EISD lower than the TISD, selects that step as a receiving target. If no step has an EISD lower than the TISD, the step with the lowest EISD among the steps can be selected.

[0130] In an IMS connection, the receiving terminal selects a stage, recognizes the corresponding SDP as an SDP offer, creates an SDP answer, and transmits it to the sending terminal. In a typical IMS connection, a firewall NAT or signaling server mediates the transmission of the SDP offer / answer between the sending and receiving terminals. Therefore, the receiving terminal sends the SDP answer to a P-CSCF, a firewall NAT, or a signaling server.

[0131] Depending on the step-by-step policy, the receiving terminal can display the other party's avatar on the screen when the step is completed or at an appropriate time.

[0132] <rtc>

[0133] FIG. 3 is a diagram illustrating an avatar transmission manifest and a transmission path of avatar data according to a connection method according to one embodiment of the present disclosure.

[0134] In OTT service provider-specific connections, the discovery of transmitting and receiving terminals is mediated by the service provider.

[0135] The transmitting terminal requests a connection to the service provider (RTC Application Provider, Figure 3) to establish a connection with the receiving terminal. In Figure 3, the data transmission path between the terminal and the service provider is indicated by RTC-8.

[0136] The transmitting terminal receives the receiving terminal's identifier from the service provider. The signaling server (WebRTC signaling function in Figure 3) assists in determining the address and optimal transmission path from the identifier. The transmitting terminal uses the receiving terminal's identifier to send a connection request to the signaling server, and the signaling server locates the receiving terminal and forwards the request.

[0137] According to one embodiment of the present disclosure, a method for transmitting a manifest can consider two cases as follows.

[0138] <Case 1>

[0139] Case 1 is a method in which a transmitting terminal transmits a transmission manifest containing SDP to a signaling server, and the signaling server transmits the manifest to a receiving terminal. Since a typical WebRTC signaling server does not pass a manifest containing multiple SDP offers, such as [transmission manifest-type2], but only processes one SDP offer at a time, it configures an integrated SDP offer, such as [transmission manifest-type3], and describes the step identifier information a=step_id for each step's data. Depending on the implementation of the signaling server, the SDP extended with the user-defined attribute may not be passed, so the service provider processes Case 1 or Case 2 depending on whether the signaling server that mediates the terminals participating in Avatar communication allows the user-defined attribute.

[0140] FIG. 4 is a diagram illustrating a signal flow for media transmission according to one embodiment of the present disclosure.

[0141] Step 1: The transmitting terminal searches for and verifies ICE candidates. The transmitting terminal requests the ICE function to determine the optimal path to the receiving terminal.

[0142] Step 2: The transmitting terminal transmits a transmission manifest containing the integrated SDP offer to the signaling server.

[0143] Step 3: The signaling server determines the location of the receiving terminal.

[0144] Step 4: The signaling server forwards the integrated SDP offer to the receiving terminal.

[0145] Step 5: The receiving terminal determines the appropriate step based on the aforementioned EISD, TISD, etc., identifies the data corresponding to the step with the step_id, and creates an SDP answer containing only the corresponding data.

[0146] Step 6: The receiving terminal transmits the created SDP answer to the signaling server.

[0147] Step 7: The signaling server forwards the SDP answer to the sending terminal.

[0148] Step 8: Media transfer begins.

[0149] Case 2

[0150] Case 2 is a method in which the transmitting terminal transmits a manifest-type 1, which contains only model and stage information, to the service provider, and the service provider then transmits the manifest to the receiving terminal. Since communication between the transmitting terminal and the receiving terminal is brokered within the service provider's service, brokering of the manifest may also be included within the scope.

[0151] When receiving a transmission manifest-type2, the receiving terminal selects a step based on the data size for each step in the manifest and transmits the sess-id or step-id of the desired step to the service provider.

[0152] When receiving a transmission manifest-type3, the receiving terminal selects a step based on the data size for each step in the manifest and transmits the step-id of the desired step to the service provider.

[0153] The service provider transmits the address and sess-id or step-id of the receiving terminal to the transmitting terminal, and the transmitting terminal transmits the SDP corresponding to the specified identifier to the signaling server as an SDP offer, and the signaling server transmits the SDP offer to the receiving terminal.

[0154] The receiving terminal generates an SDP answer and transmits it to the signaling server, and the signaling server transmits it to the transmitting terminal.

[0155] FIG. 5 is a diagram illustrating a signal flow for media transmission according to one embodiment of the present disclosure.

[0156] Step 1: The transmitting terminal transmits a transmission manifest to the service provider. The manifest contains information about the model and phase attributes, but does not include the SDP.

[0157] Step 2: The service provider then transmits the manifest back to the receiving terminal.

[0158] Step 3: The receiving terminal determines the appropriate step based on the aforementioned EISD, TISD, etc., and identifies the data corresponding to the step with the step_id.

[0159] Step 4: The receiving terminal transmits the step identifier of the desired step to the service provider.

[0160] Step 5: The service provider transmits information about the step identifier desired by the receiving terminal to the transmitting terminal.

[0161] Step 6: The transmitting terminal can search for and verify ICE candidates. The transmitting terminal requests the ICE function to determine the optimal path to the receiving terminal.

[0162] Step 7: The transmitting terminal extracts the SDP corresponding to the desired step and transmits it to the receiving terminal as an SDP offer. The SDP offer is first transmitted to the signaling server.

[0163] Step 8: The signaling server determines the location of the receiving terminal.

[0164] Step 9: The signaling server forwards the SDP offer to the receiving terminal.

[0165] Step 10: The receiving terminal generates an SDP answer and transmits it to the signaling server.

[0166] Step 11: The signaling server forwards the SDP answer to the sending terminal.

[0167] Step 12: Media transfer begins.

[0168] The core network object can be controlled to perform operations according to the various embodiments described above.

[0169] A core network object may include a transceiver, memory, and a processor. The transceiver may provide an interface for communicating with other devices within the network. That is, the transceiver converts a bit stream transmitted from the core network object to another device into a physical signal, and converts a physical signal received from another device into a bit stream. In other words, the transceiver can transmit and receive signals. Accordingly, the transceiver may be referred to as a modem, a transmitter, a receiver, a communication unit, or a communication module. In this case, the transceiver enables the core network object to communicate with other devices or systems via a backhaul connection (e.g., wired backhaul or wireless backhaul) or another connection method, or via a network.

[0170] Memory can store data such as basic programs, applications, and configuration information for the operation of core network objects. Memory can consist of volatile memory, nonvolatile memory, or a combination of volatile and nonvolatile memory. Furthermore, memory can provide stored data upon request from the processor.

[0171] The processor can control the overall operations of the core network object. For example, the processor can transmit and receive signals through the transceiver. Additionally, the processor can write and read data to and from memory. To this end, the processor can include at least one processor. The processor can control the core network object to perform operations according to the various embodiments described above.

[0172] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0173] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0174] A method of operating a transmitting terminal according to one embodiment of the present disclosure may include a step of obtaining at least one avatar model manifest including information on at least one step for representing an avatar model.

[0175] A method of operating a transmitting terminal according to one embodiment of the present disclosure may include a step of transmitting at least one avatar model manifest to a receiving terminal.

[0176] A method of operating a transmitting terminal according to one embodiment of the present disclosure may include a step of receiving transmission request information from the receiving terminal.

[0177] An operating method of a transmitting terminal according to one embodiment of the present disclosure may include a step of transmitting, to a permanent receiving terminal, at least one step for expressing the avatar model based on the transmission request information.

[0178] An operation method of a transmitting terminal according to one embodiment of the present disclosure may include a step of transmitting a session description protocol (SDP) offer to the receiving terminal based on the transmission request information.

[0179] An operating method of a transmitting terminal according to one embodiment of the present disclosure may include a step of receiving an SDP answer from the receiving terminal.

[0180] An operating method of a transmitting terminal according to one embodiment of the present disclosure may include a step of transmitting, to the receiving terminal, at least one step for representing the avatar model based on the SDP answer.

[0181] At least one avatar model manifest according to one embodiment of the present disclosure may include an SDP offer.

[0182] Transmission request information according to one embodiment of the present disclosure may include an SDP answer.

[0183] An operating method of a transmitting terminal according to one embodiment of the present disclosure may include a step of transmitting, to the receiving terminal, at least one step for representing the avatar model based on the SDP answer.

[0184] At least one avatar model manifest according to one embodiment of the present disclosure can be transmitted from the transmitting terminal to the receiving terminal via the server.

[0185] The information about the at least one step according to one embodiment of the present disclosure may include at least one of information on the number of the at least one step or information on the size of the at least one step.

[0186] According to one embodiment of the present disclosure, at least one step may include semantic information for representing the avatar model.

[0187] A method of operating a receiving terminal according to one embodiment of the present disclosure may include a step of receiving, from a transmitting terminal, one avatar model manifest including information on at least one step for representing an avatar model.

[0188] A method of operating a receiving terminal according to one embodiment of the present disclosure may include a step of transmitting transmission request information to the transmitting terminal based on the at least one avatar model manifest.

[0189] A method of operating a receiving terminal according to one embodiment of the present disclosure may include a step of receiving, from the transmitting terminal, at least one step for representing the avatar model.

[0190] An operation method of a receiving terminal according to one embodiment of the present disclosure may include a step of receiving a session description protocol (SDP) offer from the transmitting terminal.

[0191] An operation method of a receiving terminal according to one embodiment of the present disclosure may include a step of transmitting an SDP answer to the transmitting terminal based on the SDP offer.

[0192] An operation method of a receiving terminal according to one embodiment of the present disclosure may include a step of receiving, from the transmitting terminal, at least one step for representing the avatar model based on the SDP answer.

[0193] According to one embodiment of the present disclosure, at least one avatar model manifest may include an SDP offer.

[0194] The transmission request information according to one embodiment of the present disclosure may include an SDP answer.

[0195] An operation method of a receiving terminal according to one embodiment of the present disclosure may include a step of receiving, from the transmitting terminal, at least one step for representing the avatar model based on the SDP answer.

[0196] According to one embodiment of the present disclosure, at least one avatar model manifest can be transmitted from the transmitting terminal to the receiving terminal via a server.

[0197] Information about at least one step according to one embodiment of the present disclosure may include at least one of number information of the at least one step or size information of the at least one step.

[0198] At least one step according to one embodiment of the present disclosure may include semantic information for representing the avatar model.

[0199] A transmitting terminal according to one embodiment of the present disclosure may include a transceiver and at least one processor.

[0200] At least one processor according to one embodiment of the present disclosure can obtain at least one avatar model manifest including information about at least one step for representing an avatar model.

[0201] At least one processor according to one embodiment of the present disclosure can transmit the at least one avatar model manifest to a receiving terminal.

[0202] At least one processor according to one embodiment of the present disclosure can receive transmission request information from the receiving terminal.

[0203] At least one processor according to one embodiment of the present disclosure can transmit at least one step for representing the avatar model to a permanent receiving terminal based on the transmission request information.

[0204] At least one processor according to one embodiment of the present disclosure may transmit a session description protocol (SDP) offer to the receiving terminal based on the transmission request information.

[0205] At least one processor according to one embodiment of the present disclosure can receive an SDP answer from the receiving terminal.

[0206] At least one processor according to one embodiment of the present disclosure can transmit, to the receiving terminal, at least one step for representing the avatar model based on the SDP answer.

[0207] At least one processor according to one embodiment of the present disclosure can transmit, to the receiving terminal, at least one step for representing the avatar model based on the SDP answer.

[0208] At least one avatar model matifest according to one embodiment of the present disclosure can be transmitted from the transmitting terminal to the receiving terminal via the server.

[0209] Information about at least one step according to one embodiment of the present disclosure may include at least one of number information of the at least one step or size information of the at least one step.

[0210] At least one step according to one embodiment of the present disclosure may include semantic information for representing the avatar model.

[0211] A receiving terminal according to one embodiment of the present disclosure may include a transceiver and at least one processor.

[0212] At least one processor according to one embodiment of the present disclosure may receive, from a transmitting terminal, an avatar model manifest including information on at least one step for representing an avatar model.

[0213] At least one processor according to one embodiment of the present disclosure can transmit transmission request information to the transmitting terminal based on the at least one avatar model manifest.

[0214] At least one processor according to one embodiment of the present disclosure may receive, from the transmitting terminal, at least one step for representing the avatar model.

[0215] At least one processor according to one embodiment of the present disclosure may receive a session description protocol (SDP) offer from the transmitting terminal.

[0216] At least one processor according to one embodiment of the present disclosure can transmit an SDP answer to the transmitting terminal based on the SDP offer.

[0217] At least one processor according to one embodiment of the present disclosure can receive, from the transmitting terminal, at least one step for representing the avatar model based on the SDP answer.

[0218] At least one processor according to one embodiment of the present disclosure can receive, from the transmitting terminal, at least one step for representing the avatar model based on the SDP answer.

[0219] At least one avatar model manifest according to one embodiment of the present disclosure may include an SDP offer.

[0220] Transmission request information according to one embodiment of the present disclosure may include an SDP answer.

[0221] At least one avatar model manifest according to one embodiment of the present disclosure can be transmitted from the transmitting terminal to the receiving terminal via the server.< / rtc> < / ims> < / value>

Claims

1. In the operation method of the transmitting terminal, A step of obtaining at least one avatar model manifest including information on at least one step for representing an avatar model; A step of transmitting at least one avatar model manifest to a receiving terminal; A step of receiving transmission request information from the receiving terminal; and An operating method comprising the step of transmitting at least one step for representing the avatar model to a permanent receiving terminal based on the above transmission request information.

2. In the first paragraph, the step of transmitting at least one step for expressing the avatar model is, A step of transmitting a session description protocol (SDP) offer to the receiving terminal based on the above transmission request information; A step of receiving an SDP answer from the receiving terminal; and An operating method comprising: a step of transmitting at least one step for representing the avatar model to the receiving terminal based on the SDP answer; 3. In paragraph 1, At least one of the above avatar model manifests includes an SDP offer, The above transmission request information includes an SDP answer, The step of transmitting at least one step for expressing the above avatar model is: A step of transmitting at least one step for expressing the avatar model to the receiving terminal based on the SDP answer; A method of operation, wherein at least one avatar model manifest is transmitted from the transmitting terminal to the receiving terminal via a server.

4. In paragraph 1, The information about the at least one step includes at least one of the number information of the at least one step or the size information of the at least one step, A method of operation, wherein at least one of said steps includes semantic information for representing said avatar model.

5. In the operation method of the receiving terminal, A step of receiving, from a transmitting terminal, an avatar model manifest including information on at least one step for representing an avatar model; A step of transmitting transmission request information to the transmitting terminal based on at least one avatar model manifest; and An operating method comprising the step of receiving at least one step for representing the avatar model from the transmitting terminal.

6. In the fifth paragraph, the step of receiving at least one step for expressing the avatar model comprises: A step of receiving a session description protocol (SDP) offer from the transmitting terminal; A step of transmitting an SDP answer to the transmitting terminal based on the SDP offer; and An operating method comprising: a step of receiving, from the transmitting terminal, at least one step for representing the avatar model based on the SDP answer.

7. In paragraph 5, At least one of the above avatar model manifests includes an SDP offer, The above transmission request information includes an SDP answer, A step of receiving at least one step for representing the above avatar model is: A step of receiving at least one step for expressing the avatar model from the transmitting terminal based on the SDP answer; A method of operation, wherein at least one avatar model manifest is transmitted from the transmitting terminal to the receiving terminal via a server.

8. In paragraph 5, The information about the at least one step includes at least one of the number information of the at least one step or the size information of the at least one step, A method of operation, wherein at least one of said steps includes semantic information for representing said avatar model.

9. At the transmitting terminal, Transmitter and receiver; and comprising at least one processor, said at least one processor comprising: Obtaining at least one avatar model manifest including information about at least one step for representing an avatar model, Transmitting at least one avatar model manifest to a receiving terminal, Receive transmission request information from the above receiving terminal, A transmitting terminal that transmits at least one step for expressing the avatar model to a permanent receiving terminal based on the above transmission request information.

10. In the 9th paragraph, at least one processor, Transmitting a session description protocol (SDP) offer to the receiving terminal based on the above transmission request information, Receive an SDP answer from the above receiving terminal, A transmitting terminal that transmits at least one step for expressing the avatar model to the receiving terminal based on the SDP answer.

11. In paragraph 9, At least one of the above avatar model manifests includes an SDP offer, The above transmission request information includes an SDP answer, At least one processor of the above, Based on the SDP answer, transmitting at least one step for expressing the avatar model to the receiving terminal, A transmitting terminal, wherein at least one avatar model manifest is transmitted from the transmitting terminal to the receiving terminal via a server.

12. In paragraph 9, The information about the at least one step includes at least one of the number information of the at least one step or the size information of the at least one step, A transmitting terminal, wherein at least one of the steps includes semantic information for expressing the avatar model.

13. At the receiving terminal, Transmitter and receiver; and comprising at least one processor, said at least one processor comprising: Receive from a transmitting terminal an avatar model manifest including information on at least one step for representing an avatar model, Based on at least one avatar model manifest, transmitting transmission request information to the transmitting terminal, A receiving terminal that receives at least one step for expressing the avatar model from the transmitting terminal.

14. In the 13th paragraph, at least one processor, Receive a session description protocol (SDP) offer from the above transmitting terminal, Based on the above SDP offer, an SDP answer is transmitted to the transmitting terminal, A receiving terminal that receives at least one step for expressing the avatar model from the transmitting terminal based on the SDP answer.

15. In paragraph 13, At least one of the above avatar model manifests includes an SDP offer, The above transmission request information includes an SDP answer, At least one processor of the above, Based on the above SDP answer, receiving at least one step for expressing the avatar model from the transmitting terminal, A receiving terminal, wherein at least one avatar model manifest is transmitted from the transmitting terminal to the receiving terminal via a server.

Images