Method and apparatus for transmitting and receiving virtual image model

Abstract

The present disclosure relates to a 5G communication system supporting a higher data transmission rate. According to one embodiment of the present disclosure, an operation method in which a transmission terminal transmits a virtual image model may include: dividing a virtual image model into at least one piece of virtual image data and at least one virtual image list; acquiring at least one transmission list based on the at least one virtual image list; sending the at least one transmission list to a receiving terminal; and sending the at least one virtual image data to the receiving terminal based on receiving information of the receiving terminal.

Description

Technical Field

[0001] This disclosure relates to methods and apparatus for virtual avatar communication. In particular, this disclosure relates to methods and apparatus for configuring a virtual avatar list and sending and receiving virtual avatar models between terminals based on the virtual avatar list. Background Technology

[0002] Fifth-generation (5G) mobile communication technology defines wide frequency bands to achieve high-speed transmission rates and new services, not only in sub-6GHz bands such as 3.5GHz, but also in ultra-high frequency bands known as millimeter waves (mmWave) such as 28GHz and 39GHz ("above 6GHz"). Furthermore, in the case of sixth-generation (6G) mobile communication technology, known as ultra-5G systems, implementation in terahertz (THz) bands (e.g., 95GHz to 3THz) is being considered to achieve transmission rates 50 times faster and latency 1 / 10th that of 5G mobile communication technology.

[0003] In the early stages of 5G mobile communication technology development, to support enhanced mobile broadband (eMBB), ultra-reliable low-latency communication (URLLC), and massive machine-type communication (mMTC) services and meet performance requirements, relevant standardization work has been carried out. This work involves: beamforming and massive MIMO to mitigate path loss and increase the propagation distance of UHF radio waves; support for multiple parameter sets (such as multi-subcarrier spacing) and dynamic time slot format operation to efficiently utilize UHF resources; initial access technologies to support multi-beam transmission and broadband; definition and operation of the bandwidth portion (BWP); new channel coding methods (such as low-density parity-check (LDPC) codes for high-capacity data transmission and polar codes for reliable transmission of control information); L2 preprocessing; and network slicing technologies to provide dedicated networks for specific services.

[0004] Currently, considering the services that 5G mobile communication technology is intended to support, discussions are underway regarding improvements and enhancements to the initial 5G mobile communication technology, and physical layer standardization work is also progressing. The technologies involved include: vehicle-to-everything (V2X) networks that assist autonomous vehicles in making driving decisions and improve user convenience based on vehicle-transmitted location and status information; New Radio Unlicensed Bands (NR-U) designed to meet various regulatory requirements for unlicensed frequency bands; NR terminal low power technology (UE power saving); terminal-satellite direct communication technology (non-terrestrial network (NTN)) that provides coverage for areas that cannot communicate with terrestrial networks; and positioning technology.

[0005] In addition, standardization efforts are underway in the following areas of air interface architecture / protocols: Industrial Internet of Things (IIoT) to support new services through integration with other industries; Integrated Access Backhaul (IAB) to provide nodes for extended network service areas by integrating wireless backhaul and access links; Mobility enhancements including conditional handover and dual active protocol stack (DAPS) handover; and NR two-step RACH to simplify random access procedures. Meanwhile, standardization efforts are also ongoing in the areas of system architecture / services, including 5G baseline architectures (such as service-based architectures and service-based interfaces) that combine Network Functions Virtualization (NFV) and Software-Defined Networking (SDN) technologies, and Mobile Edge Computing (MEC) for providing services based on terminal location.

[0006] Once such 5G mobile communication systems are commercialized, the explosive growth of interconnected devices will access the communication network. Therefore, enhancing the functionality and performance of 5G mobile communication systems and ensuring integrated operation of interconnected devices will become essential. To this end, new research will be conducted in the following areas: Extended Reality (XR) for efficient support of Augmented Reality (AR), Virtual Reality (VR), Mixed Reality (MR), etc.; leveraging Artificial Intelligence (AI) and Machine Learning (ML) to improve 5G performance and reduce complexity; AI service support; Metaverse service support; and drone communication.

[0007] Furthermore, the development of 5G mobile communication systems can lay the foundation for the development of technologies such as novel waveforms, full-dimensional MIMO (FD-MIMO), array antennas, and massive MIMO to ensure terahertz band coverage for 6G mobile communication technologies; high-dimensional spatial multiplexing technologies that utilize metamaterial-based lenses and antennas and orbital angular momentum (OAM) to improve terahertz band signal coverage; and reconfigurable smart surfaces (RIS) technology. It can also lay the foundation for technologies such as full-duplex technology to improve the frequency efficiency of 6G mobile communication technologies and enhance system networks; AI-based communication technologies that utilize satellites and AI from the design stage to achieve system optimization and incorporate end-to-end AI support; and next-generation distributed computing technologies that leverage ultra-high-performance communication and computing resources to enable services with complexity exceeding the limits of terminal operation capabilities.

[0008] As mentioned above, due to the development of wireless communication systems, various services are available, thus requiring a solution for the smooth delivery of such services. Summary of the Invention

[0009] Technical solution

[0010] According to embodiments of this disclosure, the method for sending a virtual avatar model by a sending terminal may include: dividing the virtual avatar model into at least one virtual avatar data and at least one virtual avatar list.

[0011] According to embodiments of this disclosure, the method for a sending terminal to send a virtual avatar model may include: obtaining at least one transmission list based on the at least one virtual avatar list.

[0012] According to embodiments of this disclosure, the method for a sending terminal to send a virtual avatar model may include: sending the at least one transmission list to a receiving terminal.

[0013] According to embodiments of this disclosure, the method for sending a virtual avatar model by a sending terminal may include: sending at least one virtual avatar data to a receiving terminal based on the receiving information received by the receiving terminal.

[0014] According to embodiments of this disclosure, the sending terminal for sending virtual avatar models may include a transceiver and at least one processor.

[0015] According to embodiments of this disclosure, the at least one processor may be configured to: divide the virtual avatar model into at least one virtual avatar data and at least one virtual avatar list.

[0016] According to embodiments of this disclosure, the at least one processor may obtain at least one transport list based on the at least one virtual avatar list.

[0017] According to embodiments of this disclosure, the at least one processor may be configured to send the at least one transmission list to a receiving terminal.

[0018] According to embodiments of this disclosure, the at least one processor may be configured to: send the at least one virtual avatar data to the receiving terminal based on the received information from the receiving terminal. Attached Figure Description

[0019] Figure 1 A terminal structure for sending and receiving virtual avatar models according to an embodiment of the present disclosure is shown.

[0020] Figure 2 The present disclosure illustrates a virtual avatar transmission list and a virtual avatar data transmission path based on the connection method, according to an embodiment of the present disclosure.

[0021] Figure 3 The present disclosure illustrates a virtual avatar transmission list and a virtual avatar data transmission path based on the connection method, according to an embodiment of the present disclosure.

[0022] Figure 4 The media transmission signal flow according to an embodiment of the present disclosure is illustrated.

[0023] Figure 5 The media transmission signal flow according to an embodiment of the present disclosure is illustrated. Detailed Implementation

[0024] First, a brief description of the terminology used in this specification is given, followed by a detailed description of this disclosure.

[0025] The terminology used in this disclosure is generally used in the art in light of the functionality of this disclosure; however, these terms may vary depending on the intent, precedent, or new technology of those skilled in the art. Furthermore, in some cases, terms may be arbitrarily chosen by the applicant, the meaning of which will be described in detail in the relevant sections of this disclosure. Therefore, the terminology used herein should not be understood based solely on its simple name, but rather on its meaning and the overall description of this disclosure.

[0026] Throughout this disclosure, when an element is referred to as being “connected” to another element, it can be “directly connected” to the other element or can be “electrically connected” to the other element via one or more intermediate elements.

[0027] Throughout this disclosure, when something is referred to as “comprising” an element, it may further include one or more other elements unless otherwise stated. Furthermore, as used in the specification, terms such as “unit” and “module” may refer to a unit that performs at least one function or operation, and that unit may be implemented in hardware, software, or a combination of hardware and software.

[0028] Furthermore, the statement "at least one of A, B, and C" can refer to any one of "A", "B", "C", "A and B", "A and C", "B and C", and "A, B, and C".

[0029] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings, so that those skilled in the art can readily implement the embodiments of this disclosure. However, this disclosure can be implemented in many different forms and should not be construed as limited to the embodiments set forth herein. Furthermore, for the sake of clear description of this disclosure, portions in the drawings that are not related to the description of this disclosure will be omitted, and throughout the specification, the same reference numerals will denote the same elements.

[0030] Extended reality (XR) can be a technology that displays 3D graphical elements in a three-dimensional (3D) space that are integrated with the user's reality. Examples of XR content may include applications that provide information based on the user's location or interaction in an airport or museum, and that display a description of an object in a spatial speech bubble, play additional 3D videos, or perform content control loops through user interactions such as end buttons when the user searches for an object (such as a work of art).

[0031] In the example above, animations, control loops executed through user interaction, etc., can be pre-planned and embedded in XR content. When the user retrieves and selects an object, the animation can be displayed according to the planned time flow and then end. New interactive events not described can be regarded as errors or ignored.

[0032] The technical background of this disclosure can be virtual avatar communication that combines the aforementioned 3D content, mobile communication networks, and user interaction. As a basic scenario, when a first terminal captures user 1's actions (gestures or expressions), replaces these actions with a series of commands, provides a pre-prepared virtual avatar model or one generated through continuous user photography and artificial intelligence learning, and sends it to a second terminal of user 2, the target of the virtual avatar communication, the second terminal can manipulate the virtual avatar model based on the received virtual avatar model and commands, render the manipulated model, and display it on a monitor, allowing user 2 to see user 1's virtual avatar as an XR object combined with the real world before them. The second terminal can also perform the same operation, causing the first terminal to display user 2's virtual avatar, so that user 1 can also see user 2's virtual avatar presented in XR form.

[0033] A virtual avatar model may include meshes, textures, and skeletons. A mesh may include vertices, faces connecting three or more vertices, and vertex-to-texture mapping information. A texture may include the color of the vertices and the surface information of the faces. Skeletons may represent human anatomy and are presented in a format such as a scene graph by lines connecting the joints between the bones.

[0034] Virtual avatar models can be generated based on various images, videos, or other data obtained from user terminals or other sensor devices, and the device that performs the generation can be a separate dedicated processing device provided by the user terminal or a service provider that provides services subscribed to by the user.

[0035] The size of a virtual avatar model can vary from user to user, from the display resolution and display capabilities of the terminal displaying the virtual avatar, and from the technological maturity of the service provider.

[0036] For virtual avatar model information used in virtual avatar communication (especially real-time virtual avatar communication), the virtual avatar model should first be sent from the terminal of the user who owns the virtual avatar model to the other party's terminal. Even from the perspective of 5G uplink traffic, 20MB of data may be considered extremely large.

[0037] An example of a similar service in related technologies could be video calling. Since video calling involves immediately displaying each other's video feeds after a connection is established, the initial latency of avatar communication might be considered a significant reduction in QoS compared to traditional video calls. Therefore, a method to reduce this initial latency may be necessary.

[0038] The Proximity Services (ProSe) layer discussed in this article can refer to the entire layer that performs control operations for terminal-to-terminal (device-to-device (D2D)) or ProSe communication services. This layer may be located above the AS layer and may exchange signaling with the AS layer for terminal-to-terminal communication. Furthermore, this layer may provide data transmission capabilities on the links established for direct communication between terminals. That is, it may include IP protocols, non-IP protocols, and transport protocols (such as TCP or UDP) for message transmission.

[0039] In addition, the terminal (UE) can connect to the 5G Radio Access Network (5G-RAN) to access 5G core network equipment. The 5G core network may include the following network functions: Access and Mobility Management Function (AMF) refers to network functions that manage the mobility of the terminal. Session Management Function (SMF) refers to network functions that manage the packet data network connection provided to the terminal. This packet data network connection may be referred to as a Protocol Data Unit (PDU) session. Policy and Charging Function (PCF) refers to network functions that apply the service policies, charging policies, and PDU session policies of the mobile communication service provider for the terminal. Unified Data Management (UDM) function refers to network functions that store and manage information about subscribed users. Network Open Function (NEF) can access information used to manage the terminal in the 5G network, thereby enabling the execution of network functions (NF) connected to the 5G core network to send information about the terminal to the NF or report information about the terminal to external entities, such as subscribing to the terminal's mobility management events, subscribing to the terminal's session management events, requesting session-related information, configuring the terminal's charging information, and requesting changes to the terminal's PDU session policies. 5G-RAN can refer to a base station that provides wireless communication capabilities for terminals. User plane functions (UPFs) perform gateway functions to transmit packets sent and received by terminals. UPFs can connect to data networks to send data packets generated in the 5G system to external data networks; for example, they can connect to data networks with internet connectivity to route data packets sent by terminals to the internet.

[0040] Proximity-Based Service (ProSe) functions refer to logical functions that perform the network operations required to provide proximity services. One or more ProSe functions may exist in a Public Land Mobile Network (PLMN), which may be deployed in a distributed manner to serve specific areas or specific terminals. ProSe functions can perform the function of directly providing terminals with the information required to use ProSe services. This may mean providing terminals with the information required for direct discovery functions for terminal-to-terminal direct discovery or direct communication functions for terminal-to-terminal direct communication. Furthermore, ProSe functions can perform functions that manage the name or code of direct discovery functions for terminal-to-terminal direct discovery. In addition, ProSe functions can perform functions that trigger terminals to perform direct discovery functions and can also provide billing or security functions used by ProSe services. Terminals can connect to ProSe functions through the PC3 interface defined in 3GPP.

[0041] Embodiments of this disclosure may provide a method for configuring a virtual avatar model that includes multiple independently executable steps to reduce initial latency.

[0042] Embodiments of this disclosure may provide a system architecture for a transmitting terminal and a receiving terminal that can selectively receive and execute steps of a virtual avatar model to reduce initial latency.

[0043] #1

[0044] Embodiments of this disclosure may provide a method for configuring a virtual avatar model that includes multiple independently executable steps to reduce initial latency.

[0045] According to embodiments of this disclosure, a virtual avatar model may include various types of virtual avatar model data for representing the virtual avatar model, and a virtual avatar model list, which is metadata describing the relationships between the various types of data.

[0046] Virtual avatar model:

[0047] According to embodiments of this disclosure, the types of virtual avatar model data may include meshes, textures, skeletons, and scene descriptions.

[0048] Virtual avatar model data:

[0049] A mesh can be information used to represent a 3D model in 3D space by using vertices and faces.

[0050] Textures can be information used to represent the color information (such as color or image) of the vertices and faces of a mesh.

[0051] Skeleton can be information used to represent the structure of the human body, such as bones, perspective, and facial features.

[0052] Scene descriptions can be information used to represent the relationships between meshes, textures, and bones.

[0053] According to embodiments of this disclosure, there may be a substitution relationship between one piece of data and another among the multiple pieces of data contained in a virtual avatar model. For example, medium-resolution texture data may be substituted for low-resolution texture data.

[0054] According to embodiments of this disclosure, while the receiving terminal receives and displays the lowest resolution texture data to immediately present the virtual avatar, when it receives medium resolution texture data using idle bandwidth, it can use medium resolution texture data to replace the previous low resolution texture to display the virtual avatar. According to embodiments of this disclosure, there may be a complementary relationship between one piece of data and another among the multiple pieces of data included in the virtual avatar model. For example, medium-density mesh data can complement low-density mesh data.

[0055] Supplementary relationship:

[0056] According to embodiments of this disclosure, while receiving and displaying the lowest density mesh data (e.g., 100 vertex information per unit space) to immediately render a virtual avatar, the receiving terminal can utilize the low-density and medium-density mesh data (e.g., 10,000 vertex information per unit space) to fill the space previously occupied by the low-density mesh data (e.g., 10,100 vertex information per unit space) when receiving medium-density mesh data (e.g., 10,000 vertex information per unit space) using idle bandwidth. According to embodiments of this disclosure, the virtual avatar model list is characterized by including one or more steps that the receiving terminal can selectively receive, and including information about each step. The receiving terminal can receive information about the steps, determine the steps to be received based on the transmission time required for each step and the attribute information of each step, and receive the data required for rendering the virtual avatar based on this determination. The data transmission information in the virtual avatar model list can be appropriately converted according to the mobile communication network protocol connecting the sending terminal and the receiving terminal, then generated as a virtual avatar transmission list and exchanged. For this purpose, prompts for each transmission protocol can be provided in the list.

[0057] According to embodiments of this disclosure, the list of virtual avatar models can be configured as follows.

[0058] List of virtual avatar models:

[0059] Virtual avatar model attributes may include sub-attribute information used to distinguish virtual avatar models, such as model identifier, owner identifier, number of steps, and strategy.

[0060]

[0061] Model identifiers can be unique within users and service providers or among providers.

[0062] An owner identifier can represent the identity of the object represented by the avatar model or the identity of the owner of the avatar model file. The owner identifier can be unique within the service provider or among providers.

[0063] The number of steps can represent the number of steps provided in the list.

[0064] A model strategy can represent the policy that a receiving terminal should follow when using a virtual avatar model. According to embodiments of this disclosure, the strategy may include delete_after_close and preserve_after_close. delete_after_close indicates that the virtual avatar model should be deleted after virtual avatar communication ends. preserve_after_close indicates that the virtual avatar model can be retained even after virtual avatar communication ends and used in the next communication.

[0065] Step attributes represent information to be considered when transmitting data corresponding to each step, and may include step identifier, size, semantics, quality, and step policy. Each step may assume that the virtual avatar cannot be correctly rendered if only a portion of the step is received. In other words, artifacts appearing when rendering the virtual avatar with only a portion of the step received can be considered errors in the receiving terminal, and the sending terminal can determine, based on a policy, whether to allow or prohibit the rendering of the virtual avatar when only a portion of the step is received.

[0066]

[0067] In a virtual avatar model, a step identifier can be a unique step identifier.

[0068] The step size can be the sum of the sizes of the data contained in the step. The receiving terminal can determine reception based on the sum of the data, regardless of the quantity or type of data to be received. In particular, the size can be provided as a lossless size and a lossy size. The lossless size can be data that does not allow errors, and can refer to the sum of the size of data that should be corrected by retransmission if an error occurs during reception. The lossy size can be data that allows errors, and can refer to the sum of the size of data that does not need to be retransmitted even if an error occurs during reception.

[0069] Semantics can be a list of semantic features included in the steps. Furthermore, metadata type can be specified to define the scope of representation for the semantic features. Depending on the scope of transmission, embodiments of this disclosure may propose face, head, upper body with arms, and full body as basic semantic features.

[0070] When a step attribute includes semantics and contains semantic features, the receiving terminal can make a step reception determination based on the range represented by the step. For example, when a step only includes the face from the perspective of a receiving terminal attempting to present the whole body, it can be determined to select a step that contains more data and whole-body semantic features than that step.

[0071] Quality can represent quality information included in a step. Additionally, a quality_type and quality_value can be specified to define the range of quality representations. The quality_type can include PSNR, resolution, and num_vertex.

[0072] PSNR can represent the lossy compression quality of texture data in an image or video format relative to the original copy.

[0073] Resolution can refer to the resolution of texture data in image or video formats.

[0074] num_vertex can represent the number of vertices included in the grid data.

[0075] A step strategy can represent the policy that a receiving terminal should follow when receiving steps of a virtual avatar model. According to embodiments of this disclosure, the strategy may include allowing partial playback (partial_playback_allowed) and disallowing partial playback (partial_playback_disallowed). Partial_playback_allowed may mean that the receiving terminal may attempt to play the virtual avatar model after receiving a portion of the step data. Partial_playback_disallowed may mean that the receiving terminal may only attempt to play the virtual avatar model after all step data has been received.

[0076] Transmission prompts can provide information for generating SDP proposals (provided by the sending terminal to the receiving terminal), which are then provided by the sending terminal to the receiving terminal for sending data for each step.

[0077]

[0078] Transmission prompts may include information regarding the transmission of data belonging to each step, as contained in the avatar model inventory. Each step may include a step identifier and data transmission information for one or more data items belonging to that step. The data transmission information provides attribute information for generating a message corresponding to the transmission protocol for each data item. Using this information, the avatar model sender of the sending terminal can generate a message suitable for the transmission protocol and send it to the network client.

[0079] The step identifier can specify the step to which the data information belongs.

[0080] Data information may include m-rows, a-rows, access location, and access size.

[0081] The m-line can refer to the syntax of the Session Description Protocol (SDP) as defined in the IETF, indicating the media type, protocol, and format of the data to be transmitted. According to the SDP, since attributes begin with "m=" and are specified in a defined order, and are separated by spaces, in embodiments of this disclosure, the attribute can be specified as "m=media type protocol format".

[0082] The a-line can specify additional attributes of the data indicated by the m-line. In embodiments of this disclosure, "mid" can be used to represent the media identifier attribute of the data, and different data in the virtual avatar model can have a unique identifier. According to SDP, since the attribute is specified in the format "a=mid:<value>", in embodiments of this disclosure, the attribute can be specified as "a=mid:media identifier".

[0083] Media types defined in SDP can include “audio,” “video,” “text,” “application,” and “message.” For example, in the case of texture, the media type can be video, and in the case of mesh, the media type can be application.

[0084] The protocol can be a recommended value for the data transmission protocol, which can be replaced with other values ​​if the networks of the sending and receiving terminals do not support it. When supported, the protocol and format values ​​can be used. For example, in the case of a mesh, the protocol can be specified as UDP / DTLS / SCTP, and the format can be specified as webrtc-datachannel. For example, in the case of a texture, the protocol can be specified as RTP / AVPF, and the format can be specified as 99.

[0085] The access location can represent the path through which the virtual avatar model sender accesses data and transmits it to the network. This path can be a location in the virtual avatar model file, a location in the sending terminal's memory, or a URL address of a memory or internet server on an external network of the sending terminal.

[0086] The access size can represent the start and end positions of data in the access location.

[0087] When sending data outside the terminal, the virtual avatar transmitter can negotiate with the external server to transmit the access location portion of the external server file to a specific port of the external server address through communication between the virtual avatar transmitter and the external server. The virtual avatar transmitter can send the external server address and port to the receiving terminal by specifying the external server address and port in the SDP when specifying the SDP.

[0088] After establishing a transmission channel between the sending terminal and the receiving terminal, the virtual avatar model sender can transmit the corresponding file through each transmission channel according to the request of the receiving terminal.

[0089] According to embodiments of this disclosure, a virtual avatar model sender can configure an SDP proposal for each step by using a transmission prompt. The SDP proposal can include data contained in a step identifier specified in the transmission prompt. The m-line data can be listed using m-line information, and in the m-line format, the media, protocol, and format can be specified by using media type, protocol, and format information, respectively. The a-line data can be specified using mid information.

[0090] Because other content in the generated SDP proposal message (e.g., port information in line m or connection information in line c) uses the available port and IP address information of the sending terminal, it may not be provided in the transmission prompt. However, the virtual avatar model sender according to embodiments of this disclosure is characterized by the ability to specify different unique ports for different data.

[0091] By using different ports for each data item, a receiving terminal that receives multiple SDP proposals or merged SDP proposals as described below can simultaneously initiate the reception of data belonging to one or more steps, regardless of whether other data has been fully received.

[0092] The sending terminal can send a transmission protocol message, including a list of virtual avatars transmitted, to the receiving terminal.

[0093] The virtual avatar delivery manifest may include step information and SDP proposals corresponding to each step.

[0094] Depending on the communication architecture, multiple SDP proposals may be allowed, or only a single SDP proposal may be allowed.

[0095] In transmission list type 1 according to embodiments of the present disclosure, model attributes and step information can be transmitted. A receiving terminal receiving the transmission list can select a desired step from the step information and request transmission of the corresponding step by using a step identifier. A sending terminal can send an SDP proposal for the requested step, and a receiving terminal can initiate transmission from the sending terminal after sending an SDP response.

[0096]

[0097] According to embodiments of this disclosure, in transport manifest type 2, model attributes, step information, and multiple SDP proposals can be transmitted together. In this case, each SDP proposal can be identified by the session identifier (session_id) field value in line o of the SDP, and for each piece of data, information necessary to identify and transmit the included data can be provided.

[0098] Each step in the transport manifest's step information can specify the corresponding SDP's session_id value as the step identifier field value to specify the corresponding SDP.

[0099] In transport manifest type 2, multiple SDP proposals can be identified by the session identifier (session_id) value in line o.

[0100]

[0101] According to embodiments of this disclosure, in transport manifest type 3, model attributes, step information, and a merged SDP that combines SDP proposals into one can be transmitted together. In this case, the data included in each step may additionally include a step identifier (step_id) attribute, and the step identifier of the corresponding step may be included as the value of step_id. Transport manifest type 3 may include two steps and a merged SDP, where the step identifier "1" of the first step and the step identifier "2" of the second step can be indicated by the step_id field in the attributes of the data in the SDP proposal.

[0102]

[0103] #2

[0104] Embodiments of this disclosure may provide a system architecture for a transmitting terminal and a receiving terminal that can selectively receive and execute steps of a virtual avatar model to reduce initial latency.

[0105] Figure 1 A terminal structure for sending and receiving virtual avatar models according to an embodiment of the present disclosure is shown.

[0106] According to embodiments of the present disclosure, the transmitting terminal 110 may include an application (app) 111, a data acquisition device 112 for acquiring user gestures and expressions, an animation command generator 113 for animates a virtual avatar based on the acquired information, a virtual avatar model generator 114 for generating a virtual avatar model, a virtual avatar model storage device 115 for storing the generated virtual avatar model, a virtual avatar model transmitter 116 for transmitting the virtual avatar model based on a list of virtual avatar models, a service provider or network operator application, and a network connection device 117.

[0107] According to embodiments of the present disclosure, the receiving terminal 120 may include an application (app) 121, a virtual avatar model receiver 126 for selecting and receiving virtual avatar models from the sending terminal 110, a virtual avatar model memory 125 for storing the received virtual avatar models, a skeletal animation generator 124 for moving the virtual avatar models based on animation commands received from the sending terminal 110, a virtual avatar renderer 122 for rendering 3D models generated from the skeletal animation generator 124, a display device 123, a service provider or network operator application, and a network connection device 127.

[0108] Virtual avatar models can be transmitted between virtual avatar model transmitters and receivers. The steps to be received in the virtual avatar model transmission can be determined, and a virtual avatar transmission list can be transmitted to determine these steps. The virtual avatar transmission list can be generated from the virtual avatar model list, and the structure of the SDP proposals in the transmission list can vary depending on the type of communication architecture and the protocol message format.

[0109] The transmitting terminal 110 or the receiving terminal 120 may include a processor (not shown), which includes one or more processors or other processing devices for controlling the functions, processes, and / or methods described above or below. The operation of the transmitting or receiving terminal described above or below may be implemented by a processor (not shown). The terms "transmit" and "receive" as used herein are for illustrative purposes only and may be used interchangeably without departing from the scope of the embodiments.

[0110] #3

[0111] Operations in the Communication Architecture

[0112] According to embodiments of this disclosure, the connection for the step-by-step transmission of a virtual avatar model can be categorized into three connection types based on the service type and the type of service provider offering the connection between terminals.

[0113] - IMS connections that ensure compatibility between network operators (hereinafter referred to as network operator compatible connections).

[0114] - IMS connections that use IMS but are intended for OTT or network operator-specific services (hereinafter referred to as network operator / OTT-specific connections).

[0115] - WebRTC connections used for OTT-specific services (hereinafter referred to as OTT-specific connections)

[0116] In embodiments of this disclosure, the transport list may differ according to the connection method described above, and the step information request process between the sending terminal and the receiving terminal may also differ. In embodiments of this disclosure, the differences between the methods for sending the virtual avatar transport list in the various connection methods will be described.

[0117] <ims>

[0118] Figure 2 The present disclosure illustrates a virtual avatar transmission list and a virtual avatar data transmission path based on the connection method, according to an embodiment of the present disclosure.

[0119] In IMS-based network operator-compatible connections or network operator-dedicated connections, the sending terminal can generate a SIPINVITE message, which is the first message used to connect to the receiving terminal, including the receiving terminal's telephone number.

[0120] According to embodiments of this disclosure, the payload of a SIP INVITE message may include a avatar transport list, and the avatar transport list may include avatar model attributes, step attributes, and all SDPs for each step generated by the avatar model sender. For this purpose, transport list type 2 or transport list type 3 described above may be used.

[0121] The first message, after passing through P-CSCF, S-CSCF, HSS, etc. to check whether the terminal is eligible for communication services, is sent to the IMS AS. The data transmission channel in the SDP of the first message can be divided into media channel and data channel according to the data type. Then, according to requirements such as Network Address Translation (NAT), it is converted into a second message with IP address and port information, and then relayed to the next server or terminal.

[0122] When the SDP includes a data channel, the IMS AS can request the DCSF to open the data channel after receiving the corresponding message. The DCSF can then instruct the DCMF to open the data channel, thereby establishing the data channel.

[0123] The receiving terminal can receive a transmission list including SDP (including IP addresses and port numbers converted through the above process) and perform the following receiving terminal operations.

[0124] <Operation of the receiving terminal>

[0125] In embodiments of this disclosure, an initial delay time, a target initial delay time, and an actual delay time can be defined.

[0126] The time required to fully receive each step can be called the estimated initial service delay (EISD) for each step. The EISD may vary depending on the level of network resources available in the network where the receiving terminal is located, such as bandwidth and latency.

[0127] Users or service providers can specify a Target Initial Service Delay (TISD) value as the allowed initial delay in the application operation configuration of the receiving terminal.

[0128] The actual service latency required to receive data for service initiation can be called the Measured Initial Service Latency (MISD).

[0129] According to embodiments of this disclosure, the receiving terminal can calculate the EISD for each step and determine whether to receive the step within the TISD. The receiving terminal can calculate the EISD for each step, and when the EISD is lower than the TISD, the step can be selected as the receiving target. When no step has an EISD lower than the TISD, the step with the lowest EISD can be selected.

[0130] In an IMS connection, the receiving terminal can select a step, then identify the SDP corresponding to that step as an SDP proposal, generate an SDP response, and send it to the sending terminal. In a typical IMS connection, because a firewall NAT or signaling server mediates and transmits the SDP proposal / response between the sending and receiving terminals, the receiving terminal can send the SDP response to the P-CSCF, firewall NAT, or signaling server.

[0131] Depending on the strategy for each step, the receiving terminal may display the other party's virtual avatar on the screen when the step is completed or at an appropriate time.

[0132] <rtc>

[0133] Figure 3 The present disclosure illustrates a virtual avatar transmission list and a virtual avatar data transmission path based on the connection method, according to an embodiment of the present disclosure.

[0134] In OTT service provider-dedicated connections, service providers can mediate the discovery of sending and receiving terminals.

[0135] The sending terminal can send to the service provider ( Figure 3 The RTC application provider requests a connection to the receiving terminal. Figure 3 In this context, the data transmission path between the terminal and the service provider is represented by RTC-8.

[0136] The sending terminal can receive the identifier of the receiving terminal from the service provider. Signaling server ( Figure 3 The WebRTC signaling function helps determine the address and optimal transmission path from the identifier. The sending terminal can send a connection request to the signaling server using the receiver terminal's identifier, and the signaling server can locate the receiver terminal and send the request to it.

[0137] According to embodiments of this disclosure, the following two methods for inventory transmission can be considered.

[0138] <Scenario 1>

[0139] In Case 1, the sending terminal can send a transport list including SDPs to the signaling server, which can then send the transport list to the receiving terminal. Since typical WebRTC signaling servers do not transmit lists containing multiple SDP proposals as in [Transport List Type 2], but rather process only one SDP proposal at a time, merged SDP proposals can be configured as in [Transport List Type 3], and a step identifier (a=step_id) can be specified for the data corresponding to each step. Because extended SDPs with user-defined attributes may not be transmitted depending on the signaling server implementation, the service provider can handle Case 1 or Case 2 depending on whether the signaling server mediating the avatar communication allows user-defined attributes.

[0140] Figure 4 The media transmission signal flow according to an embodiment of the present disclosure is illustrated.

[0141] Step 1: The sending terminal can discover and verify ICE candidates. The sending terminal can request the optimal path from the ICE function and receive the best path from the receiving terminal.

[0142] Step 2: The sending terminal may send a transport list, including the merged SDP proposal, to the signaling server.

[0143] Step 3: The signaling server can determine the location of the receiving terminal.

[0144] Step 4: The signaling server can send the merged SDP proposal to the receiving terminal.

[0145] Step 5: The receiving terminal can determine the appropriate step based on the above EISD and TISD, identify the data corresponding to the step by using step_id, and generate an SDP response that includes only the data.

[0146] Step 6: The receiving terminal can send the generated SDP response to the signaling server.

[0147] Step 7: The signaling server can send the SDP response to the sending terminal.

[0148] Step 8: Media transfer can be started.

[0149] <Scenario 2>

[0150] In scenario 2, the sending terminal can send a manifest containing only model and step information (manifest type 1) to the service provider, who can then forward the manifest to the receiving terminal. Since communication between the sending and receiving terminals is relayed within the service provider's service, manifest relaying is also included.

[0151] When receiving a list type 2, the receiving terminal can select a step based on the data size of each step in the list and send the desired step's sess-id or step-id to the service provider.

[0152] When receiving a list type 3, the receiving terminal can select a step based on the data size of each step in the list and send the step-id of the desired step to the service provider.

[0153] The service provider can send the address of the receiving terminal and its sess-id or step-id to the sending terminal. The sending terminal can send the SDP corresponding to the specified identifier as an SDP proposal to the signaling server, and the signaling server can send the SDP proposal to the receiving terminal.

[0154] The receiving terminal can generate an SDP response and send it to the signaling server, and the signaling server can send the SDP response to the sending terminal.

[0155] Figure 5 The media transmission signal flow according to an embodiment of the present disclosure is illustrated.

[0156] Step 1: The sending terminal can send the transport manifest to the service provider. The transport manifest may include information about the model and step attributes, but may not include SDP.

[0157] Step 2: The service provider can send the transmission list to the receiving terminal.

[0158] Step 3: The receiving terminal can determine the appropriate step based on the above EISD and TISD, and identify the data corresponding to the step by using step_id.

[0159] Step 4: The receiving terminal can send the step identifier of the desired step to the service provider.

[0160] Step 5: The service provider can send information about the steps the receiving terminal expects to the sending terminal.

[0161] Step 6: The sending terminal can discover and verify ICE candidates. The sending terminal can request and receive the optimal path from the receiving terminal through the ICE function.

[0162] Step 7: The sending terminal can extract the SDP corresponding to the desired step and send it as an SDP proposal to the receiving terminal. The SDP proposal can be sent to the signaling server first.

[0163] Step 8: The signaling server can determine the location of the receiving terminal.

[0164] Step 9: The signaling server can send the SDP proposal to the receiving terminal.

[0165] Step 10: The receiving terminal can generate an SDP response and send the SDP response to the signaling server.

[0166] Step 11: The signaling server can send the SDP response to the sending terminal.

[0167] Step 12: Media transfer can be started.

[0168] The core network entity can be controlled to perform operations according to the various embodiments described above.

[0169] Core network entities may include transceivers, memory, and processors. A transceiver provides an interface for communicating with other devices in the network. Specifically, a transceiver converts bit streams sent by a core network entity to other devices into physical signals and converts physical signals received from other devices back into bit streams. In other words, a transceiver can both send and receive signals. Therefore, a transceiver may be referred to as a modem, transmitter, receiver, communication unit, or communication module. In this context, the transceiver allows core network entities to communicate with another device or system via a backhaul connection (e.g., wired or wireless backhaul), through another connection method, or over the network.

[0170] The memory can store data such as basic programs, application programs, or configuration information used for the operation of core network entities. The memory may include volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory. Furthermore, the memory can provide stored data upon request from the processor.

[0171] The processor can control the overall operation of the core network entity. For example, the processor can send and receive signals via a transceiver. Furthermore, the processor can write to / read data from memory. For this purpose, the processor may include at least one processor. The processor can control the core network entity to perform operations according to the various embodiments described above.

[0172] Machine-readable storage media may be provided in the form of non-transitory storage media. Here, the term "non-transitory storage media" may mean that the storage media is a tangible device and does not include signals (such as electromagnetic waves), and may mean that data can be stored in the storage media semi-permanently or temporarily. For example, "non-transitory storage media" may include buffers for temporarily storing data.

[0173] According to embodiments of this disclosure, the methods described herein according to various embodiments of this disclosure may be included and provided in a computer program product. The computer program product can be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)), or distributed online through an app store (e.g., downloaded or uploaded), or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable application) may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, the memory of an app store server, or the memory of a relay server.

[0174] According to embodiments of the present disclosure, the operation method of the sending terminal may include: obtaining at least one list of virtual avatar models, the list of at least one virtual avatar model including information about at least one step for representing the virtual avatar model.

[0175] According to embodiments of this disclosure, the operation method of the sending terminal may include: sending a list of at least one virtual avatar model to the receiving terminal.

[0176] According to embodiments of this disclosure, the operation method of the sending terminal may include: receiving transmission request information from the receiving terminal.

[0177] According to embodiments of this disclosure, the operation method of the sending terminal may include: sending at least one step for representing a virtual avatar model to the receiving terminal based on transmission request information.

[0178] According to embodiments of this disclosure, the operation method of the sending terminal may include: sending a Session Description Protocol (SDP) proposal to the receiving terminal based on transmission request information.

[0179] According to embodiments of this disclosure, the operation method of the sending terminal may include: receiving an SDP response from the receiving terminal.

[0180] According to embodiments of this disclosure, the operation method of the sending terminal may include: sending at least one step for representing a virtual avatar model to the receiving terminal based on an SDP response.

[0181] According to embodiments of this disclosure, at least one list of virtual avatar models may include an SDP proposal.

[0182] According to embodiments of this disclosure, the transmission request information may include an SDP response.

[0183] According to embodiments of this disclosure, the operation method of the sending terminal may include: sending at least one step for representing a virtual avatar model to the receiving terminal based on an SDP response.

[0184] According to embodiments of this disclosure, at least one list of virtual avatar models can be sent from a sending terminal to a receiving terminal via a server.

[0185] According to embodiments of this disclosure, information regarding at least one step may include at least one of the following: quantity information of at least one step or size information of at least one step.

[0186] According to embodiments of this disclosure, at least one step may include semantic information for representing a virtual avatar model.

[0187] According to embodiments of this disclosure, the operation method of the receiving terminal may include: receiving a list of virtual avatar models from the sending terminal, the list of virtual avatar models including information about at least one step used to represent the virtual avatar model.

[0188] According to embodiments of this disclosure, the operation method of the receiving terminal may include: sending transmission request information to the sending terminal based on at least one list of virtual avatar models.

[0189] According to embodiments of this disclosure, the operation method of the receiving terminal may include: receiving at least one step from the sending terminal for representing a virtual avatar model.

[0190] According to embodiments of this disclosure, the operation method of the receiving terminal may include: receiving a Session Description Protocol (SDP) proposal from the sending terminal.

[0191] According to embodiments of this disclosure, the operation method of the receiving terminal may include: sending an SDP response to the sending terminal based on an SDP proposal.

[0192] According to embodiments of this disclosure, the operation method of the receiving terminal may include: receiving at least one step from the sending terminal, based on an SDP response, for representing a virtual avatar model.

[0193] According to embodiments of this disclosure, at least one list of virtual avatar models may include an SDP proposal.

[0194] According to embodiments of this disclosure, the transmission request information may include an SDP response.

[0195] According to embodiments of this disclosure, the operation method of the receiving terminal may include: receiving at least one step from the sending terminal, based on an SDP response, for representing a virtual avatar model.

[0196] According to embodiments of this disclosure, at least one list of virtual avatar models can be sent from a sending terminal to a receiving terminal via a server.

[0197] According to embodiments of this disclosure, information regarding at least one step may include at least one of the following: quantity information of at least one step or size information of at least one step.

[0198] According to embodiments of this disclosure, at least one step may include semantic information for representing a virtual avatar model.

[0199] According to embodiments of this disclosure, the transmitting terminal may include a transceiver and at least one processor.

[0200] According to embodiments of the present disclosure, at least one processor may be configured to: acquire at least one list of virtual avatar models, the list of at least one virtual avatar model including information for representing at least one step of the virtual avatar model.

[0201] According to embodiments of this disclosure, at least one processor may be configured to send at least one list of virtual avatar models to a receiving terminal.

[0202] According to embodiments of this disclosure, at least one processor may be configured to receive transmission request information from a receiving terminal.

[0203] According to embodiments of the present disclosure, at least one processor may be configured to send at least one step for representing a virtual avatar model to a receiving terminal based on transmission request information.

[0204] According to embodiments of this disclosure, at least one processor may be configured to send a Session Description Protocol (SDP) proposal to a receiving terminal based on a transmission request message.

[0205] According to embodiments of this disclosure, at least one processor may be configured to receive an SDP response from a receiving terminal.

[0206] According to embodiments of the present disclosure, at least one processor may be configured to: send at least one step for representing a virtual avatar model to a receiving terminal based on an SDP response.

[0207] According to embodiments of the present disclosure, at least one processor may be configured to: send at least one step for representing a virtual avatar model to a receiving terminal based on an SDP response.

[0208] According to embodiments of this disclosure, at least one list of virtual avatar models can be sent from a sending terminal to a receiving terminal via a server.

[0209] According to embodiments of this disclosure, information regarding at least one step may include at least one of the following: quantity information of at least one step or size information of at least one step.

[0210] According to embodiments of this disclosure, at least one step may include semantic information for representing a virtual avatar model.

[0211] According to embodiments of this disclosure, the receiving terminal may include a transceiver and at least one processor.

[0212] According to embodiments of the present disclosure, at least one processor may be configured to receive a list of virtual avatar models from a sending terminal, the list of virtual avatar models including information for representing at least one step of a virtual avatar model.

[0213] According to embodiments of this disclosure, at least one processor may be configured to send transmission request information to a sending terminal based on at least one list of virtual avatar models.

[0214] According to embodiments of the present disclosure, at least one processor may be configured to receive at least one step from a transmitting terminal for representing a virtual avatar model.

[0215] According to embodiments of this disclosure, at least one processor may be configured to receive a Session Description Protocol (SDP) proposal from a sending terminal.

[0216] According to embodiments of this disclosure, at least one processor may be configured to send an SDP response to a sending terminal based on an SDP proposal.

[0217] According to embodiments of the present disclosure, at least one processor may be configured to: receive, based on an SDP response, at least one step for representing a virtual avatar model from a transmitting terminal.

[0218] According to embodiments of the present disclosure, at least one processor may be configured to: receive, based on an SDP response, at least one step for representing a virtual avatar model from a transmitting terminal.

[0219] According to embodiments of this disclosure, at least one list of virtual avatar models may include an SDP proposal.

[0220] According to embodiments of this disclosure, the transmission request information may include an SDP response.

[0221] According to embodiments of this disclosure, at least one list of virtual avatar models can be sent from a sending terminal to a receiving terminal via a server.< / rtc> < / ims>

Claims

1. An operating method for a transmitting terminal, the operating method comprising: Obtain at least one list of virtual avatar models, the list of at least one virtual avatar model including information about at least one step for representing the virtual avatar model; Send the list of at least one virtual avatar model to the receiving terminal; Receive transmission request information from the receiving terminal; as well as Based on the transmission request information, at least one step for representing the virtual avatar model is sent to the receiving terminal.

2. The operating method according to claim 1, wherein, Sending at least one step for representing a virtual avatar model includes: Based on the transmission request information, a Session Description Protocol (SDP) proposal is sent to the receiving terminal; Receive an SDP response from the receiving terminal; and Based on the SDP response, at least one step for representing the virtual avatar model is sent to the receiving terminal.

3. The operating method according to claim 1, wherein: The list of at least one virtual avatar model includes SDP proposals. The transmission request information includes an SDP response. The step of sending at least one step for representing a virtual avatar model includes: sending at least one step for representing a virtual avatar model to the receiving terminal based on the SDP response, and The list of at least one virtual avatar model is sent from the sending terminal to the receiving terminal via a server.

4. The operating method according to claim 1, wherein: Information regarding the at least one step includes at least one of the following: quantity information of the at least one step or size information of the at least one step. The at least one step includes semantic information for representing the virtual avatar model.

5. An operating method for a receiving terminal, the operating method comprising: Receive a list of virtual avatar models from the sending terminal, the list of virtual avatar models including information about at least one step used to represent a virtual avatar model; Based on the list of at least one virtual avatar model, a transmission request information is sent to the sending terminal; as well as Receive at least one step from the sending terminal for representing the virtual avatar model.

6. The operating method according to claim 5, wherein, Receiving at least one step for representing a virtual avatar model includes: Receive a Session Description Protocol (SDP) proposal from the sending terminal; Based on the SDP proposal, an SDP response is sent to the sending terminal; and Based on the SDP response, at least one step for representing the virtual avatar model is received from the sending terminal.

7. The operating method according to claim 5, wherein: The list of at least one virtual avatar model includes SDP proposals. The transmission request information includes an SDP response. The step of receiving at least one step for representing a virtual avatar model includes: receiving at least one step for representing a virtual avatar model from the sending terminal based on the SDP response, and The list of at least one virtual avatar model is sent from the sending terminal to the receiving terminal via a server.

8. The operating method according to claim 5, wherein: Information regarding the at least one step includes at least one of the following: quantity information of the at least one step or size information of the at least one step. The at least one step includes semantic information for representing the virtual avatar model.

9. A transmitting terminal, comprising: transceiver; as well as At least one processor, configured as follows: Obtain at least one list of virtual avatar models, the list including information about at least one step for representing the virtual avatar models. Send the list of at least one virtual avatar model to the receiving terminal. Receive transmission request information from the receiving terminal, and Based on the transmission request information, at least one step for representing the virtual avatar model is sent to the receiving terminal.

10. The transmitting terminal according to claim 9, wherein, The at least one processor is configured as follows: Based on the transmission request information, a Session Description Protocol (SDP) proposal is sent to the receiving terminal. Receive an SDP response from the receiving terminal; as well as Based on the SDP response, at least one step for representing the virtual avatar model is sent to the receiving terminal.

11. The transmitting terminal according to claim 9, wherein: The list of at least one virtual avatar model includes SDP proposals. The transmission request information includes an SDP response. The at least one processor is configured to: based on the SDP response, send at least one step to the receiving terminal for representing the virtual avatar model. The list of at least one virtual avatar model is sent from the sending terminal to the receiving terminal via a server.

12. The transmitting terminal according to claim 9, wherein: Information regarding the at least one step includes at least one of the following: quantity information of the at least one step or size information of the at least one step. The at least one step includes semantic information for representing the virtual avatar model.

13. A receiving terminal, comprising: transceiver; as well as At least one processor, configured as follows: Receive a list of virtual avatar models from the sending terminal, the list including information about at least one step used to represent the virtual avatar model. Based on the at least one list of virtual avatar models, a transmission request is sent to the sending terminal, and Receive at least one step from the sending terminal for representing the virtual avatar model.

14. The receiving terminal according to claim 13, wherein, The at least one processor is configured as follows: Receive Session Description Protocol (SDP) proposal from the sending terminal. Based on the SDP proposal, an SDP response is sent to the sending terminal, and Based on the SDP response, at least one step for representing the virtual avatar model is received from the sending terminal.

15. The receiving terminal according to claim 13, wherein: The list of at least one virtual avatar model includes SDP proposals. The transmission request information includes an SDP response. The at least one processor is configured to: receive, based on the SDP response, from the transmitting terminal at least one step for representing a virtual avatar model, and The list of at least one virtual avatar model is sent from the sending terminal to the receiving terminal via a server.

Images