Image content transmission method and device using edge computing service

Abstract

The present disclosure relates to an image content transmission method and apparatus using edge computing services (e.g., multi-access edge computing (MEC) services). The image content transmission method, performed by an edge data network, includes: sending a user field of view image group to an electronic device, the user field of view image group including a user field of view frame group and a first transmission error control group, the user field of view frame group including information about a first user field of view frame; and sending an additional field of view image group to the electronic device, the additional field of view image group including an additional field of view frame group and a second transmission error control group, the additional field of view frame group including information about the first additional field of view frame.

Description

Technical Field

[0001] The present disclosure relates to a method and apparatus for transmitting image content using an edge computing service (eg, a multi-access edge computing (MEC) service). Background Art

[0002] Recently, edge computing technology for sending data using edge servers has been discussed. Edge computing technology may include, for example, multi-access edge computing (MEC) or fog computing (FOC). Edge computing technology is a technology for providing data to electronic devices via a separate server (hereinafter referred to as an "edge data network" or "MEC server"), which is set at a location geographically close to the electronic device, for example, inside or around a base station. For example, an application requiring low latency among at least one application installed in the electronic device can send and receive data via an edge server set at a geographically close location, without passing through a server located in an external data network (DN) (for example, the Internet).

[0003] Recently, services using edge computing technology (hereinafter referred to as "MEC-based services" or "MEC services") have been discussed, and research and development of electronic devices that support MEC-based services has been conducted. For example, an application of an electronic device can send and receive edge computing-based data to and from an edge server (or an application of an edge server) in the application layer.

[0004] As research and development supporting MEC-based services progress, technology for providing high-resolution image content to electronic devices by using MEC has been discussed. Summary of the Invention

[0005] Technical issues

[0006] Based on the above discussion, the present disclosure relates to a method and apparatus for transmitting image content using edge computing services.

[0007] More specifically, the present disclosure provides a method for transmitting image content to an electronic device performed by an edge data network and the edge data network for providing image content to the electronic device.

[0008] In addition, the present disclosure provides a method for receiving image content from an edge data network performed by an electronic device and the electronic device for receiving image content from the edge data network.

[0009] Technical Solution

[0010] According to an embodiment, a method for sending image content performed by an edge data network includes: obtaining orientation information from an electronic device connected to the edge data network; obtaining a first local image, the first local image including a user field-of-view image corresponding to the orientation information and an additional field-of-view image adjacent to the user field-of-view image; generating a first user field-of-view frame by encoding the user field-of-view image using at least one first encoding parameter; generating a first additional field-of-view frame by encoding the additional field-of-view image using at least one second encoding parameter; sending a user field-of-view image group to the electronic device, the user field-of-view image group including a user field-of-view frame group and a first transmission error control group, the user field-of-view frame group including information about the first user field-of-view frame; and sending an additional field-of-view image group to the electronic device, the additional field-of-view image group including an additional field-of-view frame group and a second transmission error control group, the additional field-of-view frame group including information about the first additional field-of-view frame.

[0011] The user field of view image may be an image having a user field of view relative to a position indicated by the orientation information in a virtual reality (VR) image, the user field of view being identified based on information about a viewport area of ​​a display of the electronic device, and the additional field of view image may be an image having a specific additional field of view in the upper, lower, left, and right directions of the user field of view image.

[0012] The first partial image may be a partial image having a specific frame index of a VR sequence including a plurality of frames in the first VR image, and the first partial image may be associated with the orientation information indicating a position of the partial image.

[0013] The user field image packet may be transmitted to the electronic device via a first transport channel, the additional field image packet may be transmitted to the electronic device via a second transport channel, and the transmission error control packet may be a forward error correction (FEC) packet.

[0014] The first encoding parameter and the second encoding parameter may be at least one of a bit rate, a frame rate, and a resolution, the bit rate, the frame rate, and the resolution being identified based on at least one of a required delay, an available bandwidth, a number of consecutive lost packets, a packet loss rate, and a round-trip time (RTT), and the transmission parameters including at least one of an FEC code rate and an FEC block size associated with the FEC packet may include at least one of an FEC code rate and an FEC block size identified based on at least one of the required delay, the available bandwidth, the number of consecutive lost packets, the packet loss rate, and the RTT.

[0015] The method may further include: receiving a request for retransmission of packets that have not yet been recovered from the electronic device based on the FEC packet, the request being sent based on at least one of: the number of packets that have not yet been recovered among the user field of view frame packets, the time period remaining until the decoding of the user field of view image related to the packets that have not yet been recovered, and the RTT; and retransmitting the packets that have not yet been recovered to the electronic device based on the request for retransmission of the packets that have not yet been recovered.

[0016] Sending the user field of view image packets to the electronic device may include: identifying source packets including at least some of the user field of view frame packets, the size of the FEC block including the FEC packets, and the FEC code rate indicating the ratio of the source packets to the FEC blocks based on the packet loss rate, the number of consecutive lost packets, the available bandwidth, and the required time period based on the frame rate and the required delay; identifying the number of FEC blocks based on the number of the user field of view frame packets and the number of the source packets; modifying the number of the source packets included in each FEC block based on the number of the FEC blocks; and obtaining the FEC block based on the modified number of the source packets included in each FEC block.

[0017] The request to retransmit the packet that has not yet been recovered can be received from the electronic device based on the result of the following operation: the electronic device identifies whether at least a portion of the area related to the packet that has not yet been recovered is included in the latest user field of view area based on the latest orientation information of the electronic device.

[0018] It can be based on the FEC code rate, R fps 、R fbr and the available bandwidth of the electronic device to identify the frame rate of the user field of view frame and the bit rate per frame of the user field of view frame (frame data size), R fps It can represent the frame rate of the additional field of view frame compared with the user field of view frame, R fbr The bit rate per frame of the additional field of view frame compared to the user field of view frame may be represented based on the frame rate of the user field of view area and R fps Identify the frame rate of the additional field of view frame, and the frame rate of the additional field of view frame can be determined based on the bit rate per frame of the additional field of view frame and R fbr A per-frame bit rate of the user field of view frames is identified.

[0019] Status information can be received from the electronic device every first period when the edge data network is connected to the electronic device, the status information including at least one of the following: information about the viewport area of ​​the display of the electronic device, information about the frame rate of the additional field of view frames compared with the user field of view frames, information about the bit rate per frame of the additional field of view frames compared with the user field of view frames, and information about the required delay. Channel status information can be received from the electronic device every second period, the channel status information indicating at least one of the number of consecutive lost packets, the packet loss rate and the available bandwidth, and periodically measured RTT information can be received from the electronic device every third period.

[0020] The first partial image may be composed of data units of a preset field of view range, and generating the first user field of view frame may include: obtaining a first data unit of at least one preset field of view range corresponding to the user field of view image in the first partial image; and generating the first user field of view frame by encoding the first data unit of the at least one preset field of view range, and generating the first additional field of view frame may include: obtaining a second data unit of at least one preset field of view range corresponding to the additional field of view image in the first partial image; and generating the first additional field of view frame by encoding the second data unit of the at least one preset field of view range.

[0021] According to an embodiment of the present disclosure, an edge data network for sending image content to an electronic device includes: a network interface; a memory storing one or more instructions; and at least one processor configured to execute the one or more instructions to obtain orientation information from an electronic device connected to the edge data network, obtain a first local image, the first local image including a user field of view image corresponding to the orientation information and an additional field of view image adjacent to the user field of view image, generate a first user field of view frame by encoding the user field of view image using at least one first encoding parameter, generate a first additional field of view frame by encoding the additional field of view image using at least one second encoding parameter, send a user field of view image group to the electronic device, the user field of view image group including a user field of view frame group and a first transmission error control packet, the user field of view frame group including information about the first user field of view frame, and send an additional field of view image group to the electronic device, the additional field of view image group including an additional field of view frame group and a second transmission error control packet, the additional field of view frame group including information about the first additional field of view frame.

[0022] The user field image packet may be transmitted to the electronic device via a first transmission channel, the additional field image packet may be transmitted to the electronic device via a second transmission channel, and the transmission error control packet may be a forward error correction (FEC) packet.

[0023] The first encoding parameter and the second encoding parameter may be at least one of a bit rate, a frame rate, and a resolution, the bit rate, the frame rate, and the resolution being identified based on at least one of a required delay, an available bandwidth, a number of consecutive lost packets, a packet loss rate, and a round-trip time (RTT), and the transmission parameters including at least one of an FEC code rate and an FEC block size associated with the FEC packet may include at least one of an FEC code rate and an FEC block size identified based on at least one of the required delay, the available bandwidth, the number of consecutive lost packets, the packet loss rate, and the RTT.

[0024] The at least one processor can be further configured to execute the one or more instructions to, based on the FEC packet, receive a request for retransmission of the packet that has not yet been recovered from the electronic device, the request being sent based on at least one of: the number of packets that have not yet been recovered among the user field of view frame packets, the time period remaining until the decoding of the user field of view image including the packets that have not yet been recovered, and the RTT, and, based on the request for retransmission of the packet that has not yet been recovered, retransmit the packet that has not yet been recovered to the electronic device.

[0025] The at least one processor can be further configured to execute the one or more instructions to, when sending the user field of view image packets to the electronic device, identify the source packets including at least some of the user field of view frame packets, the size of the FEC block including the FEC packets, and the FEC code rate indicating the ratio of the source packets to the FEC blocks based on the packet loss rate, the number of consecutive lost packets, the available bandwidth, and the required time period based on the frame rate and the required delay, identify the number of FEC blocks based on the number of the user field of view frame packets and the number of the source packets, modify the number of the source packets included in each FEC block based on the number of FEC blocks, and obtain the FEC block based on the modified number of the source packets included in each FEC block.

[0026] The request to retransmit the packet that has not yet been recovered can be received from the electronic device based on the result of the following operation: the electronic device identifies whether at least a portion of the area related to the packet that has not yet been recovered is included in the latest user field of view area based on the latest orientation information of the electronic device.

[0027] It can be based on the FEC code rate, Rfps 、R fbr and the available bandwidth of the electronic device to identify the frame rate of the user field of view frame and the bit rate per frame of the user field of view frame, R fps It can represent the frame rate of the additional field of view frame compared with the user field of view frame, R fbr The bit rate per frame of the additional field of view frame compared to the user field of view frame may be represented based on the frame rate of the user field of view area and R fps Identify the frame rate of the additional field of view frame, and the frame rate of the additional field of view frame can be determined based on the bit rate per frame of the additional field of view frame and R fbr A per-frame bit rate of the user field of view frames is identified.

[0028] A method for providing image content received from an edge data network connected to the electronic device, performed by an electronic device, the method comprising: sending orientation information to the edge data network; obtaining a user field of view image group from the edge data network, the user field of view image group including a user field of view frame group and a first transmission error control group, the user field of view frame group including information about a first user field of view frame obtained by encoding a user field of view image corresponding to the orientation information; obtaining an additional field of view image group from the edge data network, the additional field of view image group including an additional field of view frame group and a second transmission error control group, the additional field of view frame group including information about a first additional field of view frame obtained by encoding an additional field of view image adjacent to the user field of view image; reconstructing a first user field of view image by decoding the first user field of view frame based on the obtained user field of view image group; reconstructing the first additional field of view image by decoding the first additional field of view frame based on the obtained additional field of view image group; and reproducing at least a portion of a first local image including the first user field of view image and the first additional field of view image.

[0029] According to an embodiment of the present disclosure, an electronic device for receiving image content from an edge data network includes: a network interface; a memory storing one or more instructions; and at least one processor configured to execute the one or more instructions to send orientation information to the edge data network, obtain a user field of view image group from the edge data network, the user field of view image group including a user field of view frame group and a first transmission error control group, the user field of view frame group including information about a first user field of view frame obtained by encoding a user field of view image corresponding to the orientation information, and obtain a user field of view image group from the edge data network. The network obtains an additional field of view image group, wherein the additional field of view image group includes an additional field of view frame group and a second transmission error control packet, wherein the additional field of view frame group includes information about a first additional field of view frame obtained by encoding an additional field of view image adjacent to the user field of view image, decodes the first user field of view frame based on the obtained user field of view image group to reconstruct the first user field of view image, decodes the first additional field of view frame based on the obtained additional field of view image group to reconstruct the first additional field of view image, and reproduces at least a portion of a first partial image including the first user field of view image and the first additional field of view image. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a diagram schematically illustrating a multi-access edge computing (MEC) technology in a network environment according to an embodiment of the present disclosure.

[0031] Figure 2 This is a diagram for describing a method of adaptively streaming a group for a user field of view (FoV) image and a group for additional FoV images in an extended FoV image based on directional information, dynamic encoding parameters, and transmission parameters, performed by an edge data network 2000 according to an embodiment of the present disclosure.

[0032] Figure 3 is a flowchart illustrating an operation process between an electronic device and an edge data network according to an embodiment of the present disclosure.

[0033] Figure 4 is a diagram schematically illustrating an operation process between an electronic device and an edge data network.

[0034] Figure 5 is a flowchart illustrating an operation process between an electronic device and an edge data network according to an embodiment of the present disclosure.

[0035] Figure 6 is a flowchart illustrating a method of streaming image content obtained from an edge data network, performed by an electronic device according to an embodiment of the present disclosure.

[0036] Figure 7 is a flowchart illustrating a method of streaming image content obtained from an edge data network, performed by an electronic device according to an embodiment of the present disclosure.

[0037] Figure 8a is a flowchart illustrating a method of streaming image content performed by an edge data network according to an embodiment of the present disclosure.

[0038] Figure 8b is a flowchart illustrating a method for streaming image content performed by an edge data network according to another embodiment of the present disclosure.

[0039] Figure 9 is a flowchart illustrating a method of streaming image content performed by an edge data network according to an embodiment of the present disclosure.

[0040] Figure 10 is a flowchart illustrating a method of streaming image content performed by an edge data network according to an embodiment of the present disclosure.

[0041] Figure 11 is a flowchart illustrating a method of streaming image content obtained from an edge data network, performed by an electronic device according to an embodiment of the present disclosure.

[0042] Figure 12a and Figure 12b is a flowchart illustrating a method of streaming image content obtained from an edge data network, performed by an electronic device according to an embodiment of the present disclosure.

[0043] Figure 12c is a flowchart illustrating a method of streaming image content obtained from an edge data network, performed by an electronic device according to an embodiment of the present disclosure.

[0044] Figure 13 is a flowchart illustrating a method of streaming image content obtained from an edge data network, performed by an electronic device according to an embodiment of the present disclosure.

[0045] Figure 14 is a diagram illustrating a process of transmitting a first partial image in several data units to an electronic device 1000 , performed by the edge data network 2000 , according to an embodiment of the present disclosure.

[0046] Figure 15 is a diagram schematically illustrating an operation process between an electronic device, an edge data network, and a virtual reality (VR) game interface device.

[0047] Figure 16 is a block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] Hereinafter, embodiments of the present disclosure are described in detail with reference to the accompanying drawings.

[0049] In the description of the embodiments, technical features that are well known in the technical field to which the present disclosure belongs but are not directly related to the present disclosure are not described. This is not to confuse the main points of the present disclosure, but to clearly deliver the main points of the present disclosure by omitting any unnecessary description.

[0050] For the same reason, in the accompanying drawings, some elements are exaggerated, omitted or schematically shown. In addition, the size of the elements does not fully reflect their actual sizes. In the accompanying drawings, identical or corresponding elements are provided with the same reference numerals.

[0051] Reference is made below and attached Figure 1 The advantages and features of the present disclosure and methods for achieving them will be apparent from the embodiments described above. However, the present disclosure can be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concept of the present disclosure to those skilled in the art, and the present disclosure will be limited only by the concept of the claims. Like reference numerals represent like elements throughout the specification.

[0052] Here, it may be understood that each block in the process flow chart figures, as well as the combination of the process flow chart figures, can be performed by computer program instructions. These computer program instructions can be loaded into a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the instructions executed by the processor of the computer or other programmable data processing device can generate means configured to perform the functions described in the flow chart blocks. These computer program instructions can also be stored in a computer-usable or computer-readable memory that can direct the computer or other programmable data processing device to implement the functions in a specific mode, so that the instructions stored in the computer-usable or computer-readable memory can also generate a production item involving instruction means configured to perform the functions described in the flow chart blocks. The computer program instructions can also be loaded into a computer or other programmable data processing device, so that a computer-executable process can also be generated by the following operations: a series of operating steps are performed on the computer or other programmable data processing device, so that the instructions executed in the computer or other programmable data processing device provide steps for performing the functions described in the flow chart blocks.

[0053] In addition, each block can indicate a module, segment, or portion of code that includes one or more executable instructions for performing a specific logical function. In addition, in several alternative embodiments, the functions described in the blocks can also be out of order. For example, two blocks shown in succession can actually be executed substantially simultaneously, or the blocks can sometimes be executed in reverse order according to the corresponding functions.

[0054] The term "...unit" used in the embodiments refers to a component comprising software or hardware, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC), and the "...unit" performs certain functions. However, the term "...unit" is not always limited to software or hardware. The "...unit" can be configured to be stored in an addressable storage medium or to execute one or more processors. Thus, for example, "...unit" includes components such as software components, object-oriented software components, class components, and task components, processes, functions, attributes, procedures, subroutines, program code segments, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within components and "...units" can be combined into a smaller number of components and "...units" or further divided into additional components and "...units." Furthermore, components and "...units" can be implemented to replicate one or more central processing units (CPUs) within a device or secure multimedia card. Furthermore, in embodiments, the "...unit" can include one or more processors.

[0055] Figure 1 is a diagram schematically illustrating a multi-access edge computing (MEC) technology in a network environment according to an embodiment of the present disclosure.

[0056] Reference Figure 1 , the network environment 100 of the present disclosure may include an electronic device 1000, an edge data network 2000, a cloud server 3000, and an access network (AN) 1100. However, the components included in the network environment 100 are not limited thereto.

[0057] According to an embodiment, each component included in the network environment 100 may be a physical entity unit, or a software or module unit configured to perform a separate function.

[0058] According to an embodiment, the electronic device 1000 may refer to a device used by a user. For example, the electronic device 1000 may refer to a terminal, a user equipment (UE), a mobile station, a subscriber station, a remote terminal, a wireless terminal, or a user device.

[0059] In addition, the electronic device 1000 may be a terminal for providing content to immerse a user in a virtual environment including at least one of virtual reality (VR), augmented reality (AR), and mixed reality (MR). That is, depending on the embodiment, the electronic device 1000 may be a head-mounted display (HMD) or a virtual reality headset (VRH) for providing content for VR, AR, or MR.

[0060] Reference Figure 1 , the electronic device 1000 may include a first application client (or application client) 122, a second application client 124, and an edge enabler client (or MEC enabling layer (MEL)) 130. The electronic device 1000 can perform necessary operations to use MEC services by using the edge enabler client 130. The edge enabler client 130 is described in detail below.

[0061] According to an embodiment, the electronic device 1000 may execute multiple applications. For example, the electronic device 1000 may execute a first application client 122 and a second application client 124. The multiple applications may require different network services based on at least one of the following: required data transmission rate, latency (or speed), reliability, the number of electronic devices connected to the network, the network access cycle of the electronic device 1000, or average data usage. The different network services may include, for example, enhanced mobile broadband (eMBB), ultra-reliable low latency communication (URLLC), or massive machine type communication (mMTC).

[0062] The application client of the electronic device 1000 may refer to a default application previously installed in the electronic device 1000 or an application provided by a third party. That is, the application client of the electronic device 1000 may refer to a client application to be run in the electronic device 1000 to obtain a specific application service. Several application clients may be run in the electronic device 1000. At least one of the application clients may use the services provided by the edge data network 2000. For example, the application client may be an application installed in the electronic device 1000 and executed by the electronic device 1000, and may provide a function of sending and receiving data via the edge data network 2000. The application client in the electronic device 1000 may refer to application software executed in the electronic device 1000 to use the functions provided by one or more specific edge applications.

[0063] According to an embodiment, multiple applications in the electronic device 1000, that is, the first application client 122 and the second application client 124, can perform data transmission and reception with the cloud server 3000 based on the required network service type, or perform data transmission and reception with the edge data network 2000 based on edge computing. For example, if the first application client 122 does not require low latency, the first application client 122 can perform data transmission and reception with the cloud server 3000. As another example, if the second application client 124 requires low latency, the second application client 124 can perform MEC-based data transmission and reception with the edge data network 2000.

[0064] According to an embodiment, the application in the electronic device 1000 may be referred to as an application client, a client application (or client app), or a UE application (or UE app). For convenience, hereinafter, in this disclosure, the application in the electronic device 1000 is referred to as an application client.

[0065] According to an embodiment, the AN 1100 may provide a channel for wireless communication with the electronic device 1000. For example, the AN 1100 may refer to a radio access network (RAN), a base station, an evolved Node B (eNodeB or eNB), a fifth generation (5G) node, a transmission / reception point (TRP), or a 5G NodeB (5GNB).

[0066] According to an embodiment, the edge data network 2000 may refer to a server that the electronic device 1000 accesses to use MEC services. The edge data network 2000 may be located in a location geographically close to the electronic device 1000, for example, inside or around a base station. According to an embodiment, the edge data network 2000 may send and receive data to and from the electronic device 1000 without passing through an external data network (DN) (e.g., the Internet). In an embodiment, MEC may be referred to as multi-access edge computing or mobile edge computing.

[0067] According to an embodiment, the edge data network 2000 may be referred to as an MEC host, an edge computing server, a mobile edge host, an edge computing platform, an MEC server, or the like. For convenience, hereinafter, in this disclosure, the edge data network 2000 is referred to as an MEC server. Figure 1 , the edge data network 2000 may include a first edge application 142, a second edge application 144, and an edge enabler server (or MEC platform (MEP)) 146. The edge enabler server 146 provides MEC services or performs traffic control or similar control in the edge data network 2000, and the edge enabler server 146 is described in detail below.

[0068] According to an embodiment, the edge data network 2000 can execute multiple applications. For example, the edge data network 2000 can execute a first edge application 142 and a second edge application 144. According to an embodiment, an edge application may refer to an application provided by a third party that provides MEC services in the edge data network, or may be referred to as an edge application. The edge application can be used to establish a data session with an application client to send and receive data related to the application client. That is, the edge application can establish a data session with the application client. In an embodiment, a data session may refer to a communication path established for an application client in the electronic device 1000 and an edge application in the edge data network 2000 to send and receive data to each other.

[0069] Depending on the embodiment, applications in the edge data network 2000 may be referred to as MEC applications (or MEC apps), ME (MEC) apps, edge application servers, and edge applications. For convenience, in this disclosure, applications in the edge data network 2000 are referred to as edge applications. Although the term "application" is used herein, an edge application may refer to an application server present in the edge data network 2000.

[0070] According to an embodiment, the cloud server 3000 can provide content associated with the application. For example, the cloud server 3000 can be managed by a content provider. According to an embodiment, the cloud server 3000 can send data to the electronic device 1000 via an external DN (e.g., the Internet) and receive data from the electronic device 1000.

[0071] Despite Figure 1 Although not shown, a core network (CN) and a DN may exist between the AN 1100 and the edge data network 2000. Depending on the embodiment, the DN may provide services (e.g., Internet services or Internet Protocol Multimedia Subsystem (IMS) services) to the electronic device 1000 by transmitting data (or data packets) to the electronic device 1000 and receiving data (or data packets) from the electronic device 1000 via the CN and the AN 1100. For example, the DN may be managed by a communications provider. In embodiments, the edge data network 2000 may be connected to the AN 1100 or the CN via a DN (e.g., a local DN).

[0072] According to an embodiment of the present disclosure, when the first application client 122 or the second application client 124 is executed by the electronic device 1000, the electronic device 1000 can access the edge data network 2000 via the AN 1100 to send and receive data for executing the application client.

[0073] In the present disclosure, a method for streaming image content between the electronic device 1000, the edge data network 2000, and the cloud server 3000 described above may be provided. More specifically, a method for efficiently providing content for VR, AR, or MR to a user based on user interaction information on an image reproduced by the electronic device 1000 is described. The following describes a method for transmitting image content based on a directional information stream of image content reproduced by the electronic device 1000, performed by the edge data network 2000.

[0074] Figure 2 This is a diagram for describing a method for adaptively streaming a group for a user field of view (FoV) image and a group for an additional FoV image in an extended FoV image based on directional information, dynamic encoding parameters, and transmission parameters, performed by an edge data network 2000 according to an embodiment of the present disclosure.

[0075] Reference Figure 2 , the electronic device 1000 may sense the orientation information 205 and periodically transmit the orientation information to the edge data network 2000 upon a request from the edge data network 2000 or when a value included in the orientation information changes.

[0076] In this case, the orientation information may include an angle value of the gaze of the user of the electronic device 1000, the angle value being measured by using a sensing module of the electronic device 1000. For example, the electronic device 1000 may sense orientation information (e.g., roll, pitch, and yaw values, the orientation information is not limited thereto and may include values ​​of four elements of a quaternion) of the gaze of the user who is looking at a specific partial image of the VR image by using a sensing module (e.g., a three-axis tilt sensor), and the electronic device 1000 may transmit the sensed orientation information to the edge data network 2000 so as to share information about which part of the VR image the user of the electronic device 1000 is currently looking at with the edge data network 2000.

[0077] The edge data network 2000 may obtain a first partial image 200 including a user FoV image 210 corresponding to the orientation information 205 received from the electronic device 1000 , and an additional FoV image 215 consisting of an additional FoV portion surrounding the user FoV image 210 .

[0078] Here, the term "FoV" refers to the range in which the user's eyes can observe a scene (for example, FoV can be expressed as an angle, but is not limited thereto). When observing an image with a larger FoV, the user's eyes can observe a larger area of ​​the image.

[0079] However, the range in which the user can observe an image through the display of the electronic device 1000 may be limited to the size of the viewport area of ​​the electronic device 1000. The size of the viewport area is the size of the FoV (i.e., the size of the display area) in which an image can be reproduced on the display of the electronic device 1000, and the size of the viewport area can be identified based on the capability information of the electronic device 1000. That is, the size of the viewport area is one of the hardware specifications of the display of the electronic device 1000, and the edge data network 2000 can receive the capability information of the electronic device 1000 from the electronic device 1000 via a network connection therewith, and identify the size of the viewport area of ​​the electronic device 1000 based on the capability information.

[0080] For example, in a case where the size of the viewport area of ​​the electronic device 1000 in the horizontal direction is 110° and the size of the viewport area of ​​the electronic device 1000 in the vertical direction is 90°, the electronic device 1000 can send capability information of the electronic device 1000 including information about the size of the viewport area to the edge data network 2000 via a network connection, and the edge data network 2000 can identify that the size of the viewport area in the horizontal direction is 110° and the size of the viewport area in the vertical direction is 90° based on the capability information.

[0081] Accordingly, the term "user FoV" (or "default FoV") may refer to an area observable by the user through the viewport area of ​​the electronic device 1000. In a case where the orientation of the electronic device 1000 changes according to the temporal movement of the user's head, and thus the area of ​​the image observable by the user changes, the user cannot observe the entire area of ​​the image observable with the existing user FoV, and thus the term "additional FoV" may refer to a FoV obtained in consideration of such a situation (for example, upper, lower, left, and right additional FoVs around the user FoV).

[0082] The FoV of the first partial image encoded to be displayed on the electronic device 1000 may be an extended FoV including the user FoV (default FoV) and the additional FoV. That is, the first partial image may be an extended FoV image including the user FoV image (default FoV image) and the additional FoV image.

[0083] Edge data network 2000 may encode user FoV image 210 to generate a first user FoV frame. Additionally, edge data network 2000 may encode additional FoV image 215 to generate a first additional FoV frame. For example, to account for the shape of additional FoV image 215 (e.g., a quadrilateral donut shape), additional FoV image 215 may be encoded as a quadrilateral. The first additional FoV frame may be generated by replacing user FoV image 210 with a specific color (e.g., black) and encoding an image including additional FoV image 215 and user FoV image 210 replaced with the specific color. In other words, by expressing the entire specific continuous area with a single color value, the resulting encoding consumes minimal capacity. Furthermore, the encoding of additional FoV image 215 can be performed, as described above, taking into account that existing encoding modules receive and encode quadrilateral images. However, the present disclosure is not limited thereto, and those skilled in the art will appreciate that information in additional FoV image 215 may be encoded using various methods. For example, the edge data network 2000 can generate a first additional FoV frame by replacing the user FoV image 210 with a specific pattern and encoding an image including the additional FoV image 215 and the user FoV image 210 replaced with the specific pattern. Alternatively, the edge data network 2000 can rearrange the samples of the additional FoV image 215 according to a specific scanning order to change the shape of the additional FoV image 215 to a quadrilateral shape, and then encode the resulting image having a quadrilateral shape. In this case, each of the FoV images 210 and 215 can be encoded using dynamic encoding parameters 202 based on state information. Here, the state information may include orientation information and network state information. In addition, the state information may include information about the required delay given in the VR application.

[0084] The network status information may be information about the network status obtained by continuously monitoring packets received by the electronic device 1000 or by performing round-trip time (RTT) measurements. For example, the network status information obtained based on packets received by the electronic device 1000 may be information about the packet loss rate, available bandwidth, throughput, and the number of consecutive lost packets, etc., obtained by performing periodic monitoring. Furthermore, the network status information obtained based on an RTT measurement request message sent from the electronic device 1000 may be information about the RTT measured by the electronic device 1000 (RTT is related to the current latency). That is, the electronic device 1000 may start a timer at the time the RTT measurement request message is sent to the edge data network 2000 and measure the RTT based on the timer. The RTT is the period between the time the RTT measurement request message is sent and the time a response to the RTT measurement request message is received. The RTT measurement request message may include a certain number of packets.

[0085] Meanwhile, dynamic encoding parameters based on state information can include image quality-related parameters and frame rate (e.g., frames per second). Image quality-related parameters can include resolution, bitrate per second, bitrate per frame, etc. Here, frames per second refers to the number of frames reproduced per second. As the value increases, the reproduced image appears natural, while as the value decreases, the reproduced image appears unnatural and lagging. The bitrate of an image is one parameter that indicates image quality and can indicate, for example, the target number of bits (or bytes or packets) per second when encoding the image. For the same resolution (or size), encoding an image at a higher bitrate can result in higher quality than encoding an image at a lower bitrate. The bitrate can be used as a parameter input to the encoding module to adjust the image quality. However, the present disclosure is not limited to this, and the bitrate per frame (also known as the frame data size) can further be used as a parameter indicating image quality. The data size of each frame refers to the amount of frame data transmitted (data obtained by encoding the frame) and can be expressed in bits or bytes. Generally speaking, as image quality / resolution increases, the bitrate can increase. That is, the data size (and frame rate) of each frame can be determined based on the available bandwidth, the resolution can be adjusted relative to the determined data size of each frame, or the image quality (parameters related to image quality, such as bit rate per second) can be adjusted by using various related technology video rate adaptation technologies, thereby identifying the resolution and bit rate.

[0086] The resolution of an image refers to the number of pixels of a frame to be encoded, and for the same pixel size, as the resolution increases, the size and quality of the image may increase. The resolution of an image may be related to the size of the image.

[0087] In detail, the edge data network 2000 may identify dynamic encoding parameters 202 based on the state information as described below. For example, the edge data network 2000 may identify appropriate encoding parameters so as to appropriately allocate bandwidth resources to the user FoV image and the additional FoV image according to the ratio of the frame data size or frame rate of each FoV image, taking into account the bandwidth available to the electronic device 1000. Figure 5 Describes details of identifying encoding parameters based on status information.

[0088] In this case, the edge data network 2000 may encode the user FoV image 210 to generate a user FoV frame by using relatively high image quality parameters (eg, high bit rate) and a high frame rate, and generate a user FoV frame packet 220 including information about the user FoV frame.

[0089] The edge data network 2000 may encode the additional FoV image 215 to generate an additional FoV frame by using relatively low image quality parameters (eg, low bit rate) and a low frame rate, and generate an additional FoV image packet 245 including information about the additional FoV frame.

[0090] Meanwhile, regarding the user FoV image 210, in addition to the user FoV frame packet 220, the edge data network 2000 may also generate a transmission error control packet 225 corresponding to the user FoV frame packet 220 for controlling transmission errors. Examples of transmission error control packets include forward error correction (FEC) packets. FEC packets are packets with error correction capabilities, and electronic device 1000 can correct errors associated with lost packets in received user FoV frame packets 220 based on the FEC packets. If the errors associated with lost packets are uncorrectable based on the FEC packets, electronic device 1000 can request retransmission of the lost packets. In this case, edge data network 2000 may generate transmission error control packet 225 based on transmission parameters 202 based on the status information. Transmission parameters are parameters related to the generation of transmission error packets. If the transmission error control packet is an FEC packet, the transmission parameters may be the FEC block size and the FEC code rate. The FEC block size may refer to the sum of the number of source packets and the number of FEC packets within an FEC block. The FEC code rate is a value obtained by dividing the number of FEC packets by the FEC block size, and can indicate the ratio of meaningful data (encoded image data and frames) to the total data. Source packets refer to packets that are subject to error correction based on FEC packets, and user FoV frame packets 220 can be an example of source packets. However, considering the FEC block size, some user FoV frame packets 220 can be source packets within a single FEC block.

[0091] The edge data network 2000 may generate a transmission error control packet 225 (e.g., an FEC packet) for the user FoV frame packet 220 based on the FEC block size and FEC code rate, which are dynamic transmission parameters based on the state information. The edge data network 2000 may transmit a user FoV image packet 230 including the user FoV frame packet 220 and the transmission error control packet 225 to the electronic device 1000.

[0092] At the same time, the edge data network 2000 may send an additional FoV image packet 245 to the electronic device 1000 .

[0093] For example, regarding the additional FoV image 215, the edge data network 2000 may generate a transmission error control packet 243 corresponding to the additional FoV frame packet 240 for controlling transmission errors in addition to the additional FoV frame packet 240. In this case, if the transmission error control packet included in the additional FoV image packet 245 is an FEC packet, the FEC code rate may be 0, but is not limited thereto. That is, if the FEC code rate is 0, the FEC packet that is the transmission error control packet is not included.

[0094] Here, the transport protocol associated with encapsulating each packet may be, but is not limited to, the User Datagram Protocol (UDP). Compared to the Transmission Control Protocol (TCP) used as a transport protocol, UDP has lower reliability but higher transmission rates, and thus can be used for streaming services where continuity or timeliness is more important than reliability.

[0095] The edge data network 2000 may transmit the user FoV image packet 230 and the additional FoV image packet 245 to the electronic device 1000 via different transmission channels. The term "transmission channel" may refer to a channel (i.e., a path used to transmit signals or information) for transmitting and receiving data between the edge data network 2000 and the electronic device 1000 in a transmission layer. In other words, the transmission channel may be a path used to exchange data (e.g., packets) in an end-to-end manner between a transmitting side (e.g., an edge data network) and a receiving side (e.g., an electronic device).

[0096] Based on the port number of the transport protocol (i.e., the destination port number) included in (the header of) user FoV image packet 230, electronic device 1000 can transmit user FoV frame packet 220 to an application process of the application layer associated with processing user FoV frame packet 220 (i.e., decoding the user FoV frame). Furthermore, based on the port number of the transport protocol included in (the header of) additional FoV image packet 245, electronic device 1000 can transmit additional FoV image packet 245 to an application process associated with processing additional FoV image packet 245 (i.e., decoding the additional FoV image). In other words, the port numbers of each packet can be different from one another, and the application process to which the packet is to be transmitted can be identified based on the port number.

[0097] The electronic device 1000 may reconstruct a user FoV image by decoding the user FoV frame based on the user FoV frame group 220. In addition, the electronic device 1000 may reconstruct an additional FoV image by decoding the additional FoV frame based on the additional FoV frame group 240.

[0098] Meanwhile, if the electronic device 1000 does not receive some user FoV frame packets 220 (i.e., packet loss has occurred), the electronic device 1000 may recover the lost packets based on the transmission error control packet 225. If the recovery of the lost packets is successful, the electronic device 1000 may not request retransmission of the lost packets.

[0099] In the event that recovery of lost packets fails, the electronic device 1000 may selectively send a request to the edge data network 2000 to retransmit lost packets that have not yet been recovered.

[0100] In this case, the retransmission request may include a number (i.e., a sequence number) indicating the lost packet. That is, only when it is predicted that the lost packet will be received before the decoding of the user FoV frame is completed by considering the RTT and the remaining time period until the decoding of the user FoV frame is completed, can the electronic device 1000 send a request to retransmit the lost packet to the edge data network 2000. Only when the request to retransmit the lost packet is received from the electronic device 1000, can the edge data network 2000 retransmit the lost packet in response to the request.

[0101] The electronic device 1000 may obtain a partial image including the reconstructed user FoV image and the additional FoV image, obtain a user FoV image 250 from the partial image based on the latest orientation information obtained from the sensing module, and display the user FoV image 250 on the display of the electronic device 1000 .

[0102] By transmitting the updated orientation information to the edge data network 2000, the electronic device 1000 can receive the user FoV image group and the additional FoV image group associated with the next image to be reproduced. Until the next image to be reproduced is received based on the transmitted orientation information, the electronic device 1000 can obtain the user FoV image at the current time point from the previously reconstructed partial image based on the orientation information periodically obtained from the sensing module.

[0103] Accordingly, even in the case where the motion-to-image delay (MTP delay) at the time point when the next image to be reproduced is received increases, the edge data network 2000 can prevent the user from seeing the black edge by performing local processing based on the additional FoV image so as to achieve sufficient offset within the time period required to receive the next image to be reproduced, thereby substantially compensating for the MTP delay. Here, the MTP delay may refer to the following time period: the edge data network 2000 recognizes the user's movement (e.g., the movement of the user's gaze) based on sensor information (e.g., orientation information) obtained by the sensing module of the electronic device 1000 and transmitted to the edge data network 2000, the edge data network 2000 provides a frame of a partial image in the VR image based on the sensor information, and the electronic device 1000 performs decoding and rendering on the provided frame of the partial image and then displays the rendered frame on the display.

[0104] As described above, by encoding each of the user FoV image and the additional FoV image using encoding parameters, the edge data network 2000 can perform parallel and independent processing on the user FoV image and the additional FoV image. Specifically, by encoding the user FoV image corresponding to the area the user is directly looking at at the time the image is requested at high quality and a high frames per second (FPS) and transmitting the encoded result, and encoding the additional FoV images corresponding to other areas at low quality and a low FPS and transmitting the encoded result, network resources can be efficiently utilized and parallel and independent image processing can be achieved. Specifically, by transmitting the additional FoV image with low utilization efficiency at low quality and a low frame rate, and transmitting the user FoV image with high utilization efficiency at high quality and a high frame rate, network resources can be efficiently utilized.

[0105] That is, when the user of the wearable electronic device 1000 moves his / her head, the orientation information can be updated, and the user's FoV image based on the orientation information updated in real time can be displayed on the display. In this case, because the user's focus usually moves smoothly, the focus usually does not deviate from the range of the user's FoV image corresponding to the orientation information sent in response to the image request, and because the next image is subsequently requested and received based on the updated orientation information, the additional FoV image does not need to be encoded with high image quality and high frame rate, but can be encoded with low image quality and low frame rate.

[0106] The edge data network 2000 can identify dynamic encoding parameters and transmission parameters based on the state. That is, in order to appropriately allocate resources for the user FoV image and the additional FoV image according to the available bandwidth and required latency given by the VR application, the edge data network 2000 can identify transmission parameters such as the FEC code rate or FEC block size, as well as encoding parameters related to the image quality of each area (e.g., frame data size) and encoding parameters such as the frame rate.

[0107] As described above, given the short RTT between the edge data network 2000 and the electronic device 1000, the edge data network 2000 can selectively request retransmission of lost packets only when it is expected that the retransmitted lost packets will be successfully received before image decoding is completed. Furthermore, the edge data network 2000 can quickly attempt to recover lost packets by sending FEC packets while preparing for the loss of user FoV frame packets (which significantly affects the quality of experience (QoE)). Furthermore, the QoE can be improved by requesting retransmission of lost packets only when the lost packets are unrecoverable.

[0108] Figure 3 is a flowchart illustrating an operation process between an electronic device and an edge data network according to an embodiment of the present disclosure.

[0109] Reference Figure 3 In operation S300, the edge data network 2000 may establish a network connection with the electronic device 1000, and may share state information required for the operation of the edge data network 2000 and the electronic device 1000 with the electronic device 1000. In this case, the state information may be shared only initially, periodically, or aperiodically, only when a request is received or when a state information change occurs.

[0110] For example, the edge data network 2000 may receive capability information of the electronic device 1000 from the electronic device 1000. For example, the capability information may be information indicating operation-related capabilities of the electronic device 1000, such as information about the size of a viewport area of ​​the electronic device 1000.

[0111] In addition, the capability information may include the required latency for the VR application, the initial ratio (R) of the frame data size or frame rate of the additional FoV image to the frame data size or frame rate of the user FoV image, and the initial ratio (R) of the frame data size or frame rate of the additional FoV image to the frame data size or frame rate of the user FoV image. size or R fps ) and the initial frame rate and initial frame data size of the user FoV image. For example, in full high definition (FHD) resolution, the required delay may be 20 ms, R size or R fps It may be 0.5, the initial frame rate may be 60 fps, and the initial frame data size may be several megabits to tens of megabits per frame.

[0112] In operation S302, the electronic device 1000 may obtain orientation information. The orientation information may be information indicating an angle component.

[0113] In operation S304, the electronic device 1000 may transmit the orientation information to the edge data network 2000. The electronic device 1000 may periodically transmit the sensor information to the edge data network 2000, but is not limited thereto and may aperiodically transmit the sensor information to the edge data network 2000. For example, the electronic device 1000 may transmit the sensor information only upon a request from the edge data network 2000 or when the value of the sensor information changes.

[0114] In operation S306, the edge data network 2000 may obtain a first partial image including a user FoV image corresponding to the orientation information and an additional FoV image adjacent to the user FoV image. The user FoV image may be an image having a user FoV identified based on information about a display of a viewport area of ​​the electronic device 1000, based on a position indicated by the orientation information in the VR image. The first partial image may be a partial image having a specific frame index of a VR sequence including a plurality of frames in the first VR image, and the first partial image may be associated with the orientation information indicating the position of the partial image.

[0115] In operation S308, the edge data network 2000 may encode the user FoV image using at least one first encoding parameter to generate a first user FoV frame. In this case, the first encoding parameter and the second encoding parameter to be described below may be encoding parameters identified based on the state information. Figure 5 The encoding parameters identified based on the state information are described. The first encoding parameters may include encoding parameters indicating a higher quality than the second encoding parameters for the additional FoV frame (eg, a high frame data size and a high bit rate per frame or a high bit rate or frames per second).

[0116] In operation S310 , the edge data network 2000 may generate a first additional FoV frame by encoding the additional FoV image using at least one second encoding parameter.

[0117] In operation S312, the edge data network 2000 may generate a user FoV image packet, the user FoV image packet including a user FoV frame packet and a first transmission error control packet determined by using a first transmission parameter, the user FoV frame packet including information about the first user FoV frame. In this case, the edge data network 2000 may generate a transmission error control packet based on the first transmission parameter identified based on the state information. For example, in the case where the transmission error control packet is an FEC packet, the transmission parameter may be an FEC block size and an FEC code rate. Referring to FIG. Figure 6 Describes the transmission parameters identified based on the status information.

[0118] In operation S314, the edge data network 2000 may generate an additional FoV image packet, the additional FoV image packet including an additional FoV frame packet and a second transmission error control packet determined by using the second transmission parameter, the additional FoV frame packet including information about the first additional FoV frame. In this case, the edge data network 2000 may generate the transmission error control packet based on the second transmission parameter identified based on the state information. In this case, if the second transmission error control packet included in the additional FoV image packet is an FEC packet, the FEC code rate may be 0, but is not limited thereto.

[0119] In operation S316 , the edge data network 2000 may transmit the user FoV image packet to the electronic device 1000 .

[0120] In operation S318, the edge data network 2000 may transmit the additional FoV image packet to the electronic device 1000. In this case, the additional FoV image packet and the user FoV image packet may be transmitted through different transmission channels.

[0121] In operation S320, based on the user FoV image grouping, the electronic device 1000 may decode the first user FoV frame to reconstruct the first user FoV image. In this case, in the event that some user FoV frame packets of the user FoV image grouping are lost, the electronic device 1000 may attempt to recover the lost packets based on the transmission error control packets with an error correction function among the user FoV image packets, and in the event that the lost packets are successfully recovered, the electronic device 1000 may further decode the first user FoV frame based on the recovered packets to reconstruct the first user FoV image. In the event that recovery of the lost packets has failed, the electronic device 1000 may selectively request retransmission of the packets that have not yet been recovered. Referring to the following Figure 7 and Figures 12a to 12c Describes the selective retransmission of packets that have not been recovered.

[0122] In operation S322, the electronic device 1000 may decode the first additional FoV frame based on the additional FoV image packets to reconstruct the first additional FoV image. In this case, because the additional FoV image packets may not include any transmission error control packets and the errors cannot be corrected, it is possible to selectively request retransmission of lost packets without attempting to recover the lost packets. However, the present disclosure is not limited to this, and considering the low importance of the additional FoV image packets, retransmission of lost packets may not always be requested. The operation of selectively retransmitting lost packets among the additional FoV image packets performed by the electronic device 1000 is substantially the same as the above-mentioned operation of selectively retransmitting packets that have not yet been recovered among the user FoV frame packets, and therefore, a detailed description thereof is omitted.

[0123] In operation S324, the electronic device 1000 may reproduce a portion of the first partial image including the first user FoV image and the first additional FoV image. For example, the electronic device 1000 may obtain the latest orientation information at the time of reproduction and reproduce the portion of the first partial image corresponding to the latest orientation information.

[0124] Figure 4 is a diagram schematically illustrating an operation process between an electronic device and an edge data network.

[0125] According to an embodiment, the electronic device 1000 may include a sensing module 410, a network interface 420, a processor 430, and a memory 440. However, the components in the electronic device 1000 are not limited thereto, and the electronic device 1000 may include more or fewer components.

[0126] The electronic device 1000 may decode the image received from the edge data network 2000 or the cloud server 3000 and display the image obtained as a result of the decoding on the display of the electronic device. In addition, the electronic device 1000 may obtain orientation information by using the sensing module 410.

[0127] The electronic device 1000 transmits the orientation information to the edge data network 2000 by using the network interface 420. In this case, the electronic device 1000 may transmit the frame index information obtained when the orientation information is sensed to the edge data network 2000. However, the present disclosure is not limited thereto, and the reference frame information among the reconstructed frames may be transmitted to the edge data network 2000 before the orientation information is sensed. Here, the frame index may be information indicating the encoding / decoding order of the frames, but is not limited thereto, and the frame index may be information indicating the rendering order of the frames.

[0128] The processor 430 controls the overall operation of the electronic device 1000 by executing one or more instructions in the memory 440. For example, the processor 430 can control the sensing module 410 and the network interface 420 by executing one or more instructions stored in the memory 440. According to an embodiment, the processor 430 can obtain a user FoV image packet from the edge data network 2000. The user FoV image packet includes a user FoV frame packet and a transmission error control packet. The user FoV frame packet includes information about a first user FoV frame obtained by encoding a user FoV image corresponding to the orientation information. In this case, the user FoV image can be encoded using a first encoding parameter, and the first encoding parameter can be identified based on the state information. The first encoding parameter is described above. At the same time, the transmission error control packet can be generated based on the transmission parameter identified by the edge data network 2000 based on the state information. The transmission parameter based on the state information is described above.

[0129] The processor 430 may obtain an additional FoV image packet from the edge data network 2000, the additional FoV image packet including information about a first additional FoV frame obtained by encoding an additional FoV image adjacent to the user FoV image. In this case, the additional FoV image may be encoded using a second encoding parameter, and the second encoding parameter may be identified based on the state information. The second encoding parameter is described above.

[0130] Based on the user FoV image packets, processor 430 may decode the first user FoV frame to reconstruct the user FoV image. If some of the user FoV frame packets are lost, processor 430 may attempt to recover the lost packets based on the transmission error control packets included in the user FoV image packets. If the lost packets are successfully recovered, processor 430 may further decode the first user FoV frame based on the recovered packets to reconstruct the user FoV image. If recovery of the lost packets fails, processor 430 may selectively determine whether to request retransmission of the lost packets. For example, based on the RTT and the remaining time until decoding of the user FoV frame is complete, processor 430 may determine whether to request retransmission of packets that have not yet been recovered. If it is determined that retransmission of packets that have not yet been recovered is to be requested, processor 430 may send a request to electronic device 1000 to retransmit the packets that have not yet been recovered and re-receive the packets. The processor 430 may further decode the first user FoV frame based on the re-received packets that have not been restored and are therefore retransmitted, thereby reconstructing the first user FoV image.

[0131] The processor 430 may reproduce a portion of the first partial image including the reconstructed first user FoV image and the first additional FoV image. That is, the processor 430 may obtain orientation information at the time of reproduction from the sensing module 410, obtain a portion of the first partial image based on the orientation information, and reproduce the obtained portion of the first partial image.

[0132] According to an embodiment, the memory 440 may include, but is not limited to, a first decoder module 441 and a second decoder module 442 storing instructions for decoding data generated by encoding frames (e.g., the first user FoV frame and the first additional FoV frame) and then received from the edge data network 2000. Furthermore, the memory 440 may include, but is not limited to, a user FoV image packet processing module 443 storing instructions for processing user FoV image packets obtained from the edge data network 2000. Furthermore, the memory 440 may include, but is not limited to, an additional FoV image packet processing module 444 storing instructions for processing additional FoV image packets obtained from the edge data network 2000.

[0133] In addition, the memory 440 may include, but is not limited to, a network status monitoring module 445, which stores instructions for the following operations: sending first network status information identified based on packets received from the edge data network 2000 and then processed, and an RTT measurement request message to the edge data network 2000; obtaining information about the measured RTT (i.e., second network status information) by receiving a response to the information and the message; and periodically or on demand sending network information including the first network status information and information about the measured RTT to the edge data network 2000.

[0134] In addition, the memory 440 may include, but is not limited to, a selective lost packet retransmission request module 446 that stores instructions for the following operations: when there are lost packets in some user FoV frame packets included in the user FoV image packets and recovery of the lost packets has failed, requesting retransmission of packets that have not been recovered based on the time period and RTT remaining until decoding of the user FoV image is completed, or when there are lost packets in some additional FoV image packets, requesting retransmission of the lost packets based on the time period and RTT remaining until decoding of the additional FoV image is completed.

[0135] According to an embodiment, the edge data network 2000 may include a network interface 460, a processor 470, and a memory 480. However, the components of the edge data network 2000 are not limited thereto, and the edge data network 2000 may include more components or some components may be omitted from the edge data network 2000.

[0136] The edge data network 2000 may obtain the orientation information 452 from the electronic device 1000 by using the network interface 460, and transmit to the electronic device 1000 a user FoV image packet 454 regarding the user FoV frame and an additional FoV image packet 456 generated by encoding performed by the edge data network 2000 based on the orientation information 452. In this case, the user FoV image packet 454 and the additional FoV image packet 456 may be transmitted through separate transmission channels.

[0137] The processor 470 controls the overall operation of the edge data network 2000 by executing one or more instructions in the memory 480 .

[0138] For example, the processor 470 may obtain a first partial image including a user FoV image corresponding to the orientation information and an additional FoV image adjacent to the user FoV image, and encode the user FoV image using at least one first encoding parameter to generate a first user FoV frame. In this case, the first user FoV frame may be an intra-frame coded frame (I-frame) or a predictive coded frame (P-frame) (or a bidirectionally predicted frame (B-frame)).

[0139] The processor 470 may generate a first additional FoV frame by encoding the additional FoV image using the at least one second encoding parameter.

[0140] In this case, the at least one first encoding parameter and the at least one second encoding parameter may be encoding parameters identified based on status information including at least one of directional information and network status information.

[0141] Processor 470 may generate user FoV image packets 454, including user FoV frame packets and transmission error control packets, wherein the user FoV frame packets include information about the first user FoV frame. Processor 470 may generate transmission error control packets (e.g., FEC packets) corresponding to some (source packets) of the first user FoV frame packets using transmission parameters identified based on state information (e.g., FEC block size and FEC code rate when the transmission error control packets are FEC packets).

[0142] The processor 470 may generate an additional FoV image group 456 including information about the first additional FoV frame.

[0143] The processor 470 may transmit the user FoV image group 454 and the additional FoV image group 456 to the electronic device 1000. In this case, the user FoV image group 454 and the additional FoV image group 456 may be transmitted through separate transmission channels.

[0144] The processor 470 may periodically obtain the directional information and the network status information, and identify the encoding parameters (eg, the first encoding parameters and the second encoding parameters) and the transmission parameters based on at least one of the directional information and the network information. Figure 5 and Figure 6 Describe this.

[0145] When the processor 470 receives a request for retransmission of a lost packet from the edge data network 2000, the processor 470 may retransmit the lost packet to the edge data network 2000 in response to the request. Figure 7 and Figures 12a to 12c Describe this.

[0146] According to an embodiment, the memory 480 may include: a first encoder module 481 and a second encoder module 482, which store instructions for encoding a user FoV image and an additional FoV image to be sent by the edge data network 2000 to the electronic device 1000; a FoV image group processing module 483, which is used to generate and process user FoV image groups based on information about the user FoV frame; an additional FoV image group processing module 484, which is used to generate and process additional FoV image groups based on information about the additional FoV frame; a coding parameter and transmission parameter identification module 485, which is used to identify coding parameters and transmission parameters based on status information; and a lost packet retransmission module 486, which is used to retransmit lost packets to the edge data network 2000 in response to a request for retransmission of lost packets received from the edge data network 2000, and the memory 480 may store a VR sequence 487 including the entire image data of the VR sequence, but the memory 480 is not limited thereto.

[0147] For example, the edge data network 2000 may store the VR sequence including all frames in a database (DB). The edge data network 2000 may identify the first partial image including the user FoV image and the additional FoV image from the VR sequence stored in the DB by using the orientation information obtained from the electronic device 1000.

[0148] Figure 5 is a flowchart illustrating an operation process between an electronic device and an edge data network according to an embodiment of the present disclosure.

[0149] Reference Figure 5 , in operation S500, the electronic device 1000 may obtain orientation information.

[0150] In operation S502, the electronic device 1000 may transmit orientation information to the edge data network 2000. For example, the orientation information may be transmitted periodically. For example, the orientation information may be transmitted every 5 ms, but this is not limited to this. The orientation information may be transmitted upon a request from the edge data network 2000 or when the previous orientation information changes. Simultaneously, the electronic device 1000 may transmit a request for a user FoV image and an additional FoV image (i.e., a request for an extended FoV image) to the edge data network 2000 based on the orientation information. However, the electronic device 1000 may not always transmit a request for an extended FoV, and in some cases, may transmit only the orientation information without a request for an extended FoV. Alternatively, the request for the user FoV image and the additional FoV image may be transmitted periodically. In this case, the period may be identified in consideration of the frame rate.

[0151] In operation S504 , the edge data network 2000 may obtain, from the VR image, a first partial image including a user FoV image corresponding to the orientation information and an additional FoV image adjacent to the user FoV image.

[0152] In operation S506, the electronic device 1000 may transmit network status information to the edge data network 2000. The network status information may be transmitted periodically, but is not limited thereto, and may be transmitted upon a request from the edge data network 2000 or when previous network information changes. Meanwhile, the network status information may include available bandwidth, packet loss rate, number of consecutive lost packets, throughput, etc., identified based on packets received and processed by the electronic device 1000. Furthermore, the network status information may include information on the RTT measured based on an RTT measurement request transmitted from the electronic device 1000.

[0153] The network status information may be periodically updated according to a first period based on packets received and processed by the electronic device 1000, and the updated network status information may be periodically transmitted to the electronic device 1000 according to a second period. In this case, the first period may be shorter than the second period. For example, the first period may be 50 ms and the second period may be 2 s, but the present invention is not limited thereto.

[0154] At the same time, network information identified based on packets received and processed by electronic device 1000 may be transmitted every first period, and information about the measured RTT may be transmitted every second period immediately after RTT measurement. In this case, the first period may be shorter than the second period, but is not limited thereto. For example, the first period may be 2 seconds and the second period may be 5 seconds, but is not limited thereto.

[0155] In operation S508, the edge data network 2000 may identify at least one first encoding parameter and at least one second encoding parameter based on state information including at least one of orientation information and network state information. The first encoding parameter may be a parameter for encoding the user FoV image, and the second encoding parameter may be a parameter for encoding the additional FoV image.

[0156] For example, in the case where it is recognized based on the previously obtained orientation information that there is a large amount of movement of the electronic device 1000 (eg, rotation of the user's head), the edge data network 2000 may recognize that R fps and R size Greater than the first threshold. Here, R fps may represent the ratio of the frame rate of the additional FoV image to the frame rate of the user FoV image, and R sizeThe ratio of the frame data size of the additional FoV image to the frame data size of the user FoV image may be indicated. For example, the value may be substantially 0.5, but may be greater than 0.5 if a large amount of movement of the electronic device 1000 (e.g., rotation of the user's head) is detected.

[0157] On the contrary, in the case where a small amount of movement of the electronic device 1000 (eg, rotation of the user's head) is recognized based on the previously obtained orientation information, the edge data network 2000 may recognize that R fps and R size For example, their values ​​may be substantially 0.5, but in the case where a large amount of movement of the electronic device 1000 (eg, rotation of the user's head) is recognized, they may be recognized as being less than 0.5.

[0158] The edge data network 2000 can identify the frame rate and frame size of the user FoV image based on the state information, and can fps and R size Identify the frame rate and frame size of the additional FoV images.

[0159] In this case, the edge data network 2000 may first identify transmission parameters such as FEC code rate and FEC block size based on the state information, and identify the frame rate and frame size of the user FoV image based on the FEC block size and FEC code rate. Figure 6 A description is provided of how the edge data network 2000 identifies the FEC block size and FEC code rate based on state information.

[0160] The edge data network 2000 may identify the frame rate and frame data size of the user FoV image based on the following Equation 1.

[0161] [Formula 1]

[0162] (FR fov * FS fov )(1 + N fec / N block )+(FR nonfov *FS nonfov )<= ABW

[0163] (FR fov * FS fov )(1 + N fec / N block + R fps *R size )<= ABW

[0164] (FR fov * FS fov)<= ABW / (1 + N fec / N block + R fps *R size )

[0165] Here, the frame data rate of the user FoV image (the image corresponding to the FoV area) is FR fov , the frame size of the user FoV image (the image corresponding to the FoV area) is FS fov , the frame rate of the additional FoV image (the image corresponding to the non-FoV area) is FR nonfov , the frame size of the additional FoV image is FS nonfov , FR nonfov Can be FR fov * R fps , and FS nonfov Can be FS fov * R size ABW can represent the available bandwidth of the electronic device 1000. In addition, N fec It can represent the number of FEC packets in the FEC block, N block It can represent the number of packets (source packets + FEC packets) in the FEC block, so N fec / N block It can indicate the FEC code rate of the user FoV image group.

[0166] Therefore, by using N previously identified based on the state information fec / N block 、R fps 、R size Etc., the frame data size (a parameter related to the quality of the frame) and frame rate of the user's FoV image can be identified.

[0167] Based on Equation 1, the frame rate and frame data size of the user FoV image can be identified by using various average bit rate (ABR) algorithms. In this case, examples of ABR algorithms include a method of first adjusting the frame data size and then adjusting the frame rate to prioritize image quality, or a method of sorting the data rates per second (e.g., bit rates per second) identified based on the candidate frame rates and candidate frame data sizes in order of size and identifying a frame rate and frame data size that is similar to the maximum value of the data rate per second obtained by considering the available bandwidth.

[0168] For example, in the case where it is determined that 30 Mbps is used for FEC packet data and 70 Mbps is used for image data relative to the entire data having 100 Mbps, the average frame data size needs to be 7 Mb / frame when the frame rate is 10 fps, and the average frame data size can be 3.5 Mb / frame when the frame rate is 20 fps.

[0169] In summary, after determining the amount of bits per second to allocate to image data based on the available bandwidth, any of a variety of ABR techniques can be used.

[0170] In operation S510 , the edge data network 2000 may generate a first user FoV frame by encoding a user FoV image using at least one first encoding parameter.

[0171] In operation S512 , the edge data network 2000 may generate a first additional FoV frame by encoding the additional FoV image using at least one second encoding parameter.

[0172] In operation S514 , the edge data network 2000 may generate a user FoV image packet including a user FoV frame packet and a transmission error control packet, the user FoV frame packet including information about the first user FoV frame.

[0173] In operation S516 , the edge data network 2000 may generate an additional FoV image packet including an additional FoV frame packet including information about the first additional FoV frame and a transmission error control packet.

[0174] In operation S518 , the edge data network 2000 may transmit the user FoV image packet to the electronic device 1000 through the first transmission channel.

[0175] In operation S520 , the edge data network 2000 may transmit the additional FoV image packet to the electronic device 1000 through the second transmission channel.

[0176] Operations S522 to S526 correspond to Figure 3 Operations S320 to S324 are described, and thus, descriptions of operations S522 to S526 are omitted.

[0177] In this case, the first transmission channel and the second transmission channel may be different from each other, and thus, the user FoV image group and the additional FoV image group may be processed in parallel and independently, and the electronic device 1000 may perform decoding based on the respective FoV image groups in parallel and independently.

[0178] Figure 6 is a flowchart illustrating a method of streaming image content obtained from an edge data network, performed by an electronic device according to an embodiment of the present disclosure.

[0179] In operation S600, the electronic device 1000 may obtain directional information. In this case, the directional information may be obtained every first period. For example, the first period may be 5 ms, but is not limited thereto.

[0180] In operation S602, the electronic device 1000 may transmit directional information to the edge data network 2000. In this case, the directional information may be transmitted every first period, but is not limited thereto, and may be transmitted less frequently than every first period. In this case, a period greater than the first period may be identified by considering the frame rate.

[0181] In operation S604, the electronic device 1000 may obtain first network status information such as a packet loss rate, available bandwidth, or a number of consecutive lost packets based on the result of processing the received packets. In this case, the network status information may be obtained and updated every second period. The second period may be 50 ms, but is not limited thereto.

[0182] In operation S606, the electronic device 1000 may transmit the first network status information to the edge data network 2000. In this case, the first network status information may be transmitted every third period, and the third period may be greater than the second period. For example, the third period may be 2 s, but is not limited thereto.

[0183] In operation S608 , the electronic device 1000 may transmit a packet for RTT measurement to the edge data network 2000 .

[0184] In operation S610 , the edge data network 2000 may transmit a response to a packet for RTT measurement.

[0185] In operation S612, the edge data network 2000 may measure the RTT based on the time point at which the packet for RTT measurement is transmitted and the time point at which the response to the packet for RTT measurement is received. That is, the edge data network 2000 may measure the RTT by resetting a timer when transmitting the packet for RTT measurement and obtaining the timer value when the response to the packet for RTT measurement is received from the edge data network 2000. The RTT may correspond to the current latency. That is, the RTT indicates the time period required to transmit and receive a packet between a transmitting device and a receiving device and may increase depending on network conditions (for example, when a packet is dropped at the link layer, it may be retransmitted typically up to 8 to 9 times. More frequent retransmissions increase the average RTT. Furthermore, as the number of packets accumulated in the queues of router nodes between the transmitting and receiving devices increases, the average RTT may increase due to queuing delays). RTTs measured at different time points may differ slightly from one another, and thus, an average of RTTs measured at several time points may be used.

[0186] In operation S614, the electronic device 1000 may transmit RTT measurement information (second network status information) to the edge data network 2000. In this case, a series of processes from transmission of a packet for RTT measurement to transmission of RTT measurement information may be performed every fourth period. In this case, the fourth period may be 5 seconds, but is not limited thereto.

[0187] In operation S618 , the edge data network 2000 may update the state information based on the directional information, the first network state information, and the RTT measurement information (second network state information).

[0188] Meanwhile, after initially establishing a network connection with the electronic device 1000, the edge data network 2000 may receive information about the required latency from the electronic device 1000. The information about the required latency may also be included in the status information and may be sent and updated whenever an application running on the electronic device 1000 changes.

[0189] In operation S620, the edge data network 2000 may identify the first encoding parameter, the second encoding parameter, the first transmission parameter, and the second transmission parameter based on the updated state information. Figure 5 The first encoding parameter and the second encoding parameter are described, and thus, the following describes in detail how the edge data network 2000 identifies the transmission parameter based on the status information.

[0190] For example, the edge data network 2000 may identify the FEC block size based on the following formula 2. FEC block size N block (S block ) may refer to the number of packets in an FEC block, including source packets and FEC packets. Here, an FEC packet may be an example of a transmission error control packet generated for a user FoV image. A source packet is a packet containing actual data and may be a packet containing information about at least a portion of the data generated by encoding the user FoV image (i.e., the first user FoV frame).

[0191] [Formula 2]

[0192] N block (S block ) = (number of source packets + number of FEC packets) = (((floor(1 / R loss ))+1) *floor(N loss )

[0193] Here, R loss can represent the packet loss rate, and N loss It can indicate the number of packets in which packet loss occurs continuously.

[0194] That is, when the packet loss rate is R loss Only when 1 / R loss The lost packet can only be properly recovered if one of the packets is an FEC packet.

[0195] Here, the FEC block size can be identified by considering a case where packet loss occurs continuously, and when the number of packets where packet loss occurs continuously is N loss By considering that N loss The FEC block size is identified based on Equation 2.

[0196] Meanwhile, the edge data network 2000 may identify the maximum number N of source packets included in one FEC block based on Equation 3: source .

[0197] [Formula 3]

[0198] N source = floor(1 / R loss )* floor(N loss )

[0199] The edge data network 2000 can identify the number N of FEC packets included in one FEC block based on Equation 4. fec .

[0200] [Formula 4]

[0201] N fec = floor(N loss )

[0202] The edge data network 2000 can identify the FEC code rate R based on Equation 5 fec .

[0203] [Formula 5]

[0204] R fec = N source / N block

[0205] Here, the FEC block needs to be maintained at T block required and its value can be identified based on the delay D (transport layer) required for end-to-end transmission. For example, by considering a frame interval of 90 fps, T block required Can be identified as 12 ms.

[0206] In this case, the maximum FEC block size S block max (Nblock max ) can be obtained by floor(ABW *T block required The maximum segment size (MSS) is identified. The maximum segment size (MSS) indicates the maximum size of a packet (segment) and can be expressed in bytes. If the currently identified FEC block size is larger than the maximum FEC block size, the currently identified FEC block size can be modified to the maximum FEC block size. In this case, taking into account the FEC code rate, the size of the current FEC block can be further modified to fall within the maximum FEC block size and thus have an integral number of FEC packets and source packets.

[0207] At the same time, all source packets in a frame need to be protected by FEC packets. Therefore, even if the number of packets N inserted into an FEC block among the packets in a frame is frame Less than N source In the case that the source packets can be subjected to and the number of source packets is N source In this case, the number of FEC packets is the same as the number of FEC packets. In this case, the packet of the next frame and the packet of the current frame cannot be the source packets in one FEC block.

[0208] On the other hand, the number of packets N inserted into one FEC block frame Less than N source In this case, the number of inserted FEC packets can be significantly larger than the number of source packets, which may be inefficient for recovering lost packets in each FEC block.

[0209] Therefore, the size of the FEC block may be modified by taking the number of FEC blocks into consideration.

[0210] For example, when the number of packets included in frame x is N frame(x) When , the edge data network 2000 can identify the number N of FEC blocks used to send frame x based on Equation 6 block frame(x) .

[0211] [Formula 6]

[0212] N block frame(x) = ceil(N frame / N source )

[0213] To similarly adjust the number of source packets to be inserted into each FEC block, the modified number N′ of source packets per FEC block can be identified based on Equation 7. source .

[0214] [Formula 7]

[0215] N' source = ceil(N frame / N block frame(x) )

[0216] For example, in the edge data network 2000, the FEC block size N block In the case where the number of source packets of the frame to be transmitted is 42 and the identification is 22 and the FEC scheme of (20, 22) is used, the number of FEC blocks is 3, and the number of source packets of the FEC blocks is 20, 20, and 2, respectively. However, when the above method of modifying the FEC block size for each frame is used, N' source is modified to ceil(42 / 3)=14, and thus, the number of FEC blocks is the same, but the number of source packets of the FEC blocks can be 14, 14, and 14, respectively. In this case, the number of FEC packets in each FEC block is 2, which is the same as before, however, the FEC block size is reduced, and accordingly, the packet recovery capability can be further improved.

[0217] In operation S624, the edge data network 2000 may encode the user FoV image using the first encoding parameter to generate a first user FoV frame. Here, the user FoV image may refer to an image having a user FoV angle identified according to the size of the viewport area of ​​the electronic device 1000 based on the latest orientation information included in the image request sent from the electronic device 1000.

[0218] In operation S626, the edge data network 2000 may encode the additional FoV image using the second encoding parameter to generate a first additional FoV frame. Here, the additional FoV image may be an image having a specific FoV around the user FoV image.

[0219] In operation S628, the edge data network 2000 may generate a user FoV image packet using the first transmission parameters, the user FoV image packet including the user FoV frame packet and a first transmission error control packet including information about the first user FoV frame. For example, if the transmission error control packet is an FEC packet, the transmission parameters may be the FEC block size and FEC code rate described above. The transmission error control packet (i.e., FEC packet) corresponding to the user FoV frame packet (i.e., source packet) is generated using the transmission parameters identified based on the state information.

[0220] In operation S630 , the edge data network 2000 may generate an additional FoV image packet including an additional FoV frame packet including information about the first additional FoV frame and a second transmission error control packet by using the second transmission parameter.

[0221] Figure 7 is a flowchart illustrating a method of streaming image content obtained from an edge data network, performed by an electronic device according to an embodiment of the present disclosure.

[0222] In operation S702, the edge data network 2000 may transmit a user FoV image packet to the electronic device 1000. A description of generating and then transmitting the user FoV image packet to the edge data network 2000 based on the orientation information obtained from the electronic device 1000 is provided above.

[0223] In operation S704 , the edge data network 2000 may transmit an additional FoV image packet to the electronic device 1000 .

[0224] In operation S706, the electronic device 1000 may identify whether a packet is lost in the user FoV frame of the user FoV image group. In this case, to identify whether a packet is lost, the electronic device 1000 may identify whether the packet is received within a preset time period. This is because when a packet is received after the preset time period (for example, when the packet is received after the corresponding image is decoded and reproduced), the packet cannot be used. For example, by identifying whether there is an unreceived packet in the previous FEC block at the time point when the second packet of the current FEC block is received, in order to identify whether a packet is lost in the previous FEC block, the electronic device 1000 can identify whether a packet is lost. That is, sequence numbers exist in all packets, and the range of sequence numbers of the source packets protected by each FEC packet can be identified from the FEC packet. However, because the order in which packets pass through the network may change, instead of identifying packets that have not arrived when the FEC packet is received as lost packets, considering that packets can be received in an order different from the order in which the packets are sent from the sending side, the electronic device 1000 that has received the FEC packet can wait until 2 or 3 packets with sequence numbers larger than the sequence number of the FEC packet arrive, and when there is a lost packet among the packets of the sequence number protected by the FEC packet, the electronic device 1000 can perform packet recovery by using the FEC packet.

[0225] In operation S708, in a case where it is identified that there is a lost packet, the electronic device 1000 may restore the lost packet based on the error control packet.

[0226] In operation S710, if some lost packets have not yet been recovered, the electronic device 1000 may determine whether to request retransmission of the packets that have not yet been recovered based on the RTT and the remaining time until the user FoV image is obtained by decoding (i.e., until the image obtained by decoding is reproduced). For example, if the value obtained by adding a specific margin value (for example, 0.5*RTT, but not limited to this) to the most recently measured RTT is less than the remaining time period (Timer (nextFrame)) until the user FoV image is decoded (i.e., the next frame is reproduced), the electronic device 1000 may determine that retransmission of the packets that have not yet been recovered is to be requested. Here, the timer may be set to count a time period set in consideration of the frame rate of the frame from the time each frame is decoded, and the remaining time period until the timer expires may be Timer (nextFrame).

[0227] In operation S712, when it is identified that retransmission of the packet that has not been recovered is requested, the electronic device 1000 may send a request for retransmission of the packet that has not been recovered to the edge data network 2000. When it is identified that retransmission of the packet that has not been recovered is not requested, the electronic device 1000 may not send a request for retransmission of the packet that has not been recovered to the edge data network 2000.

[0228] In operation S714 , in response to the request for retransmission of the packet that has not been recovered, the edge data network 2000 may retransmit the packet that has not been recovered to the electronic device 1000 .

[0229] In operation S716 , the electronic device 1000 may decode the first user FoV frame based on at least one of the user FoV image packet, the restored packet, and the packet that has not been restored and is thus retransmitted, to reconstruct the first user FoV image.

[0230] In operation S718, the electronic device 1000 may decode the first additional FoV frame based on the additional FoV image packet to reconstruct the first additional FoV image. In this case, the additional FoV image packet may not include any error control packets, and thus lost packets cannot be recovered. However, the electronic device 1000 may request retransmission of the packet. In this case, operations similar to operations S710 to S714 may be performed. The present disclosure is not limited to this, and the additional FoV image packet may include an error control packet. If the additional FoV image packet includes an error control packet, operations similar to operations S706 to S714 may be performed.

[0231] The OrderingWindowSize may be considered when identifying whether a packet is a lost packet. The OrderingWindowSize may indicate the number of packets to wait for to be received in an order different from the order in which the packets were sent.

[0232] Packets sent by the transmitting side can be received in an order different from the order in which they were sent. For example, if the transmitting side has sent packets 1, 2, 3, 4, and 5 in sequence, and the receiving side receives packets 2, 1, 3, 5, and 4 in sequence, the receiving side can wait until as many packets as OrderingWindowSize have been received without considering packet 1 as lost, even though packet 2 was received first. For example, if the receiving side's OrderingWindowSize is W, and the receiving side is waiting for packet X but packet X is not received, the receiving side can wait for packet (X+W-1) to be received, and when a packet with a sequence number greater than (X+W-1) is received, recognize packet X as lost and then wait for packet (X+1) to be received. If the OrderingWindowSize is 0, all packets received in an order different from the order in which they were sent are recognized as lost packets, so it is necessary to appropriately determine the OrderingWindowSize based on the RTT.

[0233] In operation S720, the electronic device 1000 may reproduce a portion of the first partial image including the first user FoV image and the first additional FoV image.

[0234] Figure 8a is a flowchart illustrating a method of streaming image content performed by an edge data network according to an embodiment of the present disclosure.

[0235] In operation S802 , the edge data network 2000 may obtain directional information from the electronic device 1000 connected to the edge data network 2000 .

[0236] In operation S804 , the edge data network 2000 may obtain a first partial image including a user FoV image corresponding to the orientation information and an additional FoV image adjacent to the user FoV image.

[0237] In operation S806, the edge data network 2000 may encode the user FoV image using at least one first encoding parameter to generate a first user FoV frame. The first encoding parameter may be a frame data rate, a frame size, etc., and may be identified based on state information including at least one of network state information and orientation information.

[0238] In operation S808, the edge data network 2000 may encode the additional FoV image using at least one second encoding parameter to generate a first additional FoV frame. The second encoding parameter may be a frame data rate, a frame size, etc., and may be identified based on the first encoding parameter and a ratio of the first encoding parameter to the second encoding parameter. The second encoding parameter may be identified based on state information including at least one of network state information and orientation information.

[0239] In operation S810, the edge data network 2000 may transmit a user FoV image packet including a user FoV frame packet and a first transmission error control packet determined by using a first transmission parameter to the electronic device 1000, wherein the user FoV frame packet includes information about a first user FoV frame.

[0240] In operation S812, the edge data network 2000 may transmit an additional FoV image packet to the electronic device 1000, the additional FoV image packet including an additional FoV frame packet and a second transmission error control packet determined by using a second transmission parameter, the additional FoV frame packet including information about the first additional FoV frame. In this case, the user FoV image packet and the additional FoV image packet may be transmitted through different transmission channels.

[0241] Figure 8b is a flowchart illustrating a method for streaming image content performed by an edge data network according to another embodiment of the present disclosure.

[0242] In operation S822 , the edge data network 2000 may obtain network status information between the edge data network 2000 and the electronic device 1000 connected to the edge data network 2000 .

[0243] In this case, the network status information may include status parameters. The status parameters may include, but are not limited to, throughput for processing tasks (such as data transmission and reception through a network connection), which is determined by considering the situation of the network, the situation of the edge data network 2000, or the situation of the electronic device 1000.

[0244] Throughput can refer to the amount of data per unit time that can be processed in an end-to-end manner (e.g., data-related tasks performed by the edge data network 2000, data transmission through the network, and processing of data-related tasks by the electronic device 1000). For example, throughput can refer to the size or speed per unit time of data sent on the network. Alternatively, throughput can refer to the size of data processed by the edge data network 2000 and the electronic device 1000, or the speed of such data processing. Throughput can be identified based on at least one evaluation index (e.g., the number of packets, bytes, or bits received per unit time).

[0245] For example, the electronic device 1000 may measure the total size of data received per unit time by the receiving module based on the transmission protocol, and transmit throughput information including the measured value to the edge data network 2000. However, the present disclosure is not limited thereto, and it will be understood by those skilled in the art that various modules may measure the throughput of relevant data and transmit information about the measured throughput to the edge data network 2000.

[0246] In operation S824 , the edge data network 2000 may obtain directional information from the electronic device 1000 connected to the edge data network 2000 .

[0247] In operation S826 , the edge data network 2000 may determine the sizes and positions of the user FoV image and additional FoV images adjacent to the user FoV image based on the obtained network state information and orientation information.

[0248] The edge data network 2000 can identify the user's FoV image based on the orientation information. For example, the edge data network 2000 can identify the location of the user's FoV image (e.g., the coordinate location of the center of the image) based on the received orientation information. The edge data network 2000 can receive the capability information of the electronic device 1000 from the electronic device 1000, identify the size of the viewport area of ​​the electronic device 1000 based on the capability information, and identify the identified size of the viewport area as the size of the user's FoV image.

[0249] The edge data network 2000 can identify the size and location of additional FoV images adjacent to the user FoV image based on network status information. If the value of at least one status parameter included in the network status information is less than or equal to a first threshold, the edge data network 2000 can identify the size and location of additional FoV images adjacent to the user FoV image corresponding to the at least one status parameter less than or equal to the first threshold. Furthermore, if the value of at least one status parameter included in the network status information is greater than or equal to a second threshold, the edge data network 2000 can identify the size and location of additional FoV images adjacent to the user FoV image corresponding to the at least one status parameter greater than or equal to the second threshold. In this case, the second threshold may be a value set based on the assumption that at least one status parameter prior to the current time point is less than or equal to the second threshold. The first threshold may be equal to the second threshold, but is not limited thereto. In this case, the size of the additional FoV image corresponding to the at least one status parameter less than or equal to the first threshold may be larger than the size of the additional FoV image corresponding to the at least one status parameter greater than the second threshold.

[0250] In operation S828, the edge data network 2000 may obtain a first partial image including a user FoV image and an additional FoV image adjacent to the user FoV image, wherein the first partial image is determined based on sizes and positions of the user FoV image and the additional FoV images adjacent to the user FoV image.

[0251] In operation S830, the edge data network 2000 may encode the user FoV image using at least one first encoding parameter to generate a first user FoV frame. The first encoding parameter may be a frame data rate, a frame size, etc., and may be identified based on state information including at least one of network state information and orientation information.

[0252] In operation S832, the edge data network 2000 may encode the additional FoV image using at least one second encoding parameter to generate a first additional FoV frame. The second encoding parameter may be a frame data rate, a frame size, etc., and may be identified based on the first encoding parameter and a ratio of the first encoding parameter to the second encoding parameter. The second encoding parameter may be identified based on state information including at least one of network state information and orientation information.

[0253] In operation S834 , the edge data network 2000 may transmit a user FoV image packet including a user FoV frame packet further including information about the first user FoV frame and a first transmission error control packet to the electronic device 1000 .

[0254] In operation S836 , the edge data network 2000 may transmit an additional FoV image packet including the additional FoV frame packet and the second transmission error control packet to the electronic device 1000 , the additional FoV image packet including information about the first additional FoV frame.

[0255] Figure 9 is a flowchart illustrating a method of streaming image content performed by an edge data network according to an embodiment of the present disclosure.

[0256] In operation S902 , the edge data network 2000 may obtain directional information from the electronic device 1000 .

[0257] In operation S904 , the edge data network 2000 may obtain a first partial image including a user FoV image corresponding to the orientation information and an additional FoV image adjacent to the user FoV image.

[0258] In operation S906, the edge data network 2000 may obtain network status information from the electronic device 1000. In this case, the network status information may be periodically obtained from the electronic device 1000, but is not limited thereto.

[0259] In operation S908 , the edge data network 2000 may identify the first encoding parameter, the second encoding parameter, the first transmission parameter, and the second transmission parameter based on state information including at least one of directional information and network state information.

[0260] In operation S910 , the edge data network 2000 may encode a user FoV image by using a first encoding parameter to generate a first user FoV frame.

[0261] In operation S912 , the edge data network 2000 may generate a first additional FoV frame by encoding the additional FoV image using the second encoding parameter.

[0262] In operation S914 , the edge data network 2000 may generate a user FoV image packet including a user FoV frame packet including information about a first user FoV frame and a first transmission error control packet by using the first transmission parameter.

[0263] In operation S916 , the edge data network 2000 may generate an additional FoV image packet including an additional FoV frame packet including information about the first additional FoV frame and a second transmission error control packet by using the second transmission parameter.

[0264] In operation S918 , the edge data network 2000 may transmit the user FoV image packet to the electronic device 1000 .

[0265] In operation S920 , the edge data network 2000 may transmit an additional FoV image packet to the electronic device 1000 .

[0266] Figure 10 is a flowchart illustrating a method of streaming image content performed by an edge data network according to an embodiment of the present disclosure.

[0267] In operation S1002 , the edge data network 2000 may obtain directional information from the electronic device 1000 connected to the edge data network 2000 .

[0268] In operation S1004 , the edge data network 2000 may obtain a first partial image including a user FoV image corresponding to the orientation information and an additional FoV image adjacent to the user FoV image.

[0269] In operation S1006 , the edge data network 2000 may encode the user FoV image by using at least one first encoding parameter to generate a first user FoV frame.

[0270] In operation S1008 , the edge data network 2000 may generate a first additional FoV frame by encoding the additional FoV image using at least one second encoding parameter.

[0271] In operation S1010 , the edge data network 2000 may transmit a user FoV image packet including a user FoV frame packet and a first transmission error control packet to the electronic device 1000 , the user FoV frame packet including information about a first user FoV frame.

[0272] In operation S1012 , the edge data network 2000 may transmit an additional FoV image packet including an additional FoV frame packet including information about the first additional FoV frame and a second transmission error control packet to the electronic device 1000 .

[0273] In operation S1014, the edge data network 2000 may receive a request for retransmission of packets that have not been recovered based on the transmission error control packet from the electronic device 1000. That is, the electronic device 1000 may attempt to recover lost packets based on the transmission error control packet received from the edge data network 2000, and, if the lost packets have not been recovered, the electronic device 1000 may selectively send a request for retransmission of the packets that have not been recovered to the edge data network 2000.

[0274] In operation S1016 , in response to the request to retransmit the packet that has not been recovered, the edge data network 2000 may retransmit the packet that has not been recovered to the electronic device 1000 .

[0275] Figure 11 is a flowchart illustrating a method of streaming image content obtained from an edge data network, performed by an electronic device according to an embodiment of the present disclosure.

[0276] In operation S1102 , the electronic device 1000 may transmit directional information to the edge data network 2000 .

[0277] In operation S1104, the electronic device 1000 may obtain a user FoV image packet from the edge data network 2000. The user FoV image packet includes a user FoV frame packet and a first transmission error control packet. The user FoV frame packet includes information about a first user FoV frame obtained by encoding a user FoV image corresponding to the orientation information. In this case, the user FoV image packet may be obtained from the edge data network 2000 via the first transmission channel.

[0278] In operation S1106, the electronic device 1000 may obtain an additional FoV image packet from the edge data network 2000. The additional FoV image packet includes an additional FoV frame packet and a second transmission error control packet. The additional FoV frame packet includes information about a first additional FoV frame obtained by encoding an additional FoV image adjacent to the user FoV image. In this case, the additional FoV image packet may be obtained from the edge data network 2000 via the second transmission channel.

[0279] In operation S1108 , based on the user FoV image grouping, the electronic device 1000 may decode the first user FoV frame to reconstruct the first user FoV image.

[0280] In operation S1110 , based on the additional FoV image group, the electronic device 1000 may decode the first additional FoV frame to reconstruct a first additional FoV image.

[0281] In operation S1112, the electronic device 1000 may reproduce at least a portion of the first partial image including the first user FoV image and the first additional FoV image. That is, the electronic device 1000 may obtain orientation information at the time of reproduction and reproduce the portion of the first partial image based on the orientation information.

[0282] Figure 12a and Figure 12bis a flowchart illustrating a method of streaming image content obtained from an edge data network, performed by an electronic device according to an embodiment of the present disclosure.

[0283] In operation S1202 , the electronic device 1000 may transmit directional information to the edge data network 2000 .

[0284] In operation S1204, the electronic device 1000 may obtain a user FoV image packet from the edge data network 2000, wherein the user FoV image packet includes a user FoV frame packet and a transmission error control packet, wherein the user FoV frame packet includes information about a first user FoV frame obtained by encoding a user FoV image corresponding to orientation information.

[0285] In operation S1206 , the electronic device 1000 may identify whether there is a lost packet among the user FoV frame packets.

[0286] In a case where it is identified in operation S1206 that there is no lost packet among the user FoV frame groups, in operation S1208, the electronic device 1000 may decode the first user FoV frame based on the user FoV frame groups to reconstruct a user FoV image.

[0287] If it is determined in operation S1206 that there is a lost packet among the user FoV frame packets, in operation S1210, the electronic device 1000 may recover the lost packet based on the transmission error control packet. In this case, the transmission error control packet may be a packet with an error recovery function.

[0288] In operation S1212, the electronic device 1000 may identify whether some lost packets have not been recovered.

[0289] In the case that it is identified in operation S1212 that no lost packets have not been recovered, in operation S1214, the electronic device 1000 may decode the first user FoV frame based on the non-lost packets and the recovered packets among the user FoV frame packets to reconstruct the user FoV image.

[0290] In operation S1216, the electronic device 1000 may determine whether the remaining time period until the decoding of the user FoV image (i.e., until the reproduction of the next frame) is less than the RTT. In this case, the electronic device 1000 may determine whether the remaining time period is less than a value obtained by adding a specific margin value to the RTT.

[0291] If it is determined in operation S1216 that the remaining time period is less than the RTT, the electronic device 1000 may decode the first user FoV frame based on the unlost packets and some recovered packets among the user FoV frame packets to reconstruct the user FoV image in operation S1218. In this case, the image quality of the reconstructed user FoV image may be degraded due to some lost packets.

[0292] In a case where it is identified in operation S1216 that the remaining time period is not less than the RTT, in operation S1220 , the electronic device 1000 may transmit a request for retransmission of packets that have not been recovered to the edge data network 2000 .

[0293] In operation S1222, the electronic device 1000 may re-receive a packet that has not been restored and is thus retransmitted in response to a request for retransmission of the packet that has not been restored.

[0294] In operation S1224, the electronic device 1000 can decode the first user FoV frame based on the unlost packets, some recovered packets, and some re-received packets that have not been recovered and are therefore retransmitted among the user FoV frame packets to reconstruct the user FoV image. At the same time, if the packets that have not been recovered are not received until the decoding of the user FoV frame is completed, the user FoV image can be reconstructed by using only the other packets.

[0295] Figure 12c is a flowchart illustrating a method of streaming image content obtained from an edge data network, performed by an electronic device according to an embodiment of the present disclosure.

[0296] In operation S1226, the electronic device 1000 may identify whether a time period remaining until decoding of the user FoV image is less than the RTT.

[0297] In case it is identified in operation S1226 that the remaining time period is less than the RTT, in operation S1228, the electronic device 1000 may decode the first user FoV frame based on the non-lost packets and some recovered packets among the user FoV frame packets to reconstruct the first user FoV image.

[0298] In the case where it is identified in operation S1226 that the remaining time period is not less than the RTT, in operation S1230, the electronic device 1000 may identify whether the area associated with the packet that has not yet been restored is included in the second user FoV area corresponding to the latest directional information. The header of the packet may include information about the position of the coding block associated with the packet, and, in the case where the positions of the blocks of the packets other than the packet that has not yet been restored are identified and the positions of some blank blocks are identified, the positions of the blocks associated with the packet that has not yet been restored can be identified, and based on these positions, the area associated with the packet that has not yet been restored can be identified. That is, after identifying that a packet corresponding to a frame has been received, the electronic device 1000 may identify which blocks in the frame are associated with the packet that has not yet been restored based on the information about the position of the coding block included in the header of the received packet.

[0299] In the case where it is identified in operation S1230 that the area associated with the packet that has not yet been restored is not included in the second user FoV area, in operation S1232, the electronic device 1000 can decode the first user FoV frame based on the non-lost packets and some restored packets among the user FoV frame packets to reconstruct the first user FoV image.

[0300] In a case where it is identified in operation S1230 that the area related to the packet that has not been restored is included in the second user FoV area, in operation S1234 , the electronic device 1000 may send a request for retransmission of the packet that has not been restored to the edge data network 2000 .

[0301] In operation S1236 , the electronic device 1000 may re-receive, from the edge data network 2000 , a packet that has not been restored and is thus retransmitted in response to a request for retransmission of the packet that has not been restored.

[0302] In operation S1238, the electronic device 1000 may decode the first user FoV frame based on non-lost packets, some recovered packets, and some re-received packets that have not been recovered and thus retransmitted among the user FoV frame packets to reconstruct the first user FoV image.

[0303] Meanwhile, although the above description has been made on the premise that the recognition in operation S1226 has a higher priority than that in operation S1230 , the present disclosure is not limited thereto, and those skilled in the art may understand that the priority may be changed.

[0304] At the same time, refer to Figures 12a to 12cIt is described above that the electronic device 1000 recovers some user FoV frame packets or requests retransmission of some packets based on the transmission error control packet included in the user FoV image packet, and the present disclosure is not limited to this. It can be understood that the electronic device 1000 can recover some additional FoV frame packets or request retransmission of some packets based on the transmission error control packet included in the additional FoV image packet in a similar manner.

[0305] Figure 13 is a flowchart illustrating a method of streaming image content obtained from an edge data network, performed by an electronic device according to an embodiment of the present disclosure.

[0306] In operation S1302 , the electronic device 1000 may transmit directional information to the edge data network 2000 .

[0307] In operation S1304 , the electronic device 1000 may transmit network status information to the edge data network 2000 .

[0308] In operation S1306, the electronic device 1000 may obtain a user FoV image group from the edge data network 2000, wherein the user FoV image group includes a user FoV frame group and a first transmission error control packet, wherein the user FoV frame group includes information about a first user FoV frame obtained by encoding a user FoV image corresponding to the orientation information by using at least one first encoding parameter identified based on status information including at least one of the orientation information and the network status information, and the first transmission error control packet is generated by using the first transmission parameter identified based on the status information.

[0309] In operation S1308, the electronic device 1000 may obtain an additional FoV image group from the edge data network 2000, wherein the additional FoV image group includes an additional FoV frame group and a second transmission error control packet, wherein the additional FoV frame group includes information about a first additional FoV frame obtained by encoding an additional FoV image adjacent to the user FoV image by using at least one second encoding parameter identified based on the state information, and the second transmission error control packet is generated by using the second transmission parameter identified based on the state information.

[0310] In operation S1310, based on the user FoV image grouping, the electronic device 1000 may decode a first user FoV frame to reconstruct a first user FoV image.

[0311] In operation S1312, based on the additional FoV image group, the electronic device 1000 may decode the first additional FoV frame to reconstruct a first additional FoV image.

[0312] In operation S1314, the electronic device 1000 may reproduce at least a portion of the first partial image including the first user FoV image and the first additional FoV image.

[0313] Figure 14 is a diagram illustrating a process of transmitting a first partial image in several data units to an electronic device 1000 , performed by the edge data network 2000 , according to an embodiment of the present disclosure.

[0314] The edge data network 2000 can divide the VR image 1500 in the vertical direction (e.g., longitude) to obtain several data units, select one or more data units ①, ②, and ③ as many as the required FoV from the several data units in the process of obtaining the first partial image, encode the selected one or more data units, and send the encoded results to the electronic device 1000.

[0315] Figure 15 is a diagram schematically illustrating the operation process between electronic devices, edge data networks, and VR game interface devices.

[0316] Apart from Figure 4 In addition to the components, Figure 15 Also shown is a VR game interface device 4000. In the field of VR games, unlike the field of VR streaming, user interaction is involved, and accordingly, the VR game interface device 4000 can be used in addition to the electronic device 1000.

[0317] Reference Figure 4 The operation process between electronic devices and edge data networks in the VR streaming field is described above, so redundant descriptions are omitted, and the operation process between electronic devices, edge data networks and VR game interface devices in the VR game field is described below.

[0318] The edge data network 2000 can generate VR game images (interactive VR images) included in the VR sequence based on instructions of the VR game processing module 1588.

[0319] Specifically, the VR game processing module 1588 may store instructions for generating a first partial image of a VR game image based on the orientation information, the rotation speed information, and the VR game interface device sensing information 1552. The VR game processing module 1588 may store instructions for outputting the VR game interface device control information 1554.

[0320] The electronic device 1000 may send the VR game interface device sensing information 1552 to the edge data network 2000 for VR game processing. In this case, the electronic device 1000 may receive the VR game interface device sensing information from the VR game interface device 4000.

[0321] The VR game interface device 4000 may include a network interface 4020, a sensing module 4010, and an actuation module 4030. The network interface 4020 may be a module for communicating with the outside, and the sensing module 4010 may be a module for sensing the user's movements. The sensing module 4010 may obtain VR game interface device sensing information.

[0322] The actuation module 4030 may be a module that provides various types of outputs (eg, tactile feedback) such as vibration to the user. The actuation module 4030 may provide various types of outputs to the user based on the VR game interface device sensing control information.

[0323] The VR game interface device 4000 may transmit VR game interface device sensing information to the electronic device 1000. In addition, the VR game interface device 4000 may receive VR game interface device control information from the electronic device 1000.

[0324] It is described above that VR interface device sensing information can be transmitted to the edge data network 2000 via the electronic device 1000, and VR interface device sensing information and VR game interface device control information can be transmitted to the VR game interface device 4000 via the electronic device 1000, but the present disclosure is not limited thereto, and they can be directly exchanged between the VR game interface device 4000 and the edge data network 2000. In this case, synchronization information for synchronization with the electronic device 1000 can also be transmitted.

[0325] Although about Figure 15 The operating process between electronic devices, edge data networks and VR game interface devices in the VR gaming field is described above, but the present disclosure is not limited to this, and those skilled in the art will understand that interactive VR can be mainly utilized in many fields where users can directly intervene in content and perform activities such as simulation, training, etc.

[0326] Figure 16 is a block diagram of an electronic device according to an embodiment of the present disclosure.

[0327] Connect to Figures 1 to 15 The electronic devices of the edge data network may correspond to Figure 16For example, in network environment 100, electronic device 110 may communicate with electronic device 1702 (here, the electronic device includes a VR gaming interface device) via a first network 1798 (e.g., a short-range wireless communication network), or may communicate with electronic device 1704 or server 1708 via a second network 1799 (e.g., a long-range wireless communication network). Depending on the embodiment, electronic device 110 may communicate with electronic device 1704 via server 1708. Depending on the embodiment, electronic device 110 may include a processor 1720, a memory 1730, an input device 1750, an acoustic output device 1755, a display device 1760, an audio module 1770, a sensor module 1776, an interface 1777, a haptic module 1779, a camera module 1780, a power management module 1788, a battery 1789, a communication module 1790, a subscriber identification module 1796, or an antenna module 1797. In some embodiments, at least one of the components (e.g., display device 1760 or camera module 1780) may be omitted from electronic device 110, or one or more other components may be further included. In some embodiments of the present disclosure, some components may be implemented as integrated circuits. For example, sensor module 1776 (e.g., a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented by being embedded in display device 1760 (e.g., a display).

[0328] The processor 1720 may execute, for example, software (e.g., program 1740) to control at least one other component (e.g., hardware or software component) of the electronic device 110 connected to the processor 1720, and perform various types of data processing or calculations. Depending on an embodiment, as at least part of the data processing or calculation, the processor 1720 may load commands or data received from another component (e.g., sensor module 1776 or communication module 1790) into the volatile memory 1732, process the commands or data stored in the volatile memory 1732, and store the resulting data in the non-volatile memory 1734.

[0329] According to an embodiment, processor 1720 may execute an application client. In response to the execution of the application client, processor 1720 may send a request message to edge data network 2000 to check the network protocol to be used by the application client. Furthermore, processor 1720 may receive response information from edge data network 2000 indicating the network protocol to be used by the application client. Based on the response message, processor 1720 may update the network protocol to be used by the UE application. Processor 1720 may select a network socket corresponding to the updated network protocol. Processor 1720 may receive data generated for the application client from edge data network 2000 using the selected network socket.

[0330] Depending on the embodiment, the processor 1720 may include a main processor 1721 (e.g., a CPU or an application processor (AP)) and an auxiliary processor 1723 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that can operate independently of the main processor 1721 or together with the main processor 1721. Additionally or alternatively, the auxiliary processor 1723 may be configured to use less power than the main processor 1721 or to be dedicated to a specified function. The auxiliary processor 1723 may be implemented separately from the main processor 1721 or may be implemented as part of the main processor 1721.

[0331] The auxiliary processor 1723 can control at least some functions or states related to at least one of the components in the electronic device 110 (for example, the display device 1760, the sensor module 1776, or the communication module 1790) in place of the main processor 1721 when the main processor 1721 is in an inactive state (for example, a sleep state), or together with the main processor 1721 when the main processor 1721 is in an active state (for example, executing an application). Depending on the embodiment, the auxiliary processor 1723 (for example, the ISP or the CP) can be implemented as part of another component (for example, the camera module 1780 or the communication module 1790) that is functionally associated with the auxiliary processor 1723.

[0332] The memory 1730 may store various pieces of data to be used by at least one component in the electronic device 110 (e.g., the processor 1720 or the sensor module 1776). The data may include, for example, software (e.g., the program 1740) and input data or output data related to commands associated with the software. The memory 1730 may include a volatile memory 1732 and a non-volatile memory 1734.

[0333] The program 1740 may be stored as software in the memory 1730 and may include, for example, an operating system (OS) 1742, middleware 1744, and applications 1746. In an embodiment, the program 1740 may include Figure 1 The first application client 122 and the second application client 124. In addition, the program 1740 may include Figure 1 edge enabler client 130 .

[0334] The input device 1750 may receive a command or data intended for a component (eg, the processor 1720 ) in the electronic device 110 from outside the electronic device 110 .

[0335] The acoustic output device 1755 may output an acoustic signal to the outside of the electronic device 110. The acoustic output device 1755 may include, for example, a speaker. The speaker may be used for general purposes such as reproducing multimedia or recording.

[0336] The display device 1760 can visually provide information to an external portion of the electronic device 110 (e.g., a user). The display device 1760 may include, for example, a display, a hologram device, or a projector, and a control circuit system configured to control the corresponding device. Depending on the embodiment, the display device 1760 may include a touch control circuit system configured to detect a touch, or a sensor circuit system configured to measure the strength of the force generated by the touch (e.g., a pressure sensor).

[0337] The audio module 1770 may convert sound into an electrical signal or vice versa. Depending on the embodiment, the audio module 1770 may obtain sound through the input device 1750, or output sound through the acoustic output device 1755 or an external electronic device (e.g., electronic device 1702 (e.g., a speaker or a headset)) directly or wirelessly connected to the electronic device 110.

[0338] The sensor module 1776 can detect the operating state (e.g., power or temperature) or the external environment state (e.g., user state) of the electronic device 110 and generate an electrical signal or data value corresponding to the detected state. Depending on the embodiment, the sensor module 1776 may include, for example, a (image) gesture sensor, a gyroscope sensor, a barometer sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor (e.g., an RGB (red-green-blue) sensor), an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, an illumination sensor, or a sensor related to an autonomous vehicle (e.g., an inertial measurement unit (IMU), a global positioning system (GPS) sensor, a camera, a light imaging detection and ranging (LIDAR) sensor, and a radio detection and ranging (RADAR) sensor).

[0339] The interface 1777 may support one or more designated protocols that may be used for an external electronic device (eg, the electronic device 1702 ) to be directly or wirelessly connected to the electronic device 110 .

[0340] The connection terminal 1778 may include a connector through which the electronic device 110 may be physically connected to an external electronic device (eg, the electronic device 1702 ).

[0341] The haptic module 1779 may convert the electric signal into mechanical stimulation (eg, vibration or motion) or electric stimulation that the user can sense through tactile or motion sense.

[0342] The camera module 1780 may capture still images or moving images. Depending on the embodiment, the camera module 1780 may include one or more lenses, image sensors, ISPs, or flashes.

[0343] The power management module 1788 may manage power to be supplied to the electronic device 110 .

[0344] The battery 1789 may supply power to at least one component in the electronic device 110 .

[0345] The communication module 1790 can establish a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 110 and an external electronic device (e.g., electronic device 1702, electronic device 1704, or server 1708), and support communication through the established communication channel. The communication module 1790 may include one or more communication processors that operate independently of the processor 1720 (e.g., AP) and support direct (e.g., wired) communication or wireless communication. Depending on the embodiment, the communication module 1790 may include a wireless communication module 1792 (e.g., a cellular communication module, a short-range communication module, or a global navigation satellite system (GNSS) communication module) and a wired communication module 1794 (e.g., a local area network (LAN) communication module or a power line communication module). A corresponding communication module among these communication modules can communicate with an external electronic device over a first network 1798 (e.g., a short-range communication network such as Bluetooth, Wi-Fi Direct, or Infrared Data Association (IrDA)) or a second network 1799 (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These types of communication modules may be integrated into one component (e.g., a single chip) or implemented by multiple separate components (e.g., multiple chips).

[0346] The wireless communication module 1792 may verify or authenticate the electronic device 110 in a communication network such as the first network 1798 or the second network 1799 by using subscriber information (eg, International Mobile Subscriber Identity (IMSI)) stored in the subscriber identification module 1796 .

[0347] The antenna module 1797 may transmit or receive a signal or power to or from the outside (eg, an external electronic device).

[0348] At least some components may be connected to each other according to a peripheral device communication scheme such as bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI) and exchange signals (eg, commands or data) with each other.

[0349] According to an embodiment, a command or data may be transmitted or received between the electronic device 110 and the external electronic device 1704 via the server 1708 connected on the second network 1799. Each of the electronic devices 1702 and 1704 may be of the same type as or different from that of the electronic device 110.

[0350] Depending on the embodiment, all or some operations performed by electronic device 110 may be performed by one or more external devices among electronic devices 1702 and 1704 and server 1708. For example, when electronic device 110 is to perform a specific function or service automatically or in response to a request from a user or another device, electronic device 110 may also request one or more of external electronic devices 1702 and 1704 to perform at least a portion of the function or service as a supplement or alternative to autonomously performing the function or service. One or more of external electronic devices 1702 and 1704 that have received the request may perform at least a portion of the requested function or service or additional functions or services related to the request, and deliver the results of the execution to electronic device 110. Electronic device 110 may process the results as is or in addition, and provide the processed results as at least a portion of the response to the request. For this purpose, for example, cloud computing, distributed computing, or client-server computing may be used.

Claims

1. A method for transmitting image content performed by an edge data network, the method comprising: obtaining directional information from an electronic device connected to the edge data network; obtaining a first partial image, wherein the first partial image includes a user field of view image corresponding to the orientation information and an additional field of view image adjacent to the user field of view image; Generate a first user view frame by encoding the user view image using at least one first encoding parameter; generating a first additional field of view frame by encoding the additional field of view image using at least one second encoding parameter; Sending a user view image packet to the electronic device, where the user view image packet includes a user view frame packet and a first transmission error control packet, and the user view frame packet includes information about the first user view frame; Sending an additional view image packet to the electronic device, where the additional view image packet includes an additional view frame packet and a second transmission error control packet, and the additional view frame packet includes information about the first additional view frame; receiving a request from the electronic device to retransmit a packet that has not yet been recovered, wherein the request is sent based on a result of identifying that at least a portion of an area associated with the packet that has not yet been recovered is included in a latest user field of view area based on latest orientation information of the electronic device and when it is predicted that the packet that has not yet been recovered can be received before decoding of the first user field of view frame is completed; as well as In response to the request to retransmit the packet that has not been restored, the packet that has not been restored is retransmitted to the electronic device.

2. The method according to claim 1, wherein The user field of view image includes an image having a user field of view relative to a position indicated by the orientation information in a virtual reality (VR) image, the user field of view being identified based on information about a viewport area of ​​a display of the electronic device, and The additional field-of-view image is an image having a specific additional field of view in the upper direction, the lower direction, the left direction, and the right direction of the user field-of-view image.

3. The method according to claim 1, wherein The first partial image includes a partial image having a specific frame index of a VR sequence including a plurality of frames in the first VR image, and the first partial image is associated with the orientation information indicating a position of the partial image.

4. The method according to claim 1, wherein The user view image group is transmitted to the electronic device via a first transmission channel, and the additional view image group is transmitted to the electronic device via a second transmission channel, and The first transmission error control packet and the second transmission error control packet are forward error correction (FEC) packets.

5. The method according to claim 4, wherein The first encoding parameter and the second encoding parameter are at least one of a bit rate, a frame rate, and a resolution, the bit rate, the frame rate, and the resolution being identified based on at least one of a required delay, an available bandwidth, a number of consecutive lost packets, a packet loss rate, and a round trip time (RTT), and The transmission parameters including at least one of an FEC code rate and an FEC block size associated with the FEC packet include at least one of the FEC code rate and the FEC block size identified based on at least one of a required delay, available bandwidth, a number of consecutive lost packets, a packet loss rate, and an RTT.

6. The method according to claim 4, wherein: Sending the user field of view image group to the electronic device includes: identifying, based on a packet loss rate, a number of consecutive lost packets, available bandwidth, and a required time period based on a frame rate and a required latency, source packets including at least some of the user field of view frame packets, a size of an FEC block including the FEC packets, and an FEC code rate indicating a ratio of the source packets to the FEC block; Identifying the number of FEC blocks based on the number of the user field of view frame packets and the number of the source packets; modifying the number of source packets included in each FEC block based on the number of FEC blocks; and The FEC blocks are obtained based on the modified number of source packets included in each FEC block.

7. The method according to claim 6, wherein: The frame rate of the user view frame and the bit rate of each frame of the user view frame are based on the FEC code rate, R fps 、R fbr and the available bandwidth of the electronic device, Among them, R fps represents the frame rate of the additional field of view frame compared to the user field of view frame, and R fbr represents the bit rate per frame of the additional field of view frame compared with the user field of view frame, The frame rate of the additional field of view frame is based on the frame rate of the user field of view area and R fps and identified, and The bit rate per frame of the user view frame is based on the bit rate per frame of the additional view frame and R fbr And identified.

8. The method according to claim 1, wherein receiving state information from the electronic device every first period when the edge data network is connected to the electronic device, the state information comprising at least one of the following: information about a viewport area of ​​a display of the electronic device, information about a frame rate of additional field of view frames compared to user field of view frames, information about a bit rate per frame of the additional field of view frames compared to the user field of view frames, and information about a required latency, receiving channel state information from the electronic device every second period, the channel state information indicating at least one of a number of consecutive lost packets, a packet loss rate, and an available bandwidth, and The periodically measured RTT information is received from the electronic device every third period.

9. The method according to claim 1, wherein The first partial image includes data units of a preset field of view range, Generating the first user field of view frame includes: Obtaining a first data unit of at least one preset field of view range corresponding to the user field of view image in the first partial image; and The first user field of view frame is generated by encoding the first data unit of the at least one preset field of view range, and Generating the first additional field of view frame includes: Obtaining a second data unit of at least one preset field of view range corresponding to the additional field of view image in the first partial image; and The first additional field of view frame is generated by encoding the second data unit of the at least one preset field of view range.

10. An edge data network for transmitting image content to an electronic device, the edge data network comprising: Network interface; a memory storing one or more instructions; at least one processor configured to execute the one or more instructions to Obtaining orientation information from an electronic device connected to the edge data network, obtaining a first partial image, wherein the first partial image includes a user field of view image corresponding to the orientation information and an additional field of view image adjacent to the user field of view image, encoding the user field of view image by using at least one first encoding parameter to generate a first user field of view frame, and encoding the additional field of view image by using at least one second encoding parameter to generate a first additional field of view frame, sending a user view image packet to the electronic device, the user view image packet including a user view frame packet and a first transmission error control packet, the user view frame packet including information about the first user view frame, sending an additional field of view image packet to the electronic device, the additional field of view image packet including an additional field of view frame packet and a second transmission error control packet, the additional field of view frame packet including information about the first additional field of view frame, receiving a request for retransmission of a packet that has not yet been recovered from the electronic device, wherein the request is sent based on a result of identifying that at least a portion of an area associated with the packet that has not yet been recovered is included in a latest user field of view area based on latest orientation information of the electronic device and when it is predicted that the packet that has not yet been recovered can be received before decoding of the first user field of view frame is completed; and In response to the request to retransmit the packet that has not been restored, the packet that has not been restored is retransmitted to the electronic device.

11. The edge data network according to claim 10, wherein: The user view image group is transmitted to the electronic device via a first transmission channel, and the additional view image group is transmitted to the electronic device via a second transmission channel, and The first transmission error control packet and the second transmission error control packet are forward error correction (FEC) packets.

12. The edge data network according to claim 11, wherein: The first encoding parameter and the second encoding parameter are at least one of a bit rate, a frame rate, and a resolution, the bit rate, the frame rate, and the resolution being identified based on at least one of a required delay, an available bandwidth, a number of consecutive lost packets, a packet loss rate, and a round trip time (RTT), and The transmission parameters including at least one of an FEC code rate and an FEC block size associated with the FEC packet include at least one of the FEC code rate and the FEC block size identified based on at least one of a required delay, available bandwidth, a number of consecutive lost packets, a packet loss rate, and an RTT.

Images