Method and device for sending video content using edge computing services

Abstract

An example method for sending video content performed by an edge data network includes: obtaining first orientation information from an electronic device connected to the edge data network; determining second predicted orientation information based on the first orientation information; determining a second predicted local image corresponding to the second predicted orientation information; sending a second predicted frame generated by encoding the second predicted local image to the electronic device; obtaining second orientation information corresponding to the second local image from the electronic device; comparing the second predicted orientation information with the obtained second orientation information; based on the result of the comparison, generating a compensation frame using at least two of the first local image corresponding to the first orientation information, the second predicted local image, or the second local image corresponding to the second orientation information; and sending the compensation frame generated based on the result of the comparison to the electronic device.

Description

Technical Field

[0001] The present disclosure relates to methods and apparatus for transmitting video content using edge computing services (eg, multi-access edge computing (MEC) services). Background Art

[0002] Recently, edge computing technology that uses edge servers to send data has been discussed. Edge computing technology may include, for example, multi-access edge computing (MEC) or fog computing (FOC). Edge computing technology is a technology that provides data to electronic devices through a separate server (hereinafter referred to as an "edge data network" or "MEC server"), which is set at a location geographically close to the electronic device, such as inside or around a base station. For example, an application requiring low latency in at least one application installed in an electronic device can send and receive data via an edge server (edge ​​data network or MEC server) set at a geographically close location without passing through a server located in an external data network (DN) (e.g., the Internet).

[0003] Recently, services using edge computing technology (hereinafter referred to as "MEC-based services" or "MEC services") have been discussed, and research and development of electronic devices that support MEC-based services have been ongoing. For example, an application of an electronic device can send and receive edge computing-based data to and from an edge server (or an application of an edge server) in an application layer.

[0004] As research and development to support MEC-based services progresses, technology for providing high-resolution video content to electronic devices using MEC has been discussed. Summary of the invention

[0005] Technical Solution

[0006] Embodiments of the present disclosure provide a method and device for transmitting video content using edge computing services.

[0007] For example, embodiments of the present disclosure provide a method for transmitting video content to an electronic device performed by an edge data network and the edge data network for providing video content to the electronic device.

[0008] Embodiments of the present disclosure provide a method for receiving video content from an edge data network performed by an electronic device and an electronic device for receiving video content from the edge data network.

[0009] Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description.

[0010] According to an example embodiment of the present disclosure, a method for sending video content performed by an edge data network includes: obtaining first orientation information from an electronic device connected to the edge data network; determining second predicted orientation information based on the first orientation information; determining a second predicted local image corresponding to the second predicted orientation information; sending a second predicted frame generated by encoding the second predicted local image to the electronic device; obtaining second orientation information corresponding to the second local image from the electronic device; comparing the second predicted orientation information with the obtained second orientation information; based on a result of the comparison, generating a compensation frame using at least two of the first local image corresponding to the first orientation information, the second predicted local image, or the second local image corresponding to the second orientation information; and sending the compensation frame generated based on the result of the comparison to the electronic device.

[0011] According to an example embodiment of the present disclosure, an edge data network for sending video content to an electronic device includes: a network interface; a memory storing one or more instructions; and a processor configured to execute the one or more instructions, wherein the processor is further configured to execute the one or more instructions to: obtain first orientation information from the electronic device connected to the edge data network; determine second predicted orientation information based on the first orientation information; determine a second predicted local image corresponding to the second predicted orientation information; send a second predicted frame generated by encoding the second predicted local image to the electronic device; obtain second orientation information corresponding to the second local image from the electronic device; compare the second predicted orientation information with the obtained second orientation information; based on the result of the comparison, generate a compensation frame using at least two of the first local image corresponding to the first orientation information, the second predicted local image, or the second local image corresponding to the second orientation information; and send the compensation frame generated based on the result of the comparison to the electronic device.

[0012] According to an example embodiment of the present disclosure, a method for providing video content performed by an electronic device, wherein the video content is received from an edge data network connected to the electronic device, the method comprising: sending first orientation information to the edge data network; obtaining from the edge data network a second prediction frame corresponding to second predicted orientation information determined based on the first orientation information; obtaining second orientation information of the second prediction frame; sending the obtained second orientation information to the edge data network; obtaining from the edge data network information as to whether the second orientation information matches the second predicted orientation information; based on whether the second orientation information matches the second predicted orientation information, receiving a compensation frame from the edge data network or generating a compensation frame that references at least a portion of the second prediction frame; based on whether the second orientation information matches the second predicted orientation information, decoding the compensation frame received from the edge data network or the generated compensation frame; and reproducing the decoded compensation frame.

[0013] According to an example embodiment of the present disclosure, an electronic device for receiving video content from an edge data network includes: a network interface; a memory storing one or more instructions; and a processor configured to execute the one or more instructions, wherein the processor is further configured to execute the one or more instructions to: send first orientation information to the edge data network; obtain from the edge data network a second predicted frame corresponding to second predicted orientation information determined based on the first orientation information; obtain second orientation information of the second predicted frame; send the obtained second orientation information to the edge data network; obtain from the edge data network information whether the second orientation information matches the second predicted orientation information; based on whether the second orientation information matches the second predicted orientation information, receive a compensation frame from the edge data network or generate a compensation frame that references at least a portion of the second predicted frame; based on whether the second orientation information matches the second predicted orientation information, decode the compensation frame received from the edge data network or the generated compensation frame; and reproduce the decoded compensation frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0015] Figure 1 is a diagram illustrating an example multi-access edge computing (MEC) technique in a network environment according to various embodiments;

[0016] Figure 2 is a diagram illustrating example operations between an electronic device and an edge data network according to various embodiments;

[0017] Figure 3 is a diagram illustrating an example relationship between partial images to be sent between an electronic device and an edge data network according to various embodiments;

[0018] Figure 4 is a sequence diagram illustrating example operations between an electronic device and an edge data network according to various embodiments;

[0019] Figure 5 is a sequence diagram illustrating example operations between an electronic device and an edge data network according to various embodiments;

[0020] Figure 6 is a diagram illustrating an example relationship between partial images to be sent between an electronic device and an edge data network according to various embodiments;

[0021] Figure 7 is a sequence diagram illustrating example operations between an electronic device and an edge data network according to various embodiments;

[0022] Figure 8 is a sequence diagram illustrating example operations between an electronic device and an edge data network according to various embodiments;

[0023] Fig. 9 is a flow chart illustrating an example method of streaming video content performed by an edge data network according to various embodiments;

[0024] Fig.10 is a flowchart illustrating an example method performed by an edge data network to determine second predicted bearing information and determine a second predicted partial image based on the second predicted bearing information according to various embodiments;

[0025] Fig.11 is a flowchart illustrating an example method performed by an edge data network to determine second predicted position information and determine a second predicted partial image based on the second predicted position information according to various embodiments;

[0026] Fig.12 is a diagram illustrating an example method of determining second predicted position information and determining a second predicted partial image based on the second predicted position information, performed by an edge data network according to various embodiments;

[0027] Fig.13 is a flowchart illustrating an example method performed by an edge data network to determine second predicted position information and determine a second predicted partial image based on the second predicted position information according to various embodiments;

[0028] Fig.14 is a flow chart illustrating an example method performed by an electronic device to stream video content obtained from an edge data network according to various embodiments;

[0029] Fig.15 is a diagram illustrating an example process performed by an electronic device to reproduce a compensation frame stored in a buffer in the electronic device according to various embodiments;

[0030] Fig.16 is a sequence diagram illustrating example operations among a cloud server, an electronic device, and an edge data network according to various embodiments;

[0031] Fig.17 is a sequence diagram illustrating example operations among a cloud server, an electronic device, and an edge data network according to various embodiments;

[0032] Fig.18 is a block diagram illustrating an example electronic device according to various embodiments. DETAILED DESCRIPTION

[0033] Hereinafter, various exemplary embodiments of the present disclosure will be described in more detail with reference to the accompanying drawings.

[0034] In the description of the embodiments, technical features that are well known in the technical field to which the present disclosure belongs and are not directly related to the present disclosure may not be described. This is not for ambiguity, but to clearly convey the main points of the present disclosure by omitting unnecessary descriptions.

[0035] Likewise, in the accompanying drawings, some components are exaggerated, omitted or schematically shown. In addition, the size of the components may not fully reflect their actual sizes. Throughout the accompanying drawings, similar reference numerals are used to refer to similar elements.

[0036] The advantages and features of the present disclosure and methods for achieving them will be clear with reference to the accompanying drawings in which example embodiments are shown. However, the present disclosure can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be thorough and complete and will fully convey the concept of the present disclosure to those skilled in the art, and the present disclosure will only be defined by the concepts of the claims.

[0037] It is understood that each block in the process flow chart and the combination of the flow chart can be performed by computer program instructions. These computer program instructions can be loaded in a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, and therefore, the instructions executed by the processor of the computer or other programmable data processing device generate a device configured to perform the functions described in (multiple) process blocks. These computer program instructions can also be stored in a computer-available or computer-readable memory, which can configure a computer or other programmable data processing device to implement functions in a specific mode. Therefore, the instructions stored in a computer-available or computer-readable memory can also generate product items involving instruction devices configured to perform the functions described in (multiple) process blocks. These computer program instructions can also be loaded in a computer or other programmable data processing device, and therefore a computer-executable process can also be generated by executing a series of operating steps on a computer or other programmable data processing device, so that the instructions executed in the computer or other programmable data processing device provide steps for performing the functions described in the process block (multiple).

[0038] In addition, each block may indicate a module, a fragment or a portion of code, including one or more executable instructions that perform a specific logical function (s). Moreover, in several alternative embodiments, the functions described in the blocks may also be out of order. For example, two consecutively shown blocks may actually be executed substantially simultaneously, or blocks may sometimes be executed in reverse order according to the corresponding functions.

[0039] The term "... unit" used in the example embodiment refers to a component including software or hardware, such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), and the "... unit" performs certain functions. However, the "... unit" is not limited to software or hardware. The "... unit" can be configured to be included in an addressable storage medium or reproduce one or more processors. Therefore, for example, the "... unit" can include, for example, components such as software components, object-oriented software components, class components and task components, processes, functions, attributes, procedures, subroutines, fragments of program code, drivers, firmware, microinstructions, circuits, data, databases, data structures, tables, arrays and variables. The functions and "... units" provided inside the component can be combined into a smaller number of components "... units", or further divided into additional components and "... units". In addition, components and "... units" can be implemented as one or more central processing units (CPUs) inside a reproduction device or a secure multimedia card. In addition, in an embodiment, the "... unit" can include one or more processors.

[0040] Throughout this disclosure, the expression "at least one of a, b, or c" refers to only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0041] Figure 1 is a diagram illustrating example multi-access edge computing (MEC) techniques in a network environment according to various embodiments.

[0042] refer to Figure 1 , the network environment 100 of the present disclosure may include an electronic device 1000, an edge data network 2000, a cloud server 3000, and an access network (AN) 1100. However, the components included in the network environment 100 are not limited to Figure 1 Components shown.

[0043] According to various embodiments, each component included in the network environment 100 may be a physical entity unit or a software or module unit capable of performing an independent function.

[0044] According to various embodiments, the electronic device 1000 may be a device used by a user. For example, the electronic device 1000 may be a terminal, a user equipment (UE), a mobile station, a subscriber station, a remote terminal, a wireless terminal, or a user device.

[0045] In addition, the electronic device 1000 may be a terminal for providing content that immerses a user in a virtual environment, the virtual environment including at least one of virtual reality (VR), augmented reality (AR), or mixed reality (MR). That is, according to various embodiments, the electronic device 1000 may be a head mounted display (HMD) or a virtual reality headset (VRH) for providing content for VR, AR, or MR.

[0046] refer to Figure 1 , the electronic device 1000 may include a first application client (or application client) 122, a second application client 124, and an edge enabler client (or MEC enabling layer (MEL)) 130. The electronic device 1000 may use the edge enabler client 130 to perform operations to use MEC services. A detailed description of the edge enabler client 130 is given below.

[0047] According to various embodiments, the electronic device 1000 may execute multiple applications. For example, the electronic device 1000 may execute a first application client 122 and a second application client 124. Multiple applications may require different network services based on, for example, the required data transmission rate, latency (or speed), reliability, the number of electronic devices accessing the network, the network access cycle of the electronic device 1000, or at least one of the average data usage. Different network services may include, for example, enhanced mobile broadband (eMBB) services, ultra-reliable low-latency communication (URLLC) services, or massive machine type communication (mMTC) services.

[0048] The application client of the electronic device 1000 may, for example, refer to a default application previously installed in the electronic device 1000 or an application provided by a third party. That is, the application client of the electronic device 1000 may refer to a client application program running in the electronic device 1000 for a specific application service. Several application clients may be running in the electronic device 1000. At least one of the application clients may use a service provided from the edge data network 2000. For example, an application client is an application installed and executed in the electronic device 1000 and may provide a function of sending and receiving data through the edge data network 2000. The application client in the electronic device 1000 may, for example, refer to application software executed in the electronic device 1000 to use functions provided by one or more specific edge applications.

[0049] According to various embodiments, multiple applications in the electronic device 1000 (e.g., the first application client 122 and the second application client 124) may perform data transmission and reception with the cloud server 3000 based on the required network service type, or perform data transmission and reception with the edge data network 2000 based on edge computing. For example, when the first application client 122 does not require low latency, the first application client 122 may perform data transmission and reception with the cloud server 3000. As another example, when the second application client 124 requires low latency, the second application client 124 may perform data transmission and reception based on MEC with the edge data network 2000.

[0050] According to various embodiments, the application in the electronic device 1000 may be referred to as an application client, a client application (client App), or a UE application (UE App). For convenience, hereinafter, in the present disclosure, the application in the electronic device 1000 is referred to as an application client.

[0051] According to various embodiments, the AN 1100 may provide a channel for wireless communication with the electronic device 1000. For example, the AN 1100 may represent a radio access network (RAN), a base station, an evolved Node B (eNode B or eNB), a fifth generation (5G) node, a transmission / reception point (TRP), or a fifth generation Node B (5GNB).

[0052] According to various embodiments, the edge data network 2000 may, for example, refer to a server that the electronic device 1000 accesses to use MEC services. The edge data network 2000 may be set at a location geographically close to the electronic device 1000, such as inside or around a base station. According to various embodiments, the edge data network 2000 may send and receive data to and from the electronic device 1000 without passing through an external data network (DN) (e.g., the Internet). According to various embodiments, MEC may refer to multi-access edge computing or mobile edge computing.

[0053] According to various embodiments, the edge data network 2000 may be referred to as an MEC host, an edge computing server, a mobile edge host, an edge computing platform, an MEC server, etc. For convenience, hereinafter, in the present disclosure, the edge data network 2000 refers to an MEC server. Figure 1 , the edge data network 2000 may include a first edge application 142, a second edge application 144, and an edge enabler server (or MEC platform (MEP)) 146. The edge enabler server 146 provides MEC services, performs business control, etc. in the edge data network 2000, and a detailed description of the edge enabler server 146 will be provided below.

[0054] According to various embodiments, the edge data network 2000 can execute multiple applications. For example, the edge data network 2000 can execute a first edge application 142 and a second edge application 144. According to various embodiments, the edge application may, for example, refer to an application provided by a third party in the edge data network that provides MEC services. The edge application can be used to establish a data session with an application client in order to send and receive data associated with the application client. That is, the edge application can establish a data session with the application client. According to various embodiments, the data session can represent a communication path established for an application client in the electronic device 1000 and an edge application in the edge data network 2000 to send and receive data to and from each other.

[0055] According to various embodiments, the application in the edge data network 2000 may be referred to as an MEC application (MEC App), ME (MEC) App, an edge application server, or an edge application. For convenience, hereinafter, in the present disclosure, the application in the edge data network 2000 is referred to as an edge application. Herein, although the term "application" is used, the edge application may, for example, refer to an application server present in the edge data network 2000.

[0056] According to various embodiments, the cloud server 3000 may provide content associated with an application. For example, the cloud server 3000 may be managed by a content provider. According to various embodiments, the cloud server 3000 may send and receive data to and from the electronic device 1000 via an external DN (e.g., the Internet).

[0057] Although not in Figure 1 1100 and the edge data network 2000. Although not shown in the figure, a core network (CN) and a DN may exist between the AN 1100 and the edge data network 2000. According to various embodiments, the DN may provide services (e.g., Internet services or Internet Protocol Multimedia Subsystem (IMS) services) to the electronic device 1000 by sending and receiving data (or data packets) to and from the electronic device 1000 via the CN and the AN 1100. For example, the DN may be managed by a network operator. According to various embodiments, the edge data network 2000 may be connected to the AN 1100 or the CN via a DN (e.g., a local DN).

[0058] According to various embodiments, when the first application client 122 or the second application client 124 is executed in the electronic device 1000, the electronic device 1000 may access the edge data network 2000 via the AN 1100 to send and receive data for executing the application client.

[0059] In an example embodiment, a method for streaming video content among an electronic device 1000, an edge data network 2000, and a cloud server 3000 may be provided. In more detail, a method for effectively providing content for VR, AR, or MR to a user based on user interaction information about a video reproduced by the electronic device 1000 is described. In the following, Figure 2 An example method of streaming video content based on position information of the video content reproduced by the electronic device 1000, performed by the edge data network 2000, is shown.

[0060] Figure 2 is a diagram illustrating an example operation between the electronic device 1000 and the edge data network 2000 according to various embodiments.

[0061] According to various embodiments, the electronic device 1000 may include a sensing module 210, a network interface 220, a processor 230, and a memory 240. However, the components in the electronic device 1000 are not limited to the components in the network interface 220. Figure 2 Components shown in , and the electronic device 1000 may include a greater or lesser number of components.

[0062] The electronic device 1000 may decode the image received from the edge data network 2000 or the cloud server and display the decoded image on the display of the electronic device 1000. In addition, the electronic device 1000 may use the sensing module 210 to obtain the orientation information of the reproduced image. Figure 4 As described above, the orientation information may include an angle value of a gaze of the user of the electronic device 1000 measured using the sensing module 210 .

[0063] The electronic device 1000 may use the network interface 220 to transmit the information 252 about the orientation information and the frame index of the partial image currently reproduced to the edge data network 2000. The frame index may be, for example, information indicating the encoding / decoding order of the frames, but is not limited thereto, and the frame index may be information indicating the presentation order of the frames.

[0064] The processor 230 controls general operations of the electronic device 1000 by executing one or more instructions in the memory 240. For example, the processor 230 may control the sensing module 210 and the network interface 220 by executing one or more instructions stored in the memory 240. According to various embodiments, the processor 230 may receive an intra-coded frame (I frame) or a predictive coded frame (P frame) from the edge data network 2000 as a predicted frame related to a second predicted partial image, the second predicted partial image being determined based on first position information of a first partial image reproduced by the electronic device 1000.

[0065] In addition, according to various embodiments, the processor 230 may decode an I frame or a P frame related to the second predicted partial image. The processor 230 may obtain second position information corresponding to the second partial image before or when reproducing the second predicted partial image.

[0066] In addition, the processor 230 may receive information about a comparison result between the second predicted partial image determined based on the first orientation information and the second partial image corresponding to the second orientation information from the edge data network 2000. According to another example embodiment, the processor 230 may receive information about whether the second predicted orientation information corresponding to the second predicted partial image matches the second orientation information corresponding to the second partial image from the edge data network 2000.

[0067] In addition, based on the information about whether the second predicted orientation information matches the second orientation information, the processor 230 may receive a compensation frame 254 or 295 corresponding to the second partial image from the edge data network 2000, the compensation frame 254 or 295 referring to at least a portion of an I frame or a P frame related to the first partial image and the second predicted partial image, or generate a compensation frame 296 referring to an I frame (not shown) or a frame 294 related to the received second predicted partial image. For example, the compensation frame 296 may refer to the I frame (not shown) or the frame 294 related to the received second predicted partial image. Figure 4 The operation performed by the processor 230 is described in detail, namely, processing the compensation frame 254 received from the edge data network 2000 when the edge data network 2000 determines that the second predicted local image does not match the second local image (or based on determining that the second predicted orientation information does not match the second orientation information).

[0068] In addition, the processor 230 may decode the compensation frame 254 received from the edge data network 2000 or the generated compensation frame 296 based on the information on whether the second predicted orientation information matches the second orientation information, and reproduce the decoded compensation frame.

[0069] In addition, according to various embodiments, the memory 240 may include a decoder module 242, a comparison module 244, and a compensation frame generation module 246. The decoder module 242 is configured to store instructions for decoding a coded frame received from the edge data network 2000. The comparison module 244 is configured to determine whether the decoded image received from the edge data network 2000 matches the image actually viewed by the user of the electronic device 1000, or to compare the predicted orientation information of the predicted frame previously predicted by the edge data network 2000 and sent to the electronic device 1000 with the orientation information obtained based on the actual input of the user of the electronic device 1000. The compensation frame generation module 246 is configured to store instructions for executing the following method in a computer, which generates a compensation frame, for example, using only the predicted partial image previously stored in the buffer of the electronic device 1000 when the predicted orientation information of the predicted frame received from the edge data network 2000 matches the orientation information as information about the angle value actually viewed by the user of the electronic device 1000, but is not limited thereto.

[0070] According to various embodiments, the edge data network 2000 may include a network interface 260, a processor 270, and a memory 280. However, the components of the edge data network 2000 are not limited to the components in the network interface 260, the processor 270, and the memory 280. Figure 2 Components shown in , and edge data network 2000 may include a larger number of components, or some components may be omitted therefrom.

[0071] The edge data network 2000 may obtain the frame index and the orientation information from the electronic device 1000 using the network interface 260 , and transmit the compensation frame 254 generated in the edge data network 2000 to the electronic device 1000 .

[0072] The processor 270 controls the overall operation of the edge data network 2000 by executing one or more instructions in the memory 280. For example, the processor 270 may determine the second predicted position information based on the first position information obtained from the electronic device 1000, and determine the second predicted partial image based on the determined second predicted position information.

[0073] The processor 270 generates a compensation frame 254, which refers to at least a portion of the I frame 291, the P frame 292, or the frame 287 in the VR sequence 288. For example, based on determining that the second predicted orientation information does not match the second orientation information, the processor 270 generates a compensation frame 232 corresponding to the second partial image based on the first partial image corresponding to the first orientation information (e.g., at least a portion of the I frame 291 or at least a portion of the P frame 293), the second predicted partial image (at least a portion of the I frame 291 or at least a portion of the P frame 293), and the second partial image corresponding to the second orientation information (e.g., at least a portion of the frame 287). The compensation frame 232 can be sent to the electronic device 1000.

[0074] The processor 270 generates a compensation frame 232 corresponding to the second partial image based on the second predicted partial image (at least a portion of the I frame 291 or at least a portion of the P frame 293) and the second partial image corresponding to the second position information (e.g., at least a portion of the frame 287) based on the determination that the second predicted orientation information matches the second orientation information. The compensation frame 232 may be discarded.

[0075] In addition, according to various embodiments, the memory 280 may include an encoder module 282, a comparison module 284, a reproduction mode prediction module 286, and a VR sequence 288. The encoder module 282 is configured to store instructions for encoding an image to be sent from the edge data network 2000 to the electronic device 1000. The comparison module 284 is configured to store instructions for determining whether the predicted orientation information predicted by the edge data network 2000 matches the orientation information including the angle value actually viewed by the user of the electronic device 1000. The reproduction mode prediction module 286 is configured to predict the image reproduction mode of the electronic device 1000 based on the orientation information as information about the angle value actually viewed by the user of the electronic device 1000. The VR sequence 288 includes all image data of the VR sequence to be described below, but is not limited thereto.

[0076] Figure 3is a diagram illustrating an example relationship between partial images to be transmitted between an electronic device and an edge data network according to various embodiments.

[0077] According to various embodiments, the edge data network 2000 may not transmit the entire VR image for VR to the electronic device 1000, but may only transmit the partial image in the VR image corresponding to the orientation information obtained from the electronic device 1000, so as to effectively provide the video content to the user of the electronic device 1000. For example, the edge data network 2000 may not transmit the frames of the first VR image 302 and the second VR image 304 to the electronic device 1000, but may only transmit the frames of the first partial image 312, the second predicted partial image 314, and the second partial image 316 to the electronic device 1000, so as to effectively transmit the video content to the electronic device 1000.

[0078] According to various embodiments, Figure 3 310 may be partial images for frames (e.g., frames 302, 304, and / or 306) in a VR sequence 310 including a plurality of frames. For example, the first partial image 312 may have the same frame index as the first VR image 302 including the first partial image 312, and the second predicted partial image 314 and the second partial image 316 may have the same frame index as the second VR image 304 including the second predicted partial image 314 and the second partial image 316.

[0079] In addition, according to various embodiments, as all or some VR images, the VR sequence 310 may be a set of all or some of the consecutive frames among the plurality of frames included in the VR image. For example, the VR sequence 310 may include a first VR image 302 having a first frame index, a second VR image 304 having a second frame index, and a third VR image 306 having a third frame index. Herein, the frame corresponding to the first frame index, the frame corresponding to the second frame index, and the frame corresponding to the third frame index may be consecutive frames, but are not limited thereto, and another frame may be included therebetween.

[0080] In addition, each of the first partial image 312, the second predicted partial image 314, and the second partial image 316 may include orientation information, which includes the angle value of the gaze of the user of the electronic device 1000 when viewing a partial image in the VR image. For example, the first partial image 312 may include first orientation information, and the position of the first partial image 312 in the first VR image 302 may be determined based on the first orientation information. In addition, the second predicted partial image 314 may include second predicted orientation information, and the position of the second predicted partial image 314 in the second VR image 304 may be determined based on the second predicted orientation information. The second partial image 316 may include second orientation information, and the position of the second partial image 316 in the second VR image 304 may be determined based on the second orientation information.

[0081] That is, the orientation information may indicate the position of each partial image in the VR image, and the position of each partial image in the VR image may be determined based on the orientation information. In addition, according to various embodiments, the range of the partial image in the VR image that can be identified based on the orientation information may depend on the display area (e.g., the size of the viewport area) in the display device that the user of the electronic device 1000 can view.

[0082] In addition, according to various embodiments, the electronic device 1000 may use a sensor (e.g., a three-axis slope sensor) to sense the orientation information (e.g., roll, pitch, and / or yaw values) of the user's eyes looking at a specific partial image in the VR image, and match the sensed orientation information with the partial image corresponding to the orientation information. The electronic device 1000 may share information about which part of the VR image the user of the electronic device 1000 is currently looking at with the edge data network 2000 by sending the sensed orientation information to the edge data network 2000.

[0083] According to various embodiments, hereinafter, it will be assumed that Figure 4 and Figure 5 Description is made assuming that the first partial image 312 is a partial image of the first VR image 302 having a first frame index in the VR sequence 310 , and the second predicted partial image 314 and the second partial image 316 are partial images of the second VR image 304 having a second frame index in the VR sequence 310 .

[0084] Figure 4 is a diagram illustrating an example operation between the electronic device 1000 and the edge data network 2000 according to various embodiments.

[0085] In operation S402, the electronic device 1000 may obtain first position information using the sensing module 210 in the electronic device. According to various embodiments, the first position information may, for example, include only information indicating the position of the first partial image 312 in the first VR image 302, and the first partial image 312 is reproduced by the electronic device 1000. However, according to various embodiments, the first position information may include multiple pieces of position information of partial images reproduced until the first partial image 312 is reproduced by the electronic device 1000. In operation S404, the electronic device 1000 may send the first position information to the edge data network 2000.

[0086] In operation S406, the edge data network 2000 may determine the second predicted position information based on the first position information. For example, the edge data network 2000 may determine the second predicted position information based on the first position information obtained from the electronic device 1000, so that the second predicted position information is the same as the first position information, or determine the second predicted position information based on a change among the plurality of position information included in the first position information. Fig.10 , Fig.11 , Fig.12 and Fig.13 The method for determining the second predicted position information performed by the edge data network 2000 is described in detail.

[0087] In operation S408, the edge data network 2000 may determine a second predicted partial image 314 corresponding to the second predicted position information. The second predicted partial image 314 refers to, for example, a partial image predicted by the edge data network 2000 for viewing by a user of the electronic device 1000, and a second partial image 316 to be described below refers to, for example, a partial image corresponding to a position (e.g., second position information) of a partial image actually viewed by the user, which is obtained from the electronic device 1000.

[0088] In operation S410, the edge data network 2000 may generate a second prediction frame by encoding the determined second prediction partial image 314. For example, the edge data network 2000 may generate an I or P frame by encoding the second prediction partial image 314. The I frame and the P frame as described above and the bidirectionally coded frame (B frame) to be described below may be a data frame for compressing an image according to a lossy compression algorithm developed in the Moving Picture Experts Group (MPEG) of the International Organization for Standardization. In addition, the B frame may indicate a data frame for compressing an image at a high compression ratio by referencing at least one of the P frame or the I frame.

[0089] In operation S412, the edge data network 2000 may transmit the generated second prediction frame to the electronic device 1000. According to various embodiments, the edge data network 2000 may transmit the second prediction frame to the electronic device 1000, and also separately store the second prediction frame transmitted to the electronic device 1000 in a database (DB) of the edge data network 2000. In operation S414, the electronic device 1000 may cache the second prediction frame obtained from the edge data network 2000 in a buffer of the electronic device 1000, and obtain second position information by using the sensing module 210 in the electronic device 1000.

[0090] In more detail, the electronic device 1000 may obtain the second position information using the sensing module 210 in the electronic device 1000 at a time point 314 (a time point before a preset time thereof) when the electronic device 1000 reproduces the second predicted partial image, at a time point after a preset time has passed from a time point when the first partial image 312 has been reproduced (a time point before a preset time thereof), or at a time point before a preset time from a time point when the second predicted partial image 314 is predicted to be reproduced. In operation S416, the electronic device 1000 may send the obtained second position information to the edge data network 2000.

[0091] For example, instead of decoding the second predicted frame and using the decoded second predicted frame to immediately reproduce the second predicted partial image 314, the electronic device 1000 can obtain information about the position of the partial image viewed by the user in the second VR image 304 as second orientation information, and send the obtained second orientation information to the edge data network 2000 to check whether the second predicted partial image 314 based on the second predicted orientation information currently cached in the buffer of the electronic device 1000 matches the partial image that the user is currently going to view in the second VR image 304.

[0092] In operation S418, the edge data network 2000 may determine whether the second predicted orientation information determined in operation S406 matches the second orientation information obtained from the electronic device 1000. For example, when the average value of the difference between the second orientation information and the angle components (e.g., pitch, roll, and / or yaw) in the second predicted orientation information is greater than or equal to a preset threshold angle, the edge data network 2000 may determine that the second predicted orientation information does not match the second orientation information.

[0093] In addition, according to various embodiments, the edge data network 2000 may compare the second predicted orientation information with the second orientation information or compare the second predicted partial image 314 corresponding to the second predicted orientation information with the second partial image 316 corresponding to the second orientation information. For example, the edge data network 2000 may store the VR sequence 310 including all frames in a DB in the edge data network 2000. The edge data network 2000 may use the second orientation information obtained from the electronic device 1000 to identify the second partial image 316 in the VR sequence 310 stored in the DB. The edge data network 2000 may compare the pixel value of the identified second partial image 316 with the pixel value of the second predicted partial image 314 corresponding to the second predicted orientation information, and determine that the second partial image 316 matches the second predicted partial image 314 when the number of pixels whose pixel value difference is greater than or equal to a preset threshold is greater than or equal to a predetermined threshold.

[0094] In operation S420, the edge data network 2000 may transmit information about a comparison result indicating that the second predicted orientation information does not match the second orientation information to the electronic device 1000. In operation S422, the electronic device 1000 may wait for reproduction of the second predicted partial image 314 based on the information about a comparison result indicating that the second predicted orientation information does not match the second orientation information, which is received from the edge data network 2000.

[0095] In operation S424 , the edge data network 2000 may generate a compensation frame corresponding to the second partial image 316 based on the first partial image 312 , the second predicted partial image 314 , and the second partial image 316 based on determining that the second predicted orientation information does not match the second orientation information.

[0096] In more detail, based on determining that the second predicted orientation information does not match the second orientation information, the edge data network 2000 can use the second orientation information obtained from the electronic device 1000 to access the VR sequence 310 stored in the DB, thereby obtaining the second local image 316 in the VR sequence 310 corresponding to the second orientation information.

[0097] According to various embodiments, reference Figure 3 , since the first partial image 312 is currently being reproduced by the electronic device 1000, the first frame generated by encoding the first partial image 312 and the second predicted frame transmitted to the electronic device 1000 in operation S412 may have been stored in the DB of the edge data network 2000. The edge data network 2000 may generate a compensation frame corresponding to the second partial image 316 so as to refer to at least a portion of the first frame and the second predicted frame transmitted to the electronic device 1000 in operation S412.

[0098] Specifically, the edge data network 2000 can determine information about the difference between image information in a first frame corresponding to the first local image 312 and image information in a second frame corresponding to the second local image 316 as first difference information; determine information about the difference between image information in a second predicted frame corresponding to the second predicted local image 314 and image information in the second frame as second difference information; and use the first difference information and the second difference information to generate a compensation frame corresponding to the second local image 316.

[0099] According to various embodiments, the first difference information may include first reference block information indicating a block address (or position) in the first frame and a block address (or position) in the second frame (which is used to determine the first difference information) and information about a pixel difference between a block in the first frame and a block in the second frame (which is used to determine the first difference information). For example, the first reference block information may include motion vector information. The motion vector information (i.e., shift information) may indicate a difference between a block position in the first frame and a block position in the second frame.

[0100] In addition, according to various embodiments, the second difference information may include: second reference block information indicating a block address (or position) in the second frame and a block address (or position) in the second predicted frame (which is used to determine the second difference information), and information about a pixel difference between a block in the second frame and a block in the second predicted frame (which is used to determine the second difference information).

[0101] According to various embodiments, the edge data network 2000 can sequentially obtain a P frame corresponding to the first partial image 312, a compensation frame corresponding to the second partial image 316, and a P frame corresponding to the second predicted partial image 314 from the encoder by inputting images to the encoder in the order of the first partial image 312, the second partial image 316, and the second predicted partial image 314, but the encoder encoding sequence is not limited thereto. That is, the encoder in the edge data network 2000 can encode the first partial image 312 as a first frame, encode the second predicted partial image 314 as a second predicted frame, and then encode the second partial image 316 so as to refer to at least a portion of the encoded first frame and at least a portion of the encoded second predicted frame.

[0102] In operation S426, the edge data network 2000 may send the compensation frame generated in operation S424 to the electronic device 1000. In operation S428, the electronic device 1000 may decode the compensation frame obtained from the edge data network 2000 and reproduce the decoded compensation frame. For example, the electronic device 1000 may obtain first reference block information and second reference block information in the compensation frame, and decode the compensation frame by referring to some blocks in the first frame corresponding to the first reference block information and some blocks in the second prediction frame corresponding to the second reference block information. The electronic device 1000 may decode the compensation frame and pass the decoded compensation frame to the display of the electronic device 1000, so that the second partial image 316 is provided to the user of the electronic device 1000.

[0103] Figure 5 is a sequence diagram illustrating example operations between the electronic device 1000 and the edge data network 2000 according to various embodiments.

[0104] The above referenced Figure 4 An example operation process between the electronic device 1000 and the edge data network 2000 when the second predicted position information does not match the second position information is described, and will now be referred to Figure 5 To describe an example operation process between the electronic device 1000 and the edge data network 2000 when the second predicted position information matches the second position information. Figure 5 Operations S502 to S516 may correspond to Figure 4 Operations S402 to S416 are described in detail, and thus their detailed description is not repeated here.

[0105] In operation S518, the edge data network 2000 may determine whether the second predicted orientation information determined in operation S506 matches the second orientation information obtained from the electronic device 1000. For example, when the average value of the difference between the second orientation information and the angle components (e.g., pitch, roll, and / or yaw) in the second predicted orientation information is greater than or equal to a preset threshold angle, the edge data network 2000 may determine that the second predicted orientation information matches the second orientation information.

[0106] In operation S520, the edge data network 2000 may transmit information about a comparison result indicating that the second predicted orientation information matches the second orientation information to the electronic device 1000. In operation S522, the edge data network 2000 may generate a compensated frame by encoding the second predicted partial image 314 stored in the DB based on determining that the second predicted orientation information matches the second orientation information.

[0107] In more detail, based on determining that the second predicted orientation information matches the second orientation information, the edge data network 2000 may determine that the second predicted partial image 314 is identical to the second partial image 316, and generate a compensation frame that references at least some blocks in the second predicted frame. In operation S524, the electronic device 1000 may generate a compensation frame having the same image information as the image information of the second predicted frame received in operation S512 based on a comparison result indicating that the second predicted orientation information matches the second orientation information. The electronic device 1000 may reproduce an image by, for example, using only the compensation frame obtained from the edge data network 2000 and the compensation frame generated by the electronic device 1000, as described below with reference to FIG. Fig.16 As described above, the video content can be efficiently provided to the user without affecting the decoding order of the I or P frames cached in the buffer of the electronic device 1000. Therefore, even when a comparison result indicating that the second predicted orientation information matches the second orientation information is received, the electronic device 1000 can also generate a compensation frame having the same image information as the image information of the second predicted frame.

[0108] In more detail, the edge data network 2000 can generate a compensation frame using difference information of the difference between the image information in the second prediction frame and the image information in the second frame, and because it has been determined that the second prediction orientation information matches the second orientation information, the difference between the image information in the second prediction frame and the image information in the second frame is determined to be 0, so that the second prediction frame and the compensation frame can have the same image information. In addition, as described above with reference to Figure 4 As described, in addition to information about pixel differences between blocks in the second predicted frame and blocks in the second frame, difference information of the difference between image information in the second predicted frame and image information in the second frame also includes reference block information indicating a block address (or position) in the second predicted frame and a block address (or position) in the second frame, and the reference block information is used to determine the difference information.

[0109] In operation S526, the edge data network 2000 may discard the generated compensation frame. For example, because it has been determined that the second predicted partial image 314 matches the second partial image 316, the edge data network 2000 does not need to send a separate frame, so the edge data network 2000 may discard the generated compensation frame and not send the generated compensation frame to the electronic device 1000.

[0110] According to various embodiments, when the second predicted orientation information matches the second orientation information, the edge data network 2000 does not have to generate a separate compensation frame and transmit the generated compensation frame to the electronic device 1000, and thus unlike operation S522, the edge data network 2000 may not generate a separate compensation frame. However, according to another exemplary embodiment, even when it is determined that the second predicted orientation information matches the second orientation information, the edge data network 2000 may generate a compensation frame that refers to the second predicted frame, thereby synchronizing with the frame decoding operation of the electronic device 1000. In operation S528, the electronic device 1000 may decode the compensation frame generated in operation S524 and reproduce the decoded compensation frame. For example, the electronic device 1000 may obtain reference block information in the compensation frame, and decode the compensation frame by referring to some blocks in the second predicted frame and some blocks in the first frame based on the obtained reference block information. At present, the second predicted frame includes the same image information as the compensation frame, and thus the electronic device 1000 may decode the compensation frame by referring only to the second predicted frame. The electronic device 1000 may decode the compensating frame generated in operation S524 and transfer the decoded compensating frame to a display of the electronic device 1000 , so that the second predicted partial image 314 is provided to a user of the electronic device 1000 .

[0111] As described, in order to efficiently provide video content to a user without affecting the decoding order of I or P frames cached in the buffer of the electronic device 1000, even when a comparison result indicating that the second predicted orientation information matches the second orientation information is received, in operation S524, a compensation frame including image information identical to that of the second predicted frame is generated, but various embodiments are not limited thereto, and the electronic device 1000 may not generate a compensation frame separately. In this case, the electronic device 1000 may decode the second predicted frame cached in operation S514 and pass the decoded second predicted frame (the second predicted partial image 314) to the display of the electronic device 1000, so that the second predicted partial image 314 is reproduced (provided to the user of the electronic device 1000).

[0112] exist Figure 4 and Figure 5 In addition to the first position information, the edge data network 2000 may also obtain a frame index of the first partial image 312 currently reproduced by the electronic device 1000 from the electronic device 1000. The edge data network 2000 may determine a frame index of the second predicted frame by referring to the frame index of the first partial image 312 obtained from the electronic device 1000.

[0113] As described above, when the edge data network 2000 determines that the second predicted orientation information matches the second orientation information, because the second predicted local image 314 corresponding to the second predicted frame sent from the edge data network 2000 in operation S512 is the same as the current second local image 316 corresponding to the second orientation information, the electronic device 1000 can decode the second predicted frame previously cached as an I or P frame in the buffer of the electronic device 1000 and then reproduce the decoded second predicted frame.

[0114] However, when the edge data network 2000 determines that the second predicted orientation information matches the second orientation information and the electronic device 1000 therefore reproduces the image using the second predicted frame received from the edge data network 2000 and previously cached in the buffer as an I or P frame, the situation where the image is reproduced using the compensation frame when the edge data network 2000 determines that the second predicted orientation information does not match the second orientation information may coexist, and therefore, the decoding operation of the electronic device 1000 performed according to the buffer management or the reference relationship between the frames may not be performed efficiently. Therefore, the electronic device 1000 according to the present disclosure can effectively provide the VR image to the user by the following operations: when the image prediction of the edge data network 2000 is successful, the compensation frame generated by the electronic device 1000 is decoded and then the decoded compensation frame is cached in the buffer; when the image prediction of the edge data network 2000 fails, the compensation frame received from the edge data network 2000 is decoded and then the decoded compensation frame is sequentially cached in the buffer as in operation S428; and then only the compensation frame stored in the buffer is presented.

[0115] Figure 6 is a diagram illustrating an example relationship between partial images to be transmitted between an electronic device and an edge data network according to various embodiments.

[0116] The edge data network 2000 may transmit only a partial image in the VR image corresponding to the orientation information obtained from the electronic device 1000 to the electronic device 1000, instead of transmitting the entire VR image for VR. The description has been made on the assumption that the edge data network 2000 transmits the second prediction frame to the electronic device 1000. Figure 3 , Figure 4 and Figure 5 , but will be described under the assumption that the edge data network 2000 sends both the second prediction frame and the third prediction frame to the electronic device 1000 Figure 6 , Figure 7 and Figure 8 .

[0117] For example, the edge data network 2000 can effectively send video content to the electronic device 1000 by sending frames of the first local image 612, the second predicted local image 614, the third predicted local image 616 and the third re-predicted local image 622 to the electronic device 1000 instead of sending frames of the first VR image 602, the second VR image 604 and the third VR image 606 to the electronic device 1000.

[0118] According to various embodiments, Figure 6 6 may be partial images for the corresponding frames 602, 604, and 606 in the VR sequence 610 including a plurality of frames. For example, the first partial image 612 may have the same frame index as the first VR image 602 including the first partial image 612, and the second predicted partial image 614 and the second partial image 618 may have the same frame index as the second VR image 604. In addition, the third predicted partial image 616 and the third re-predicted partial image 622 may have the same frame index as the third VR image 606.

[0119] In addition, according to various embodiments, as all or some VR images, the VR sequence 610 may be a set of all or some continuous frames among a plurality of frames included in the VR image. For example, the VR sequence 610 may include a first VR image 602 having a first frame index, a second VR image 604 having a second frame index, and a third VR image 606 having a third frame index. In addition, the first partial image 612, the second predicted partial image 614, the third predicted partial image 616, and the third re-predicted partial image 622 may include orientation information for determining which partial image in the VR image is viewed by the user of the electronic device 1000.

[0120] For example, the first partial image 612 may include first orientation information, and the position of the first partial image 612 in the first VR image 602 may be determined based on the first orientation information. In addition, the second predicted partial image 614 may include second predicted orientation information, and the position of the second predicted partial image 614 in the second VR image 604 may be determined based on the second predicted orientation information. The third predicted partial image 616 may include third predicted orientation information, and the position of the third predicted partial image 616 in the third VR image 606 may be determined based on the third predicted orientation information. The position of the third re-predicted partial image 622 in the third VR image 606 may be determined based on the third re-predicted orientation information.

[0121] The orientation information may represent the position of each partial image in the VR image, and the position of each partial image in the VR image may be determined based on the orientation information. According to various embodiments, the orientation information representing the position of the partial image in the VR image may include a three-axis angle value obtained from the sensing module 210 attached to the electronic device 1000, and the three-axis angle may be changed based on the input of the user wearing the electronic device 1000.

[0122] The predicted frame (e.g., an I or P frame 632) includes data generated by encoding a partial image in a VR image, and the partial image 634 in the VR image represents an image that appears when the frame is decoded, and the decoded frame is displayed on the display of the electronic device 1000.

[0123] According to various embodiments, hereinafter, it will be assumed that Figure 7 and Figure 8 Described herein, it is assumed that the first partial image 612 is a partial image in the first VR image 602 having a first frame index in the VR sequence 610, the second predicted partial image 614 and the second partial image 618 are partial images in the second VR image 604 having a second frame index in the VR sequence 610, and the third predicted partial image 616 and the third re-predicted partial image 622 are partial images in the third VR image 606 having a third frame index.

[0124] Figure 7 is a sequence diagram illustrating example operations between the electronic device 1000 and the edge data network 2000 according to various embodiments.

[0125] In operation S702, the electronic device 1000 may obtain first orientation information. According to various embodiments, the electronic device 1000 may use the sensing module 210 in the electronic device 1000 to obtain a three-axis angle value indicating the position of the first partial image 612 in the first VR image 602 as the first orientation information. According to another example embodiment, the first orientation information may include multiple pieces of orientation information of the partial image reproduced by the electronic device 1000 before reproducing the first partial image 612. In operation S704, the electronic device 1000 may send the first orientation information to the edge data network 2000.

[0126] In operation S706, the edge data network 2000 may determine (primarily predict) second predicted location information and third predicted location information based on the first location information. Figure 6 and Figure 7It is shown that the edge data network 2000 determines two pieces of prediction orientation information (second prediction orientation information and third prediction orientation information), but the present disclosure is not limited thereto, and the edge data network 2000 may determine several pieces of prediction orientation information for two or more partial images in at least one group of pictures (GOP).

[0127] In addition, despite Figure 6 and Figure 7 It is shown that the second predicted local image 614 corresponding to the second predicted orientation information is a local image in the second VR image 604 with a second frame index, and the third predicted local image 616 is a local image in the third VR image 606 with a third frame index. The second predicted local image 614 and the third predicted local image 616 can be local images in VR images with a predetermined frame index interval.

[0128] According to various embodiments, the edge data network 2000 may determine the second predicted position information and the third predicted position information based on the first position information, so that the second predicted position information and the third predicted position information are the same as the first position information. However, various embodiments are not limited thereto, and the second predicted position information and the third predicted position information may be determined according to a method for determining predicted position information performed by the edge data network 2000, referring to Fig.10 , Fig.11 , Fig.12 and Fig.13 To describe the method.

[0129] In operation S708, the edge data network 2000 may determine a second predicted partial image 614 corresponding to the second predicted orientation information and a third predicted partial image 616 corresponding to the third predicted orientation information. For example, the edge data network 2000 may use the determined second predicted orientation information to identify the second predicted partial image 614 corresponding to the second predicted orientation information from the VR sequence 610 stored in the DB, and in the same manner, use the third predicted orientation information to identify the third predicted partial image 616 from the VR sequence 610 in the DB.

[0130] In operation S710, the edge data network 2000 may generate a second prediction frame and a third prediction frame by encoding the second prediction partial image 614 and the third prediction partial image 616. According to various embodiments, the second prediction frame and the third prediction frame generated by the edge data network 2000 may be an I frame or a P frame. In operation S712, the edge data network 2000 may send the generated second prediction frame and the third prediction frame to the electronic device 1000. According to various embodiments, the edge data network 2000 may store the second prediction frame and the third prediction frame sent to the electronic device 1000 separately in the DB of the edge data network 2000. According to another example embodiment, the edge data network 2000 may only store the second prediction orientation information and the third prediction orientation information corresponding to the second prediction frame and the third prediction frame sent to the electronic device 1000, respectively, separately in the DB.

[0131] In operation S714, the electronic device 1000 may reproduce the second predicted partial image 614 and obtain second position information corresponding to the second partial image 618. In more detail, the electronic device 1000 may decode the second predicted frame among the second predicted frame and the third predicted frame received from the edge data network 2000 and use the decoded second predicted frame to reproduce the second predicted partial image 614. When the electronic device 1000 reproduces the second predicted partial image 614, the third predicted frame may be cached in a buffer of the electronic device 1000.

[0132] According to various embodiments, the electronic device 1000 may obtain the second position information from the sensing module 210 in the electronic device 1000 at a time point when the second predicted partial image 614 is predicted to be reproduced, at a time point when a preset time has passed from the time point when the second partial image 614 is reproduced, or when the second predicted partial image 614 is being reproduced. That is, the electronic device 1000 may reproduce the second predicted partial image 614 predicted by the edge data network 2000 and also obtain the position of the partial image located at a point that the user of the electronic device 1000 is actually going to watch at present as the second position information.

[0133] According to various embodiments, when the second orientation information obtained by the electronic device 1000 does not match the second predicted orientation information predicted by the edge data network 2000, the video content may not be displayed on a portion of the viewport area of ​​the display device of the electronic device 1000. Therefore, according to various embodiments, in order to solve the problem of not displaying the video content on a portion of the viewport area of ​​the electronic device 1000 when the image prediction of the edge data network 2000 fails, the edge data network 2000 may transmit to the electronic device 1000 a predicted frame generated by encoding a predicted partial image having a size larger than a preset size of the viewport area of ​​the electronic device 1000. In operation S716, the electronic device 1000 may transmit the second orientation information to the edge data network 2000.

[0134] In operation S718, the edge data network 2000 may determine (secondarily predict) third re-predicted orientation information based on the second orientation information. In operation S720, the edge data network 2000 may determine whether the third predicted orientation information matches the third re-predicted orientation information. For example, when the average of the difference between the third predicted orientation information and the angular components (e.g., pitch, roll, and / or yaw) in the third re-predicted orientation information is greater than or equal to a preset threshold angle, the edge data network 2000 may determine that the third predicted orientation information does not match the third re-predicted orientation information.

[0135] Furthermore, according to another exemplary embodiment, instead of Figure 4 Operation S418 is similar to comparing the third prediction orientation information with the third re-prediction orientation information. The third prediction partial image 616 corresponding to the third prediction orientation information may be compared with the third re-prediction partial image 622 corresponding to the third re-prediction orientation information.

[0136] In operation S722, the edge data network 2000 may transmit information about a comparison result indicating that the third predicted orientation information does not match the third re-predicted orientation information to the electronic device 1000. In operation S724, the electronic device 1000 may wait for reproduction of the third predicted partial image 616 based on the comparison result indicating that the third predicted orientation information does not match the third re-predicted orientation information.

[0137] In operation S726, the edge data network 2000 may generate a compensation frame corresponding to the third re-predicted partial image 622 based on the second predicted partial image 614, the third predicted partial image 616, and the third re-predicted partial image 622 in response to determining that the third predicted orientation information does not match the third re-predicted orientation information.

[0138] In more detail, based on determining that the third predicted orientation information does not match the third re-predicted orientation information, the edge data network 2000 can use the third re-predicted orientation information newly determined based on the second orientation information obtained from the electronic device 1000 to identify the third re-predicted partial image 622 from the VR sequence 610 in the DB.

[0139] According to various embodiments, the second predicted partial image 614 is being reproduced by the electronic device 1000, and the second predicted frame generated by encoding the second predicted partial image 614 may have been stored in the DB of the edge data network 2000. The edge data network 2000 may generate a compensation frame corresponding to the third re-predicted partial image 622 so as to refer to at least some blocks in the second predicted frame and the third predicted frame transmitted to the electronic device 1000 in operation S712.

[0140] Specifically, the edge data network 2000 can determine information about the difference between image information in a second predicted frame corresponding to the second predicted local image 614 and image information in a third re-predicted frame corresponding to the third re-predicted local image 622 as third difference information; determine information about the difference between image information in a third re-predicted frame corresponding to the third re-predicted local image 622 and image information in a third predicted frame corresponding to the third predicted local image 616 as fourth difference information, and use the third difference information and the fourth difference information to generate a compensation frame corresponding to the third re-predicted local image 622.

[0141] According to various embodiments, the third difference information may include: third reference block information representing a block address (or position) in the second prediction frame and a block address (or position) in the third re-prediction frame (which is used to determine the third difference information); and information about a pixel difference between a block in the second prediction frame and a block in the third re-prediction frame (which is used to determine the third difference information).

[0142] In addition, according to various embodiments, the fourth difference information may include: fourth reference block information representing a block address (or position) in the third re-predicted frame and a block address (or position) in the third predicted frame (which is used to determine the fourth difference information), and information about a pixel difference between a block in the third re-predicted frame and a block in the third predicted frame (which is used to determine the fourth difference information).

[0143] According to various embodiments, the edge data network 2000 may sequentially obtain a P frame corresponding to the second predicted local image 614 and a P frame corresponding to the third predicted local image 616 from the encoder by inputting the second predicted local image 614, the third re-predicted local image 622, and the third predicted local image 616 into the encoder in this order, but the encoding sequence of the encoder is not limited thereto. That is, the encoder in the edge data network 2000 may generate a compensation frame by encoding the second predicted local image 614 as a second predicted frame, encoding the third predicted local image 616 as a third predicted frame, and then encoding the third re-predicted local image 622 so as to refer to at least some blocks in the encoded second predicted frame and at least some blocks in the encoded third predicted frame. In operation S728, the edge data network 2000 may send the generated compensation frame to the electronic device 1000.

[0144] In operation S730, the electronic device 1000 may decode the compensation frame and reproduce the decoded compensation frame. For example, the electronic device 1000 may identify the third reference block information and the fourth reference block information in the compensation frame, and decode the compensation frame by referring to some blocks in the second prediction frame corresponding to the identified third reference block information and some blocks in the third prediction frame corresponding to the fourth reference block information. The electronic device 1000 may decode the compensation frame and transmit the decoded compensation frame to the display device of the electronic device 1000, so that the third re-predicted partial image 622 is reproduced.

[0145] According to various embodiments, before predicting a time point at which the third re-predicted partial image 622 is reproduced, while the electronic device 1000 reproduces the second predicted partial image 614 , the electronic device 1000 may cache the decoded compensation frame in a buffer of the electronic device 1000 .

[0146] Although the edge data network 2000 predicts the second predicted orientation information and sends the predicted second predicted orientation information to the electronic device 1000, when the second orientation information obtained by the electronic device 1000 does not match the second predicted orientation information when the electronic device 1000 is reproducing the second predicted local image 614, there is a high possibility that the third orientation information obtained by the electronic device 1000 does not match the third predicted orientation information when the electronic device 1000 is reproducing the third predicted local image 616 corresponding to the third predicted orientation information.

[0147] Therefore, according to the above reference Figure 7According to the method described, the edge data network 2000 can determine third predicted orientation information based on the first orientation information, and then determine third re-predicted orientation information based on the newly obtained second orientation information, so as to more accurately predict the image that the user of the electronic device 1000 will view.

[0148] Figure 8 is a sequence diagram illustrating example operations between the electronic device 1000 and the edge data network 2000 according to various embodiments.

[0149] The above referenced Figure 7 The operation process between the electronic device 1000 and the edge data network 2000 when the third predicted position information does not match the third re-predicted position information is described, and now reference will be made to Figure 8 The operation process between the electronic device 1000 and the edge data network 2000 is described when the third predicted location information matches the third re-predicted location information. Figure 8 Operations S802 to S818 may correspond to Figure 7 Operations S702 to S718 are described in detail, and thus their detailed description is not repeated here.

[0150] In operation S820, the edge data network 2000 may determine whether the third predicted orientation information matches the third re-predicted orientation information. For example, when the average of the difference between the third predicted orientation information and the angle components (e.g., pitch, roll, and / or yaw) in the third re-predicted orientation information is less than a preset threshold angle, the edge data network 2000 may determine that the third predicted orientation information matches the third re-predicted orientation information.

[0151] In operation S822, the edge data network 2000 may transmit information about a comparison result indicating that the third predicted orientation information matches the third re-predicted orientation information to the electronic device 1000. In operation S824, the edge data network 2000 may generate a compensated frame by encoding the third predicted partial image 616 pre-stored in the DB when the third predicted orientation information matches the third re-predicted orientation information.

[0152] In more detail, based on determining that the third prediction orientation information matches the third re-prediction orientation information, the edge data network 2000 may generate a compensation frame that references at least some blocks in the third prediction frame generated by encoding the third prediction partial image 616 .

[0153] In operation S826, based on the comparison result indicating that the third predicted orientation information matches the third re-predicted orientation information, the electronic device 1000 may generate a compensation frame having the same image information as the image information of the third predicted frame received in operation S812. The electronic device 1000 may, for example, reproduce an image using only the compensation frame to efficiently provide video content to a user without affecting the decoding order of I or P frames cached in a buffer of the electronic device 1000. Therefore, even when a comparison result indicating that the third predicted orientation information matches the third re-predicted orientation information is received, the electronic device 1000 may generate a compensation frame having the same image information as the image information of the third predicted frame. Operation S826 may correspond to Figure 5 Operation S524 is described below, and therefore its detailed description is not repeated here.

[0154] In operation S828, the edge data network 2000 may discard the generated compensation frame. For example, because the third predicted orientation information has been determined to match the third re-predicted orientation information, the edge data network 2000 may determine that the third predicted partial image 616 is the same as the third re-predicted partial image 622, and because the third predicted frame generated by encoding the third predicted partial image 616 is currently transmitted to the electronic device 1000, the edge data network 2000 does not need to transmit a separate compensation frame to the electronic device 1000. Therefore, the edge data network 2000 may discard the generated compensation frame without transmitting the generated compensation frame to the electronic device 1000.

[0155] According to various embodiments, when the third predicted orientation information matches the third re-predicted orientation information, the edge data network 2000 may not generate a separate compensation frame. However, even when the third predicted orientation information matches the third re-predicted orientation information, in order to synchronize with the decoding operation performed by the electronic device 1000, the edge data network 2000 according to the present disclosure may separately generate a compensation frame according to the operation of generating a compensation frame performed by the electronic device 1000.

[0156] In operation S830, the electronic device 1000 may decode the compensation frame generated in operation S826 and reproduce the decoded compensation frame. For example, the electronic device 1000 may obtain reference block information in the compensation frame, and decode the compensation frame by referring to some blocks in the frame referenced by the compensation frame based on the obtained reference block information. Currently, the third prediction frame includes the same image information as the compensation frame, so the electronic device 1000 may decode the compensation frame by referring to only the third prediction frame.

[0157] Fig. 9 is a flow chart illustrating an example method of streaming video content performed by an edge data network in accordance with various embodiments.

[0158] In operation S910, the edge data network 2000 may obtain first position information from the electronic device 1000. Operation S910 may correspond to Figure 4 Operation S402 is described below, and therefore its detailed description is not repeated here.

[0159] In operation S920, the edge data network 2000 may determine second predicted position information based on the first position information. Operation S920 may correspond to Figure 4 Operation S406. Fig.10 , Fig.11 , Fig.12 and Fig.13 Operation S920 is described in more detail.

[0160] In operation S930, the edge data network 2000 may determine a second predicted partial image corresponding to the second predicted position information. Operation S930 may correspond to Figure 4 Operation S408 is described below, and therefore its detailed description is not repeated here.

[0161] In operation S940, the edge data network 2000 may transmit the second predicted frame generated by encoding the second predicted partial image to the electronic device 1000. Operation S940 may correspond to Figure 4 Operations S410 to S420 are described in detail, and thus a detailed description thereof is not repeated here.

[0162] In operation S950, the edge data network 2000 may obtain second position information from the electronic device 1000. Operation S950 may correspond to Figure 4 Operation S414 is described below, and therefore its detailed description is not repeated here.

[0163] In operation S960, the edge data network 2000 may compare the second predicted position information with the second position information. Fig. 9 , after operation S960, the edge data network 2000 may further perform sending information about a comparison result between the second predicted position information and the second position information to the electronic device 1000. Operation S960 may correspond to Figure 4 Operation S418 is described below, and therefore its detailed description is not repeated here.

[0164] In operation S970, the edge data network 2000 may generate a compensation frame based on a comparison result between the second predicted orientation information and the second orientation information. For example, based on determining that the second predicted orientation information does not match the second orientation information, the edge data network 2000 may generate a compensation frame corresponding to the second partial image based on the first partial image, the second predicted partial image, and the second partial image. The specific method of generating the compensation frame performed by the edge data network 2000 when the second predicted orientation information does not match the second orientation information may correspond to Figure 4 Operation S424 is described below, and therefore its detailed description is not repeated here.

[0165] In addition, based on determining that the second predicted orientation information matches the second orientation information, the edge data network 2000 may determine that the second predicted partial image is the same as the second partial image, and generate a compensation frame that references at least some blocks in the second predicted frame. Based on determining that the second predicted orientation information matches the second orientation information, the method of generating a compensation frame performed by the edge data network 2000 may correspond to operation S522, and thus a detailed description thereof is not repeated here.

[0166] In operation S980, the edge data network 2000 may send a compensation frame generated based on a comparison result between the second predicted orientation information and the second orientation information to the electronic device 1000. For example, when the second predicted orientation information matches the second orientation information, the edge data network 2000 may discard the generated compensation frame. However, when the second predicted orientation information does not match the second orientation information, the edge data network 2000 may send the generated compensation frame to the electronic device 1000. Operation S980 may correspond to Figure 4 Operation S426 or Figure 5 S526, so its detailed description is not repeated here.

[0167] Fig.10 is a flowchart illustrating an example method performed by an edge data network to determine second predicted position information and to determine a second predicted partial image based on the second predicted position information according to various embodiments.

[0168] Now refer to Fig.10 A specific example of the operation of determining (S920) second predicted orientation information based on first orientation information and determining (S930) a second predicted partial image corresponding to the determined second predicted orientation information, performed by the edge data network 2000, is described.

[0169] In operation S1010, the edge data network 2000 may determine the second predicted orientation information so that the first orientation information obtained from the electronic device 1000 is the same as the second predicted orientation information. For example, the first orientation information and the second predicted orientation information may, for example, include a pitch angle component, a roll angle component, and a yaw angle component as three-axis angle components. According to various embodiments, when the first orientation information obtained from the electronic device 1000 includes 30 degrees, 45 degrees, and 60 degrees in the order of the pitch angle component, the roll angle component, and the yaw angle component, the edge data network 2000 may determine the second predicted orientation information to be 30 degrees, 45 degrees, and 60 degrees.

[0170] In operation S1020, the edge data network 2000 may determine a second predicted partial image using the second predicted orientation information. For example, the edge data network 2000 may store VR sequences for all frames in a DB and identify a second predicted partial image from the VR sequences in the DB using the second predicted orientation information.

[0171] According to various embodiments, in addition to the first orientation information, the edge data network 2000 may further receive, from the electronic device 1000, information about a frame index indicating a specific frame number in a VR sequence predicted to be reproduced by the electronic device 1000. For example, when the information about the frame index obtained from the electronic device 1000 indicates a second frame index in the VR sequence, the edge data network 2000 may identify a second VR image corresponding to the second frame index of the VR sequence pre-stored in the DB, and identify a second predicted partial image corresponding to the second predicted orientation information in the identified second VR image.

[0172] Fig.11 is a flowchart illustrating an example method performed by an edge data network to determine second predicted position information and to determine a second predicted partial image based on the second predicted position information according to various embodiments.

[0173] Now refer to Fig.11 A method of determining second predicted position information and determining a second predicted partial image corresponding to the determined second predicted position information, performed by the edge data network 2000 according to various embodiments is described.

[0174] In operation S1110, the edge data network 2000 may determine a first change between the orientation information of the partial image reproduced by the electronic device 1000 before the first partial image and the first orientation information. For example, when the orientation information includes a three-axis angle component, the orientation information of the partial image reproduced by the electronic device 1000 before the first partial image is (x0, y0, z0), and the first orientation information is (x1, y1, z1), then the first change may be (x1-x0, y1-y0, z1-z0).

[0175] In operation S1120, the edge data network 2000 may determine the second predicted orientation information so that the second change between the second predicted orientation information and the first orientation information is the same as the first change. For example, when the second predicted orientation information is (X, Y, Z), the edge data network 2000 may determine the second change to be (X-x1, Y-y1, Z-z1). The edge data network 2000 may determine the second predicted orientation information so that the second change (X-x1, Y-y1, Z-z1) is the same as the first change (x1-x0, y1-y0, z1-z0).

[0176] In operation S1130, the edge data network 2000 may determine a second predicted partial image based on the second predicted position information. The method of identifying the second predicted partial image from the DB using the second predicted position information performed by the edge data network 2000 may correspond to Figure 4 Operation S408 is described below, and therefore its detailed description is not repeated here.

[0177] Fig.12 is a diagram illustrating an example method of determining second predicted position information and determining a second predicted partial image based on the second predicted position information, performed by an edge data network according to various embodiments.

[0178] As mentioned above Fig.10 As described, the edge data network 2000 can determine the second predicted position information so that the second predicted position information is the same as the first position information. In addition, similar to the method for determining the second predicted position information, the edge data network 2000 can determine the third predicted position information so that the third predicted position information is the same as the second predicted position information.

[0179] That is to say, reference Fig.12 With the fixed prediction 1210, the edge data network 2000 can predict local images for each frame index so that a VR image 1212 with a frame index N, a VR image 1214 with a frame index N+1, and a VR image 1216 with a frame index N+2 have the same orientation information.

[0180] However, according to other example embodiments, as described above with reference to Fig.11 As described above, the edge data network 2000 can determine the second predicted orientation information so that the changes between the pieces of orientation information of the local images in two adjacent frames are the same. Fig.12Based on the acceleration linear prediction 1220, the edge data network 2000 can predict the orientation information of the local image in the VR image 1226 with the frame index N, so that the change between the orientation information of the local image in the VR image 1224 with the frame index N-1 and the orientation information of the local image in the VR image 1222 with the frame index N-2 is the same as the change between the orientation information of the local image in the VR image 1226 with the frame index N and the orientation information of the local image in the VR image 1224 with the frame index N-1.

[0181] Similarly, the edge data network 2000 can predict the orientation information of the local image in the VR image 1228 with the frame index N+1 and the orientation information of the local image in the VR image 1229 with the frame index N+2, so that the change between the orientation information of the local image in the VR image 1228 with the frame index N+1 and the orientation information of the local image in the VR image 1226 with the frame index N is the same as the change between the orientation information of the local image in the VR image 1229 with the frame index N+2 and the orientation information of the local image in the VR image 1226 with the frame index N+1.

[0182] Fig.13 is a flowchart illustrating an example method performed by an edge data network to determine second predicted position information and to determine a second predicted partial image based on the second predicted position information according to various embodiments.

[0183] In operation S1310, the edge data network 2000 may determine the reproduction mode based on the position information of the partial image reproduced by the electronic device 1000 before the first partial image and the buffer management information of the reproduced partial image. For example, the edge data network 2000 may determine the reproduction mode based on the change between pieces of position information of partial images having two adjacent frame indices that have been reproduced before the first partial image and the buffer management information of the partial images having two frame indices.

[0184] In operation S1320, the edge data network 2000 may generate an image reproduction model based on the reproduction mode. For example, the image reproduction model may include a plurality of neural network layers, each of the plurality of neural network layers having a plurality of weight values, and performing a neural network operation by an operation between an operation result of a previous layer and the plurality of weight values.

[0185] Multiple weight values ​​of multiple neural network layers can be obtained or optimized through the training results of the artificial intelligence model. For example, multiple weight values ​​can be updated to reduce or minimize the loss value or cost value obtained by the artificial intelligence model during the training process. The artificial neural network may include a deep neural network (DNN) such as a convolutional neural network (CNN), a deep neural network (DNN), a recursive neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recursive deep neural network (BRDNN), or a deep Q network, but is not limited thereto.

[0186] In operation S1330, the edge data network 2000 may obtain second predicted orientation information from the image reproduction model by inputting the first orientation information to the image reproduction model. For example, the edge data network 2000 may train the image reproduction model based on the reproduction mode determined in operation S1310, and when orientation information in a VR image having a previous frame index is input, the image reproduction model may output orientation information in a VR image having a subsequent frame index or a frame index separated from the previous frame index by a predetermined interval.

[0187] Fig.14 is a flow chart illustrating an example method performed by an electronic device to stream video content obtained from an edge data network according to various embodiments.

[0188] In operation S1410, the electronic device 1000 may transmit the first position information to the edge data network 2000. For example, the first position information may include position information indicating the position of the first partial image in the first VR image, the first partial image being reproduced by the electronic device 1000. However, according to other example embodiments, the first position information may include a plurality of pieces of position information of partial images reproduced before the first partial image is reproduced by the electronic device 1000.

[0189] In operation S1420, the electronic device 1000 may obtain a second predicted frame corresponding to the second predicted orientation information determined based on the first orientation information from the edge data network 2000. For example, the second predicted frame obtained by the electronic device 1000 may be an I or P frame generated by encoding a second predicted partial image corresponding to the second predicted orientation information in the second VR image. Fig.14 Not shown in the figure, the electronic device 1000 may cache the second prediction frame obtained from the edge data network 2000 in a buffer of the electronic device 1000 .

[0190] In operation S1430, the electronic device 1000 may obtain the second orientation information corresponding to the second predicted frame. For example, the electronic device 1000 may obtain the second orientation information from the sensing module 210 in the electronic device 1000 at a time point when the electronic device 1000 reproduces the second predicted partial image (a time point before a preset time thereof), at a time point when a preset time has passed from a time point when the first partial image has been reproduced (a time point before a preset time thereof), or at a time point before a preset time from a time point when the second predicted partial image is predicted to be reproduced. However, according to other example embodiments, the electronic device 1000 may decode the second predicted frame, reproduce the second predicted partial image using the decoded second predicted frame, and obtain the second orientation information corresponding to the second predicted frame when reproducing the second predicted partial image. In operation S1440, the electronic device 1000 may send the second orientation information to the edge data network 2000. In operation S1450, the electronic device 1000 may obtain information about a comparison result indicating whether the second orientation information matches the second predicted orientation information from the edge data network 2000.

[0191] In operation S1460, the electronic device 1000 may receive a compensation frame from the edge data network 2000 or generate a compensation frame referring to at least some blocks in the second predicted frame based on whether the second orientation information matches the second predicted orientation information. For example, based on the information about the comparison result indicating that the second orientation information does not match the second predicted orientation information, the electronic device 1000 may wait for the reproduction of the second predicted partial image and obtain a compensation frame corresponding to the second partial image, which is received from the edge data network 2000 while waiting for the reproduction of the second predicted partial image. However, based on the information about the comparison result indicating that the second orientation information matches the second predicted orientation information, the electronic device 1000 may generate a compensation frame having the same image information as that of the second predicted frame.

[0192] In operation S1470, the electronic device 1000 may decode the compensation frame received from the edge data network 2000 or the compensation frame generated by the electronic device 1000 based on whether the second position information matches the second predicted position information.

[0193] For example, the electronic device 1000 may decode the compensated frame received from the edge data network 2000 based on the information about the comparison result indicating that the second orientation information does not match the second predicted orientation information. The electronic device 1000 may decode the compensated frame by referring to some blocks in the first frame and some blocks in the second predicted frame based on the reference block information in the compensated frame.

[0194] According to other example embodiments, the electronic device 1000 may decode the compensated frame generated by the electronic device 1000 based on the information about the comparison result indicating that the second orientation information matches the second predicted orientation information. Currently, the compensated frame generated by the electronic device 1000 includes the same image information as the image information of the second predicted frame, so the electronic device 1000 may decode the compensated frame by referring to only at least some blocks in the second predicted frame. In operation S1480, the electronic device 1000 may reproduce the decoded compensated frame.

[0195] Fig.15 is a diagram illustrating an example process performed by the electronic device 1000 to reproduce a compensating frame stored in a buffer in the electronic device 1000 according to various embodiments.

[0196] The electronic device 1000 may include at least one buffer 1502. For example, the electronic device 1000 may store I and P frames 1504 generated by encoding a partial image predicted by the edge data network 2000 in the at least one buffer 1502. When receiving information about a comparison result indicating that the second orientation information does not match the second predicted orientation information from the edge data network 2000, the electronic device 1000 may obtain a compensation frame 1512 from the edge data network 2000 in a state of waiting for reproduction of the second predicted partial image.

[0197] However, when receiving information about a comparison result indicating that the second position information matches the second predicted position information, the electronic device 1000 may generate a compensation frame 1514 having the same image information as that of the second predicted frame previously obtained from the edge data network 2000 .

[0198] The electronic device 1000 may cache the compensation frames 1512 and 1514 as the compensation frame 1506 in at least one buffer 1502 in which a prediction frame corresponding to a partial image predicted by the edge data network 2000 is stored based on the buffer management information.

[0199] The electronic device 1000 may decode the compensation frames 1512 and 1514 buffered in the at least one buffer 1502 by referring to at least some blocks in at least one I or P frame adjacent to the compensation frame to be decoded, and reproduce the video content using only the decoded compensation frames. The specific method of decoding the compensation frames with reference to the adjacent frames performed by the electronic device 1000 may correspond to Figure 4 Operation S428 or Figure 5 Operation S528 is performed, and therefore its detailed description is not repeated here.

[0200] The electronic device 1000 according to the present disclosure can decode I frames, P frames, and compensation frames by referring to I or P frames cached in a buffer. However, when decoding encoded frames received from the edge data network 2000, the electronic device 1000 according to the present disclosure can use only I or P frames, but when reproducing video content, the electronic device 1000 can use only compensation frames among the decoded frames to provide video content to the user. The electronic device 1000 can provide video content by transmitting only the decoded compensation frames to the display device in the electronic device 1000, so that only information about the real-time partial video content desired by the user of the electronic device 1000 can be effectively obtained from the edge data network 2000.

[0201] Fig.16 is a sequence diagram illustrating example operations among the cloud server 3000, the electronic device 1000, and the edge data network 2000 according to various embodiments.

[0202] According to various embodiments, the cloud server 3000, the electronic device 1000, and the edge data network 2000 may be executed. Figure 4 and Figure 5 The operation process between the electronic device 1000 and the edge data network 2000 in the embodiment of the present invention. In operation S1602, the electronic device 1000 may obtain first position information. Operation S1602 may correspond to Figure 4 In operation S402, the electronic device 1000 may transmit the obtained first position information to the cloud server 3000.

[0203] In operation S1606, the cloud server 3000 may determine the second predicted position information based on the first position information. The method for determining the second predicted position information based on the first position information performed by the cloud server 3000 may correspond to the method described above with reference to Fig.10 , Fig.11 , Fig.12 and Fig.13 The operation of determining the second predicted position information performed by the edge data network 2000 is described, and therefore its detailed description is not repeated here.

[0204] In operation S1608, the cloud server 3000 may determine a second predicted partial image corresponding to the second predicted orientation information. Operation S1608 may correspond to the above-mentioned Fig.10 , Fig.11 , Fig.12 and Fig.13 The operation of determining the second predicted local image corresponding to the second predicted orientation information performed by the edge data network 2000 is described, and therefore its detailed description is not repeated here.

[0205] In operation S1610, the cloud server 3000 may generate a second predicted frame by encoding the second predicted partial image. According to various embodiments, the cloud server 3000 may send the second predicted frame to the electronic device 1000 and also store the second predicted frame in the DB of the cloud server 3000. In operation S1612, the cloud server 3000 may send the second predicted frame to the electronic device 1000.

[0206] In operation S1614, the electronic device 1000 may cache the second predicted frame obtained from the cloud server 3000 in a buffer of the electronic device 1000, and obtain the second orientation information from the sensing module 210 in the electronic device 1000. Operation S1614 may correspond to Figure 4 In operation S414, the electronic device 1000 may transmit the second position information to the cloud server 3000.

[0207] In operation S1618, the cloud server 3000 may determine whether the second predicted orientation information matches the second orientation information. For example, when the average value of the difference between the second orientation information and the angle components (e.g., pitch, roll, and / or yaw) in the second predicted orientation information is greater than or equal to a preset threshold angle, the cloud server 3000 may determine that the second predicted orientation information does not match the second orientation information.

[0208] In operation S1620, the cloud server 3000 may transmit information about a comparison result indicating that the second predicted orientation information does not match the second orientation information to the edge data network 2000 and the electronic device 1000. In operation S1622, the edge data network 2000 may generate a compensation frame corresponding to the second partial image based on the first partial image, the second predicted partial image, and the second partial image. Operation S1622 may correspond to Figure 4 Operation S424 is described below, and therefore its detailed description is not repeated here.

[0209] In operation S1624, the electronic device 1000 may wait for the reproduction of the second predicted partial image based on the information about the comparison result indicating that the second predicted orientation information does not match the second orientation information. In operation S1626, the edge data network 2000 may send the generated compensation frame to the electronic device 1000. In operation S1628, the electronic device 1000 may decode the compensation frame and use the decoded compensation frame to reproduce the second partial image. For example, the electronic device 1000 may decode the compensation frame by referring to some blocks in the first frame corresponding to the first partial image and at least some blocks in the second predicted frame and pass the decoded compensation frame to the display device in the electronic device 1000, so that the second partial image is reproduced.

[0210] Fig.17 is a sequence diagram illustrating example operations among the cloud server 3000, the electronic device 1000, and the edge data network 2000 according to various embodiments.

[0211] The above referenced Fig.16 The operation process among the electronic device 1000, the cloud server 3000 and the edge data network 2000 when the second predicted position information does not match the second position information is described, and now reference will be made to Fig.17 To describe the operation process among the cloud server 3000, the electronic device 1000 and the edge data network 2000 when the second predicted position information matches the second position information. Fig.17 Operations S1702 to S1710 may correspond to Fig.16 Operations S1602 to S1610 are described in detail, and thus their detailed description is not repeated here.

[0212] In operation S1712, the cloud server 3000 may send the second predicted frame to the edge data network 2000 and the electronic device 1000. In operation S1714, the electronic device 1000 may cache the second predicted frame received from the cloud server 3000 in a buffer of the electronic device 1000 and obtain the second position information. Operation S1714 may correspond to Figure 4 In operation S414, the electronic device 1000 may transmit the second position information to the cloud server 3000.

[0213] In operation S1718, the cloud server 3000 may determine whether the second predicted orientation information matches the second orientation information. For example, when the average value of the difference between the second orientation information and the angle components (e.g., pitch, roll, and / or yaw) in the second predicted orientation information is greater than or equal to a preset threshold angle, the cloud server 3000 may determine that the second predicted orientation information matches the second orientation information.

[0214] In operation S1720, the cloud server 3000 may transmit information about a comparison result indicating that the second predicted orientation information matches the second orientation information to the edge data network 2000 and the electronic device 1000. In operation S1722, the edge data network 2000 may generate a compensation frame using the second predicted partial image. For example, the edge data network 2000 may generate a compensation frame having image information identical to that of the second predicted partial image.

[0215] In operation S1724, the electronic device 1000 may generate a compensation frame using the second predicted partial image. Currently, since it is determined that the second predicted orientation information matches the second orientation information, the electronic device 1000 may generate a compensation frame that refers to at least some blocks in the second predicted frame. Operation S1724 may correspond to Figure 5 Operation S524 is described below, and therefore its detailed description is not repeated here.

[0216] In operation S1726, the edge data network 2000 may discard the compensation frame generated by the edge data network 2000. In operation S1728, the electronic device 1000 may decode the compensation frame generated in operation S1724 and reproduce the decoded compensation frame. At present, the second prediction frame includes the same image information as the compensation frame, so the electronic device 1000 may decode the compensation frame by, for example, referring only to the second prediction frame. Operation S1728 may correspond to Figure 5 Operation S528 is performed, and therefore its detailed description is not repeated here.

[0217] Fig.18 is a block diagram illustrating an example of an electronic device 110 according to various embodiments.

[0218] Figures 1 to 17 The electronic device 1000 connected to the edge data network 2000 may correspond to Fig.18 For example, in the network environment 1700, the electronic device 110 may communicate with the electronic device 1702 through a first network 1798 (e.g., a short-range wireless communication network), or communicate with the electronic device 1704 or the server 1708 through a second network 1799 (e.g., a long-range wireless communication network). According to various embodiments, the electronic device 110 may communicate with the electronic device 1704 through the server 1708. According to various embodiments, the electronic device 110 may include a processor 1720, a memory 1730, an input device 1750, an acoustic output device 1755, a display device 1760, an audio module 1770, a sensor module 1776, an interface 1777, a haptic module 1779, a camera module 1780, a power management module 1788, a battery 1789, a communication module 1790, a user identification module 1796, and an antenna module 1797. In some example embodiments, in the electronic device 110, at least one of the components (e.g., the display device 1760 or the camera module 1780) may be omitted, or one or more other components may be additionally included. In some example embodiments, some components may be implemented as integrated circuits. For example, the sensor module 1776 (e.g., a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented as embedded in the display device 1760 (e.g., a display).

[0219] The processor 1720 may, for example, execute software (e.g., program 1740) to control at least one other component (e.g., hardware or software component) of the electronic device 110 connected to the processor 1720, and perform various data processing or calculations. According to various embodiments, as at least part of the data processing or calculation, the processor 1720 may load a command or data received from another component (e.g., sensor module 1776 or communication module 1790) in the volatile memory 1732, process the command or data stored in the volatile memory 1732, and store the resultant data in the non-volatile memory 1734.

[0220] According to various embodiments, the processor 1720 may execute the application client, and according to the execution of the application client, the processor 1720 may send a request message for checking the network protocol to be used by the application client to the edge data network 2000. In addition, the processor 1720 may receive a response message indicating the network protocol to be used by the application client from the edge data network 2000. The processor 1720 may update the network protocol to be used by the UE application based on the response message. The processor 1720 may select a network socket corresponding to the updated network protocol. The processor 1720 may receive data generated for the application client from the edge data network 2000 using the selected network socket.

[0221] According to various embodiments, the processor 1720 may include a main processor 1721 (e.g., a central processing unit (CPU) or an application processor (AP)) and an auxiliary processor 1723 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that operates independently of the main processor 1721 or operates together with the main processor 1721. Additionally or alternatively, the auxiliary processor 1723 may be configured to use lower power than the main processor 1721 or to be dedicated to a designated function. The auxiliary processor 1723 may be implemented independently of the main processor 1721 or as part of the main processor 1721.

[0222] The auxiliary processor 1723 may control at least some functions or states related to at least one of the components in the electronic device 110 (e.g., the display device 1760, the sensor module 1776, or the communication module 1790) instead of the main processor 1721 when the main processor 1721 is in an inactive state (e.g., a sleep state), or perform the above control together with the main processor 1721 when the main processor 1721 is in an active state (e.g., executing an application). According to various embodiments, the auxiliary processor 1723 (e.g., an ISP or a CP) may be implemented as a part of another component (e.g., a camera module 1780 or a communication module 1790) that is functionally associated with the auxiliary processor 1723.

[0223] The memory 1730 may store various data to be used for at least one component (e.g., the processor 1720 or the sensor module 1776) in the electronic device 110. The data may include, for example, software (e.g., program 1740) and input data or output data about commands associated with the software. The memory 1730 may include a volatile memory 1732 and a non-volatile memory 1734.

[0224] The program 1740 may be stored as software in the memory 1730 and may include, for example, an operating system (OS) 1742, middleware 1744, and applications 1746. According to various embodiments, the program 1740 may include, for example, Figure 1 The first application client 122 and the second application client 124. In addition, the program 1740 may include Figure 1 The edge enabler client 130 of the embodiment of the present invention.

[0225] The input device (including input circuit) 1750 may receive commands or data to be used for components in the electronic device 110 (e.g., the processor 1720) from outside the electronic device 110. The input device 1750 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus).

[0226] The acoustic output device 1755 can output an acoustic signal to the outside of the electronic device 110. The acoustic output device 1755 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or record playback, and the receiver can be used to receive incoming calls. According to various embodiments, the receiver can be separated from the speaker or as part of the speaker. The display device 1760 can visually provide information to the outside of the electronic device 110 (e.g., a user). The display device 1760 may, for example, include a display, a holographic device or a projector and a control circuit configured to control the corresponding device. According to various embodiments, the display device 1760 may include: a touch circuit configured to sense touch or a sensor circuit (e.g., a pressure sensor) configured to measure the strength of the force generated by touch.

[0227] The audio module 1770 may convert sound into an electrical signal, or vice versa. According to various embodiments, the audio module 1770 may obtain sound through the input device 1750 and output the sound through the acoustic output device 1755 or an external electronic device (e.g., electronic device 1702 (e.g., speaker or earphone)) directly or wirelessly connected to the electronic device 110.

[0228] The sensor module (including sensing circuit) 1776 can sense the operating state (e.g., power or temperature) or the external environment state (e.g., user state) of the electronic device 110, and generate an electrical signal or data value corresponding to the sensed state. According to various embodiments, the sensor module 1776 may include, for example, a gesture sensor, a gyroscope sensor, an inertial sensor, a barometer sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor (e.g., an RGB (red, green, blue) sensor), an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, an illumination sensor, or a sensor related to an autonomous vehicle (e.g., an inertial measurement unit (IMU), a global positioning system (GPS) sensor, a camera, a light image detection and ranging (LIDAR) sensor, and a radio detection and ranging (RADAR) sensor).

[0229] The interface (including interface circuit) 1777 may support one or more specified communication protocols that can be used to directly or wirelessly connect to an external electronic device (e.g., electronic device 1702) of the electronic device 110. The connection terminal 1778 may include a connector through which the electronic device 110 can be physically connected to the external electronic device (e.g., electronic device 1702). The tactile module 1779 may convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or electrical stimulus that a user can perceive through tactile or kinesthetic sensations. The camera module 1780 may capture still images or moving pictures. According to various embodiments, the camera module 1780 may include one or more cameras, lenses, image sensors, ISPs, or flashes.

[0230] The power management module 1788 may manage power to be supplied to the electronic device 110 .

[0231] The battery 1789 may supply power to at least one component in the electronic device 110 .

[0232] The communication module (including the communication circuit) 1790 can establish a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 110 and an external electronic device (e.g., the electronic device 1702, the electronic device 1704, or the server 1708), and support communication through the established communication channel. The communication module 1790 may include one or more communication processors that operate independently of the processor 1720 (e.g., AP) and support direct (e.g., wired) communication or wireless communication. According to various embodiments, the communication module 1790 may include: a wireless communication module (including a wireless communication circuit) 1792 (e.g., a cellular communication module, a short-range communication module, or a global navigation satellite system (GNSS) communication module), and a wired communication module (including a wired communication circuit) 1794 (e.g., a local area network (LAN) communication module or a power line communication module). A corresponding communication module among these communication modules can communicate with an external electronic device through a first network 1798 (e.g., a short-range communication network such as Bluetooth, direct Wi-Fi, or infrared data association) or a second network 1799 (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These types of communication modules can be integrated into one component (e.g., a single chip) or implemented by multiple separate components (e.g., multiple chips).

[0233] The wireless communication module 1792 may use user information (eg, an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 1796 to verify or authenticate the electronic device 110 in a communication network such as the first network 1798 or the second network 1799 .

[0234] The antenna module 1797 may transmit or receive a signal or power to or from the outside (eg, an external electronic device).

[0235] At least some components may be connected to each other in a peripheral device communication scheme (eg, bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and exchange signals (eg, commands or data) with each other.

[0236] According to various embodiments, a command or data may be transmitted or received between the electronic device 110 and the external electronic device 1704 via the server 1708 connected through the second network 1799. Each of the electronic devices 1702 and 1704 may be the same type as that of the electronic device 110 or a different type.

[0237] According to various embodiments, all or some operations performed by the electronic device 110 may be performed by one or more external devices among the electronic devices 1702 and 1704 and the server 1708. For example, when it is assumed that the electronic device 110 performs a certain function or service automatically or in response to a request from a user or another device, as a supplement or replacement for autonomously performing the function or service, the electronic device 110 may request one or more external electronic devices 1702 and 1704 to perform at least part of the function or service. The one or more external electronic devices 1702 and 1704 that have received the request may perform at least part of the requested function or service or additional functions or services related to the request, and pass the result of the execution to the electronic device 110. The electronic device 110 may process the result as it is or perform additional processing and provide the processing result as at least part of the response to the request. To this end, cloud computing, distributed computing, or client-server computing, for example, may be used.

[0238] The disclosed example embodiments provide a method capable of efficiently transmitting video content between an edge data network and an electronic device.

[0239] Although the present disclosure has been shown and described with reference to various exemplary embodiments, it will be understood that the various exemplary embodiments are intended to be illustrative rather than restrictive. It will be further understood by those skilled in the art that various changes in form and details may be made without departing from the true spirit and full scope of the present disclosure including the appended claims and their equivalents.

Claims

1. A method for sending video content performed by an edge data network, the method comprising: obtaining first location information from an electronic device connected to the edge data network; Determine second predicted position information and third predicted position information based on the first position information; Determine a second predicted partial image corresponding to the second predicted orientation information and a third predicted partial image corresponding to the third predicted orientation information; transmitting, to the electronic device, a second predicted frame generated by encoding the second predicted partial image and a third predicted frame generated by encoding the third predicted partial image; obtaining second position information corresponding to the second partial image from the electronic device; Determining third re-predicted position information based on the second position information; determining a third re-predicted partial image corresponding to the third re-predicted position information; comparing the third re-predicted position information with the third predicted position information; generating a compensation frame corresponding to the third re-predicted partial image based on the second predicted partial image, the third predicted partial image and the third re-predicted partial image based on a result of the comparison; and A compensation frame generated based on a result of the comparison is transmitted to the electronic device.

2. The method according to claim 1, wherein: The first partial image is a partial image in a first VR image with a first frame index of a virtual reality VR sequence including a plurality of frames, the second predicted partial image and the second partial image are partial images in a second VR image with a second frame index of the VR sequence, the third predicted partial image and the third re-predicted partial image are partial images in a third VR image with a third frame index of the VR sequence, and each of the first partial image, the second predicted partial image and the second partial image includes respective orientation information indicating a position of the corresponding partial image.

3. The method according to claim 2, wherein: Determining the second predicted partial image includes: determining the second predicted position information which is the same as the first position information; and Based on the determined second prediction orientation information, the second prediction partial image is determined.

4. The method according to claim 2, wherein: Determining the second predicted partial image includes: determining a first change between position information of a partial image reproduced by the electronic device before the first partial image and the first position information; determining the second predicted position information so that a second change between the second predicted position information and the first position information is the same as the first change; and Based on the determined second prediction orientation information, the second prediction partial image is determined.

5. The method according to claim 2, wherein: Determining the second predicted partial image includes: determining a reproduction mode based on position information of a partial image reproduced by the electronic device prior to the first partial image and buffer management information of the reproduced partial image; generating a user-specific image reproduction model based on the determined reproduction mode; obtaining the second predicted orientation information from the image reconstruction model by inputting the first orientation information into the generated image reconstruction model; and Based on the obtained second predicted orientation information, the second predicted partial image is determined.

6. The method according to claim 2, wherein: The second predicted frame includes an intra-frame coded frame or a predicted coded frame generated by encoding the second predicted partial image, the intra-frame coded frame is an I frame, and the predicted coded frame is a P frame.

7. The method according to claim 2, wherein: The second orientation information is obtained from the electronic device between a time point at which a preset time has elapsed from a time point at which the first partial image is reproduced by the electronic device and a time point at which the second predicted partial image or the second partial image is predicted to be reproduced by the electronic device.

8. The method according to claim 2, wherein: Comparing the third re-predicted orientation information with the third predicted orientation information includes: when a difference between an angle component in the third re-predicted orientation information and the third predicted orientation information is greater than or equal to a preset threshold, determining that the third re-predicted orientation information does not match the third predicted orientation information.

9. The method according to claim 2, wherein: Based on determining that the third re-prediction orientation information matches the third prediction orientation information, the compensated frame is generated by encoding the third predicted partial image.

10. The method according to claim 2, wherein: Based on determining that the third prediction orientation information does not match the third re-prediction orientation information, the compensated frame is generated using third difference information between the third re-prediction partial image and the second prediction partial image and fourth difference information between the third re-prediction partial image and the third prediction partial image.

11. An edge data network for transmitting video content to an electronic device, the edge data network comprising: Network interface; a memory storing one or more instructions; as well as a processor configured to execute the one or more instructions, The processor is further configured to: obtaining first location information from the electronic device connected to the edge data network, determining second predicted position information and third predicted position information based on the first position information, determining a second predicted partial image corresponding to the second predicted orientation information and a third predicted partial image corresponding to the third predicted orientation information, transmitting to the electronic device a second predicted frame generated by encoding the second predicted partial image and a third predicted frame generated by encoding the third predicted partial image, obtaining second position information corresponding to the second partial image from the electronic device, Based on the second position information, determining third re-predicted position information, determining a third re-predicted partial image corresponding to the third re-predicted position information, comparing the third re-predicted position information with the third predicted position information, generating a compensation frame corresponding to the third re-predicted partial image based on the second predicted partial image, the third predicted partial image and the third re-predicted partial image based on a result of the comparison, and A compensation frame generated based on a result of the comparison is transmitted to the electronic device.

12. The edge data network according to claim 11, wherein: The processor is further configured to, by executing the one or more instructions: determining a reproduction mode based on position information of a partial image reproduced by the electronic device before the first partial image and buffer management information of the reproduced partial image, generating a user-specific image reproduction model based on the determined reproduction mode, obtaining the second predicted orientation information from the image reconstruction model by inputting the first orientation information into the generated image reconstruction model; as well as Based on the obtained second predicted orientation information, the second predicted partial image is determined.

13. The edge data network according to claim 11, wherein: The processor is further configured to execute the one or more instructions to: generate the compensated frame by encoding the third predicted partial image based on determining that the third re-predicted orientation information matches the third predicted orientation information.

14. The edge data network according to claim 11, wherein: The processor is also configured to execute the one or more instructions to: based on determining that the third re-predicted orientation information does not match the third predicted orientation information, generate the compensated frame using third difference information between the third re-predicted partial image and the second predicted partial image and fourth difference information between the third re-predicted partial image and the third predicted partial image.

15. A non-transitory computer-readable recording medium having a program recorded thereon, the program being used to execute on a computer a method performed by an edge data network, the method comprising: obtaining first location information from an electronic device connected to the edge data network; Determine second predicted position information and third predicted position information based on the first position information; Determine a second predicted partial image corresponding to the second predicted orientation information and a third predicted partial image corresponding to the third predicted orientation information; transmitting, to the electronic device, a second predicted frame generated by encoding the second predicted partial image and a third predicted frame generated by encoding the third predicted partial image; obtaining second position information corresponding to the second partial image from the electronic device; Determining third re-predicted position information based on the second position information; determining a third re-predicted partial image corresponding to the third re-predicted position information; comparing the third re-predicted position information with the third predicted position information; generating a compensation frame corresponding to the third re-predicted partial image based on the second predicted partial image, the third predicted partial image and the third re-predicted partial image based on a result of the comparison; and A compensation frame generated based on a result of the comparison is transmitted to the electronic device.

Images