Electronic device and control method therefor

Abstract

This electronic device comprises: a communication interface connected to a wireless network; a memory for storing one or more instructions; and one or more processors connected to the communication interface and the memory so as to control the electronic device. The one or more processors execute the one or more instructions to: transmit content to an external device in real time through the communication interface; when transmission delay information is received from the external device, identify skip levels for a plurality of frames included in the content on the basis of the received transmission delay information and characteristics of the content; encode the content on the basis of the identified skip levels for the plurality of frames; and transmit information about the skip levels for the plurality of frames and the encoded content to the external device through the communication interface.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more particularly, to an electronic device for transmitting and receiving images via a wireless network and a method for controlling the same.

[0002] Advances in electronic technology have led to the development and proliferation of various types of electronic devices. In particular, the development and distribution of display devices such as TVs and mobile devices are rapidly progressing.

[0003] Broadcasting, OTT, cloud gaming, and mirroring services are services that transmit compressed and encoded video streams from the sender to the receiver for display. The receiver can receive data and requests from the user and transmit them to the sender, while the sender can provide services tailored to the received user data.

[0004] Meanwhile, compression encoding processes content in units of Group of Pictures (GOPs) with inter-frame dependencies. Therefore, when a delay occurs, older content can be discarded and the latest content can be decoded and displayed at the start of a new GOP. Therefore, delays can accumulate until at least one GOP ends, potentially resulting in significant delays. While setting the GOP to a smaller size can reduce delay accumulation, the frequent generation of intra-frames, which are the beginning of a GOP, can result in image quality degradation. Furthermore, setting the GOP to a larger size can result in content disruption as delayed content is discarded and a new GOP is displayed, potentially degrading service quality. Here, a GOP can refer to a video data unit that bundles several frames of screen data before and after it.

[0005] An electronic device according to one or more embodiments includes a communication interface connectable to a wireless network, a memory storing one or more commands, and one or more processors. The one or more processors transmit content to an external device in real time through the communication interface via the wireless network by executing the one or more commands, identify a plurality of skip levels per frame included in the content based on transmission delay information received from the external device and characteristics of the content, encode the content based on the identified plurality of skip levels per frame, and transmit the plurality of skip level information per frame and the encoded content to the external device via the wireless network via the communication interface.

[0006] According to one or more embodiments, the one or more processors, by executing the one or more commands, when a skippable frame is identified among a plurality of frames included in the content, encode the content so that the identified skippable frame is skippable when decoded by the external device, and transmit the encoded content, in which skip level information indicating whether each of the plurality of frames is skippable is included in one area of ​​each of the plurality of frames, to the external device through the wireless network via the communication interface.

[0007] According to one or more embodiments, the one or more processors can encode the content by excluding the identified skippable frame from the reference frame when a skippable frame is identified among a plurality of frames included in the content by executing the one or more instructions.

[0008] According to one or more embodiments, the one or more processors can identify a plurality of frame-by-frame skip levels based on at least one of data continuity, content type, or sound size when the content is audio content, and can identify a plurality of frame-by-frame skip levels based on at least one of scene change, content type, motion size, or frame rate when the content is video content, by executing the one or more instructions.

[0009] According to one or more embodiments, the one or more processors, by executing the one or more instructions, can identify the plurality of frame-by-frame skip levels so that the number of skippable frames is greater than or equal to a threshold number when the content is identified as first type content, and can identify the plurality of frame-by-frame skip levels so that the number of skippable frames is less than or equal to a threshold number when the content is identified as second type content.

[0010] According to one or more embodiments, the one or more processors may encode the content by skipping skippable frames based on the identified plurality of skip levels per frame when the electronic device and the external device are in a 1:1 transmission / reception situation by executing the one or more commands, and may encode the content so that the skippable frames are skippable when decoded by the external device based on the identified plurality of skip levels per frame when the electronic device and the external device are in a 1:N (N>1) transmission / reception situation.

[0011] According to one or more embodiments, an electronic device includes a communication interface connectable to a wireless network, a memory storing one or more commands, and one or more processors. The one or more processors, by executing the one or more commands, receive content in real time from an external device through the communication interface and decode the content through the wireless network, transmit transmission delay information to the external device through the wireless network based on a decoding delay state of the content through the communication interface, and, when the content including a plurality of skip level information per frame identified based on the transmission delay information and the characteristics of the content is received, decode the content including the plurality of skip level information per frame based on the identified plurality of skip level information per frame.

[0012] According to one or more embodiments, the one or more processors, by executing the one or more instructions, monitor the number of decoding delay frames to identify whether frame skipping is necessary, and if it is determined that frame skipping is necessary, identify skippable frames based on the plurality of frame-by-frame skip level information, and decode content including the received plurality of frame-by-frame skip level information by skipping the identified frames.

[0013] According to one or more embodiments, the one or more processors, by executing the one or more instructions, can transmit transmission delay information requesting additional skippable frames to the external device when it is determined that additional skippable frames are required based on the number of decoding delay frames and the plurality of frame-by-frame skip level information.

[0014] According to one or more embodiments, the one or more processors can, by executing the one or more instructions, identify audio frames and video frames to be finally skipped based on the plurality of frame-by-frame skip level information and synchronization of audio content and video content, and decode content including the received plurality of frame-by-frame skip level information by skipping the identified audio frames and the identified video frames.

[0015] According to one or more embodiments, a method for controlling an electronic device includes the steps of transmitting content to an external device in real time through a wireless network, identifying a plurality of skip levels per frame included in the content based on transmission delay information received from the external device and characteristics of the content, encoding the content based on the identified plurality of skip levels per frame, and transmitting the plurality of skip levels per frame and the encoded content to the external device through the wireless network.

[0016] According to one or more embodiments, a method for controlling an electronic device includes the steps of receiving and decoding content in real time from an external device through a wireless network, transmitting transmission delay information to the external device based on a decoding delay state of the content, and, when the content including a plurality of skip level information for each frame identified based on the transmission delay information and a characteristic of the content is received, decoding the received content based on the plurality of skip level information for each frame identified.

[0017] According to one or more embodiments, a non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, the operation comprising: transmitting content to an external device in real time through a wireless network; identifying a plurality of skip levels per frame included in the content based on transmission delay information received from the external device and a characteristic of the content; encoding the content based on the identified plurality of skip levels per frame; and transmitting the plurality of skip levels per frame and the encoded content to the external device through the wireless network.

[0018] According to one or more embodiments, a non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, the operation comprising: receiving and decoding content in real time from an external device through a wireless network; transmitting transmission delay information to the external device based on a decoding delay state of the content; and, when the content including a plurality of frame-by-frame skip level information identified based on the transmission delay information and a characteristic of the content is received, decoding the received content based on the identified plurality of frame-by-frame skip level information.

[0019] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings.

[0020] FIG. 1 is a drawing for explaining an implementation example of an electronic system according to one embodiment of the present disclosure.

[0021] FIG. 2 is a block diagram showing the configuration of an electronic device (100) according to one or more embodiments.

[0022] FIG. 3 is a flowchart for explaining a method of controlling an electronic device (100) according to one or more embodiments.

[0023] FIG. 4 is a flowchart for explaining a method of controlling an electronic device (100) according to one or more embodiments.

[0024] FIG. 5A is a diagram illustrating a method for identifying a skip level per frame according to one or more embodiments.

[0025] FIG. 5b is a diagram illustrating a method for identifying a skip level per frame according to one or more embodiments.

[0026] FIG. 5c is a diagram illustrating a method for identifying a skip level per frame according to one or more embodiments.

[0027] FIG. 6 is a flowchart for explaining a method of controlling an electronic device (100) according to one or more embodiments.

[0028] FIG. 7 is a flowchart for explaining a method of controlling an electronic device (100) according to one or more embodiments.

[0029] FIG. 8A is a block diagram specifically illustrating the configuration of an electronic device (200) according to one or more embodiments.

[0030] FIG. 8b is a block diagram illustrating a detailed configuration of an electronic device (200) according to one or more embodiments.

[0031] FIG. 9 is a flowchart for explaining a method of controlling an electronic device (200) according to one or more embodiments.

[0032] FIG. 10 is a flowchart for explaining a method of controlling an electronic device (200) according to one or more embodiments.

[0033] FIG. 11 is a flowchart for explaining a method of controlling an electronic device (200) according to one or more embodiments.

[0034] FIG. 12 is a drawing for explaining the operation between a transmitting device (100) and a receiving device (200) according to one or more embodiments.

[0035] FIG. 13 is a diagram illustrating operation in unicast mode according to one or more embodiments.

[0036] FIG. 14 is a diagram for explaining operation in multicast mode according to one or more embodiments.

[0037] FIG. 15a is a diagram illustrating a compression mode identification method according to one or more embodiments.

[0038] FIG. 15b is a diagram illustrating a compression mode identification method according to one or more embodiments.

[0039] FIG. 15c is a diagram illustrating a compression mode identification method according to one or more embodiments.

[0040] FIG. 15d is a diagram illustrating a compression mode identification method according to one or more embodiments.

[0041] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0042] The terms used in the embodiments of this disclosure may vary depending on current widespread use, the emergence of new technologies, and other factors, taking into account the functions described herein. Furthermore, in certain cases, the applicant may arbitrarily select terms, in which case their meanings will be described in the relevant description of the disclosure. Therefore, the terms used in this disclosure should not be defined simply as names, but rather based on their meaning and the overall content of the disclosure.

[0043] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0044] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to cases where (1) only A is included, (2) only B is included, or (3) both A and B are included.

[0045] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0046] When it is said that a component (e.g., a first component) is “operatively or communicatively coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0047] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0048] In some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0049] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0050] In the embodiments, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of "modules" or "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "module" or "part" that needs to be implemented as specific hardware.

[0051] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0052] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0053] FIG. 1 is a drawing for explaining an implementation example of an electronic system according to one embodiment of the present disclosure.

[0054] According to FIG. 1, the electronic system may include a transmitting device (100), a receiving device (200), and a source device (300).

[0055] The source device (300) is a device that provides content and can be implemented as a server, game console, PC, etc. As an example, the transmission device (100) can provide a real-time video streaming service, and the real-time video streaming service can include a live broadcasting service, a cloud gaming service, a mirroring service, a video communication service, etc.

[0056] The transmitting device (100) can be implemented as a set-top box or an oc (one connected) box as shown in FIG. 1, but is not limited thereto, and can be applied to any device having image processing and communication functions.

[0057] The receiving device (200) can be implemented as a TV as shown in FIG. 1, but is not limited thereto, and can be applied to any device having communication and display functions, such as a monitor, a smart phone, a tablet PC, a notebook PC, a HMD (Head mounted Display), a NED (Near Eye Display), a LFD (large format display), a digital signage, a DID (Digital Information Display), a video wall, a projector display, etc.

[0058] The receiving device (200) can receive content from the transmitting device (100). For example, the content can include at least one of video content or audio content. For example, in the case of video content, various compressed images or images of various resolutions can be received. For example, the receiving device (200) can receive images in a compressed form such as MPEG (Moving Picture Experts Group) (e.g., MP2, MP4, MP7, etc.), JPEG (joint photographic coding experts group), AVC (Advanced Video Coding), H.264, H.265, HEVC (High Efficiency Video Codec), etc. Alternatively, the electronic device (100) can receive any one of SD (Standard Definition), HD (High Definition), Full HD, Ultra HD, or higher resolution images.

[0059] In one example, a transmitting device (100) and a receiving device (200) may communicate via a wireless network. For example, the transmitting device (100) may compress a video signal received from a source device (300), insert necessary information into a header, and transmit it to the receiving device (200) via a wireless network.

[0060] Meanwhile, wireless networks providing streaming services can experience deterioration, such as temporary transmission failures or reduced bandwidth. When network deterioration occurs in real-time streaming services, transmission delays accumulate, disrupting the proper operation of real-time services. Therefore, when transmission delays occur, content corresponding to the delay must be discarded and newer content provided to prevent further delays.

[0061] Accordingly, the following describes various embodiments for minimizing transmission delay while maintaining quality according to network conditions and content characteristics in low-latency streaming services such as real-time streaming, and for recovering to a minimum delay situation as quickly as possible when delay occurs.

[0062] FIG. 2 is a block diagram showing the configuration of an electronic device (100) according to one or more embodiments.

[0063] According to an example, the electronic device (100) may be implemented as a transmitting device (100) illustrated in FIG. 1. For example, the electronic device (100) may be implemented as a set-top box or an oc (one connected) box, but is not limited thereto, and may be applied to any device having image processing and communication functions.

[0064] According to FIG. 2, the electronic device (100) includes a communication interface (110), a memory (120), and one or more processors (130).

[0065] It goes without saying that the communication interface (110) can be implemented as various interfaces depending on the implementation example of the electronic device (100). For example, the communication interface (110) can support wireless communication such as Bluetooth, AP-based Wi-Fi (Wireless LAN network), Zigbee, Wireless LAN (Local Area Network), WAN (Wide Area Network), etc. However, it is not limited thereto, and it can also support wired communication such as Ethernet, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), Optical, Coaxial, etc. According to one example, the communication interface (110) can communicate with other electronic devices, external servers, and / or remote control devices.

[0066] The memory (120) can store data required for various embodiments. Depending on the purpose of data storage, the memory (120) may be implemented as a memory embedded in the electronic device (100) or as a memory detachable from the electronic device (100). For example, data for operating the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding the functions of the electronic device (100) may be stored in a memory detachable from the electronic device (100). Meanwhile, in the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD). In addition, in the case of memory that can be attached or detached to the electronic device (100'), it may be implemented as at least one of memory cards (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. It can be implemented in the form of.

[0067] One or more processors (130) control the overall operation of the electronic device (100). Specifically, one or more processors (130) may be connected to each component of the electronic device (100) to control the overall operation of the electronic device (100). For example, one or more processors (130) may be electrically connected to the display (130) and the memory (120) to control the overall operation of the electronic device (100). The processor (130) may be composed of one or more processors.

[0068] One or more processors (130) may perform operations of the electronic device (100) according to various embodiments by executing at least one instruction stored in the memory (120).

[0069] The one or more processors (130) may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors (130) may control one or any combination of other components of the electronic device, and may perform operations related to communication or data processing. The one or more processors (130) may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to one or more embodiments of the present disclosure by executing one or more instructions stored in a memory.

[0070] When a method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0071] One or more processors (130) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (130) are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure.

[0072] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0073] In the embodiments of the present disclosure, a processor may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. Hereinafter, for the convenience of explanation, one or more processors (130) will be referred to as a processor (130).

[0074] According to one embodiment, the processor (130) can transmit content to an external device in real time via the communication interface (110). For example, the content can include at least one of video content or audio content.

[0075] According to one embodiment, when transmission delay information is received from an external device (e.g., FIG. 1, 200), the processor (130) may identify a plurality of frame-by-frame skip levels included in the content based on the received transmission delay information and the characteristics of the content.

[0076] For example, transmission delay information may include the number of delayed frames, which is delay information due to the deterioration of the wireless network. For example, the number of delayed frames may include at least one of the number of audio delayed frames or the number of video delayed frames. However, the present invention is not limited thereto, and the transmission delay information may be level information that grades the number of delayed frames. In other words, the transmission delay information may be various types of information that can indicate the low-delay requirement of the content.

[0077] For example, content characteristics may differ depending on whether the content is audio or video. For example, for audio content, the content characteristics may include at least one of data continuity, content type, or sound volume. Furthermore, for video content, the content characteristics may include at least one of scene change, content type, motion volume, or frame rate. Here, the content type may indicate not only the content provision form, such as real-time streaming content or OTT (content type) content, but also the content genre, such as game content or movie content. Furthermore, even within a single game content, there may be parts that require low latency because the user directly plays them (i.e., non-skippable parts) and parts that do not require low latency because they are intended for viewing (i.e., skippable parts). Therefore, whether frames can be skipped may be determined on a chapter-by-chapter or scene-by-scene basis.

[0078] For example, if the content is audio content, the processor (130) may identify multiple frame-by-frame skip levels based on at least one of data continuity, content type, or sound size. Additionally, if the content is video content, the processor (130) may identify multiple frame-by-frame skip levels based on at least one of scene change, content type, motion size, or frame rate.

[0079] For example, if the processor (130) identifies the content as first type content, it may identify multiple skip levels per frame so that the number of skippable frames is greater than or equal to a threshold number. For example, the first type content may be content with a high level of low latency requirements, such as game content.

[0080] For example, if the processor (130) identifies the content as second type content, it may identify multiple skip levels per frame so that the number of skippable frames is less than a threshold number. For example, the second type content may be content with a low level of low latency requirements, such as movie content.

[0081] For example, multiple frame-by-frame skip levels may be information indicating whether skipping is possible during decoding by the receiving device (200). For example, the skip level may have a value of "1" if skippable and "0" if non-skippable, but is not necessarily limited thereto. For example, even if skippable, different numbers may be indicated depending on the priority.

[0082] According to one embodiment, the processor (130) may encode content based on the skip levels of the identified plurality of frames. For example, if a skippable frame is identified among a plurality of frames included in the content, the processor (130) may encode the content so that the identified frame is skippable during decoding. For example, if a skippable frame is identified among a plurality of frames included in the content, the processor (130) may encode the content by excluding the identified frame from the reference frame. Here, the reference frame may be a frame that serves as a reference when performing a block matching algorithm to estimate motion while compressing a video. That is, the processor (130) may encode the content so that a skippable frame is excluded from the reference frame, so that even if the corresponding frame is skipped during decoding, no problem occurs in motion estimation.

[0083] According to one embodiment, the processor (130) may transmit skip level information (or skip flag) for each of a plurality of frames and encoded content to the receiving device (200) via the communication interface (110). For example, skip level information indicating whether skipping is possible for each of a plurality of frames may be included in an area of ​​each of the plurality of frames. For example, the skip level information may be included in a header area of ​​a compressed video packet corresponding to each of the plurality of frames. In addition, video information such as a video size and a video compression rate may be included in the header area of ​​a compressed video packet corresponding to each of the plurality of frames.

[0084] According to one embodiment, the processor (130) can control encoding based on streaming mode.

[0085] For example, when the electronic device (100) and the receiving device (200) are in a 1:1 transmission / reception mode (e.g., unicast mode), the processor (130) may encode content by skipping skippable frames based on the identified multiple frame-by-frame skip levels. For example, the processor (130) may control the encoder to perform encoding by directly performing frame skipping based on the number of delayed frames included in the transmission delay information. For example, when the current streaming mode is unicast mode, performing skipping in the encoder can reduce the transmission amount and the work of the decoder.

[0086] For example, when the electronic device (100) and the receiving device (200) are in a 1:N transmission / reception mode (e.g., multicast / broadcast mode), the processor (130) may encode content so that skippable frames are skippable during decoding at the receiving device (200) based on the identified skip levels for each frame. For example, the processor (130) may set the skip levels for each frame so that the decoder can skip as many delayed frames as are included in the transmission delay information. For example, although encoder skip selection is possible even in multicast / broadcast mode, there may be other client devices that do not require skipping, so performing skipping at the decoder may be advantageous for the overall quality. For example, when a plurality of frames are set to a skippable level, consecutive frames or non-consecutive frames may be set to a skippable level.

[0087] FIG. 3 is a flowchart for explaining a method of controlling an electronic device (100) according to one or more embodiments.

[0088] According to FIG. 3, the electronic device (100) can transmit content to the receiving device (200) in real time via a wireless network (S310).

[0089] Next, the electronic device (100) can identify whether transmission delay information is received from the receiving device (200) (S320).

[0090] When the electronic device (100) receives transmission delay information from the receiving device (200) (S320:Y), the electronic device (100) can identify a plurality of frame-by-frame skip levels included in the content based on the received transmission delay information and the characteristics of the content (S330).

[0091] Next, the electronic device (100) can encode content based on the identified plurality of frame-by-frame skip levels (S340).

[0092] Next, the electronic device (100) can transmit multiple frame-by-frame skip level information and encoded content to an external device (S350).

[0093] Meanwhile, in Fig. 3, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0094] FIG. 4 is a flowchart for explaining a method of controlling an electronic device (100) according to one or more embodiments.

[0095] According to FIG. 4, the electronic device (100) can transmit content to the transmission device (100) in real time via a wireless network (S410).

[0096] Next, the electronic device (100) can identify whether transmission delay information is received from the receiving device (200) (S420).

[0097] When the electronic device (100) receives transmission delay information from the receiving device (200) (S420:Y), it can identify whether the content is audio content (S430).

[0098] If the content is identified as audio content (S430:Y), the electronic device (100) can identify multiple frame-by-frame skip levels included in the audio content based on the received transmission delay information and a first criterion (S440). For example, the first criterion may include at least one of data continuity, content type, or sound volume.

[0099] If the electronic device (100) identifies that the content is not audio content (S430:N), it can identify whether the content is video content (S450).

[0100] If the content is identified as video content (S450:Y), the electronic device (100) can identify multiple frame-by-frame skip levels included in the video content based on the received transmission delay information and a second criterion (S460). For example, the second criterion may include at least one of scene change, content type, motion size, or frame rate.

[0101] Next, the electronic device (100) can encode the content so that the skippable frames are skippable when decoded by the receiving device (200) based on the identified plurality of frame-by-frame skip levels (S470).

[0102] Next, the electronic device (100) can transmit multiple frame-by-frame skip level information and encoded content to the receiving device (200) (S480).

[0103] Meanwhile, in Fig. 4, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0104] FIGS. 5A to 5C are drawings for explaining a frame-by-frame skip level identification method according to one or more embodiments.

[0105] For example, the electronic device (100) may identify a frame-by-frame skip level based on at least one of data continuity, content type, or sound size when the content is audio content.

[0106] For example, as illustrated in FIG. 5A, the electronic device (100) may identify a frame corresponding to a mute section in audio content as skippable. For example, the electronic device (100) may encode content by setting a skippable level for a frame identified as skippable and excluding the frame from reference frames to enable skipping during decoding.

[0107] For example, if the content is video content, the electronic device (100) may identify a frame-by-frame skip level based on at least one of scene change, content type, motion size, or frame rate. In this case, the receiving device (200) may determine whether to skip a corresponding frame during decoding based on the frame-by-frame skip level.

[0108] For example, as illustrated in FIG. 5B, the electronic device (100) may identify a frame corresponding to a mute section in video content as skippable, or identify a frame corresponding to a section with a large scene change as non-skippable. For example, the electronic device (100) may encode the content by setting a skippable level for a frame identified as skippable and excluding the frame from the reference frame so that it is skippable during decoding. Additionally, the electronic device (100) may set a non-skippable level for a frame identified as non-skippable.

[0109] For example, as illustrated in FIG. 5c, the electronic device (100) may identify frames corresponding to sections with large motion magnitudes in video content as non-skippable. Additionally, the electronic device (100) may set a non-skippable level for frames identified as non-skippable.

[0110] FIG. 6 is a flowchart for explaining a method of controlling an electronic device (100) according to one or more embodiments.

[0111] According to FIG. 6, the electronic device (100) can transmit content to the receiving device (200) in real time via a wireless network (S610).

[0112] When the electronic device (100) receives transmission delay information from the receiving device (200) (S620), it can identify whether the content is first type content (S630).

[0113] If the electronic device (100) identifies the content as the first type content (S630:Y), the electronic device (100) can identify a plurality of skip levels for each frame so that the number of skippable frames is greater than or equal to a threshold number based on the received transmission delay information (S640). For example, the first type content may be content with a high level of low-latency requirements, such as game content. In this case, the electronic device can prevent low-latency due to network degradation by skipping as many frames as possible during decoding. For example, the electronic device (100) can identify the skip level of each frame so that as many frames as possible are skipped during decoding based on the number of delayed frames included in the transmission delay information.

[0114] If the electronic device (100) identifies that the content is not the first type content (S630:N), it can identify whether the content is the second type content (S650).

[0115] If the electronic device (100) identifies the content as second type content (S650:Y), it can identify multiple skip levels per frame based on the received transmission delay information so that the number of skippable frames is less than a threshold number (S660). For example, the second type content may be content with a low level of low delay requirements, such as movie content. Since this is a case where low delay is acceptable, it can be configured to skip as few frames as possible during decoding.

[0116] Next, the electronic device (100) may encode the content so that the skippable frames are skippable when decoded by the receiving device (200) based on the identified plurality of frame-by-frame skip levels (S670). For example, the electronic device (100) may encode the content so that the skippable frames are not included in the reference frame.

[0117] Next, the electronic device (100) can transmit multiple frame-by-frame skip level information and encoded content to the receiving device (200) (S680).

[0118] Meanwhile, in Fig. 6, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0119] FIG. 7 is a flowchart for explaining a method of controlling an electronic device (100) according to one or more embodiments.

[0120] According to FIG. 7, the electronic device (100) can transmit content to the receiving device (200) in real time via a wireless network (S710).

[0121] When the electronic device (100) receives transmission delay information from the receiving device (200) (S720:Y), the electronic device (100) can identify a plurality of frame-by-frame skip levels included in the content based on the received transmission delay information and the characteristics of the content (S730).

[0122] Next, when the electronic device (100) and the receiving device (200) are identified as a 1:1 transmission / reception situation (S740:Y), the electronic device (100) can encode the content by skipping skippable frames based on the identified multiple frame-by-frame skip levels (S750).

[0123] Next, the electronic device (100) can transmit the encoded content to the receiving device (200) (S760).

[0124] Meanwhile, if the electronic device (100) and the receiving device (200) are identified as being in a 1:1 transmission / reception situation, the electronic device (100) can identify whether it is a 1:N transmission / reception situation (S770).

[0125] When the electronic device (100) and the receiving device (200) are identified as a 1:N transmission / reception situation (S770:Y), the electronic device (100) can encode content so that skippable frames can be skipped when decoded by an external device based on the identified multiple frame-by-frame skip levels (S780).

[0126] Next, the electronic device (100) can transmit multiple frame-by-frame skip level information and encoded content to the receiving device (200) (S790).

[0127] Meanwhile, in Fig. 7, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0128] FIG. 8A is a block diagram specifically illustrating the configuration of an electronic device (200) according to one or more embodiments.

[0129] According to an example, the electronic device (200) can be implemented as the receiving device (200) illustrated in FIG. 1. For example, the electronic device (200) can be implemented as a TV, but is not limited thereto, and can be applied to any device having communication and display functions, such as a monitor, a smart phone, a tablet PC, a notebook PC, a head mounted display (HMD), a near eye display (NED), a large format display (LFD), a digital signage, a digital information display (DID), a video wall, a projector display, etc.

[0130] According to FIG. 8a, the electronic device (200) includes a communication interface (210), a memory (220), and one or more processors (230). Implementation examples of the communication interface (210), the memory (220), and the one or more processors (230) are identical / similar to the communication interface (110), the memory (120), and the one or more processors (130) illustrated in FIG. 2, and therefore, a detailed description thereof will be omitted.

[0131] According to one embodiment, the processor (230) can receive and decode content in real time from the transmitting device (100) via the communication interface (210).

[0132] According to one embodiment, the processor (230) may transmit transmission delay information to the transmitting device (100) based on the decoding delay status of the content.

[0133] According to one embodiment, when content including a plurality of skip level information per frame identified based on transmission delay information and content characteristics is received, the processor (230) may decode the received content based on the identified plurality of skip level information per frame. In one example, the plurality of skip levels per frame may be information indicating whether skipping is possible during decoding. For example, the skip level may have a value of "1" if skippable and "0" if non-skippable, but is not necessarily limited thereto. For example, even if skippable, different numerical values ​​may be indicated depending on the priority.

[0134] According to one embodiment, the processor (230) can monitor the number of decoding delay frames to determine whether frame skipping is necessary. If the processor (230) determines that frame skipping is necessary, the processor (230) can identify skippable frames based on skip level information for each frame, and decode the received content by skipping the identified frames.

[0135] For example, the processor (230) can identify the number of decoding delay frames based on at least one of a stream buffer state in which a stream received from a transmitting device (100) is stored or an output buffer state in which a frame output through a display (240) is stored.

[0136] According to one embodiment, the processor (230) may identify the number of decoding delay frames based on the output mode. For example, the output mode may include at least one of a Variable Refresh Rate (VRR) mode, a Frame Rate Control (FRC) mode, and a Normal mode.

[0137] According to one embodiment, if the processor (230) identifies that additional skippable frames are required based on the number of decoding delay frames and the plurality of frame-by-frame skip level information, the processor (230) may transmit transmission delay information requesting additional skippable frames to the transmitting device (100).

[0138] According to one embodiment, the processor (230) may identify audio frames and video frames to be ultimately skipped based on a plurality of frame-by-frame skip level information and synchronization of audio content and video content. Subsequently, the processor (230) may decode the received content by skipping the identified audio frames and the identified video frames.

[0139] According to one embodiment, the processor (140) may calculate a decoding speed of the decoder and / or an output speed and / or an output sequence of the display (120) based on the output buffer status and the output mode, and control the decoder and / or the display (120) based thereon.

[0140] For example, if the processor (140) identifies that the stream stored in the stream buffer is less than the minimum threshold value (S430:Y), the processor (140) may control at least one of the frame output rate, frame interpolation, or frame repetition output based on the output buffer status and the output mode (S440). Here, the output mode may include at least one of the VRR (Variable Refresh Rate) mode, the FRC (Frame Rate Control) mode, and the Nomal mode.

[0141] According to one example, the processor (230) can finally identify the transmission delay information to be requested from the transmitting device (100) based on the number of decoding delay frames and the network status information. Here, the network status information may include at least one of the available bandwidth of the network, packet loss rate, the Round-Trip Time of the packet, the Delay-Gradient of the packet, RSSI (Received signal strength indicator) information, communication channel information, link speed information, channel interference information, or retry rate information. In other words, the network status information may be various network-related information that is affected by network congestion, distance, and transmission speed. The Round-Trip Time (or Round-Trip Delay) of the packet may refer to the time it takes for the packet to travel back and forth from the network to the receiving end. The Delay-Gradient of the packet may be the difference between the transmission time interval of packets transmitted from the transmitting end and the reception time interval of packets received at the receiving end. The communication channel information may include information on at least one of a channel and a frequency. The link speed information may be information on the rate of packets transmitted from the transmitting end to the receiving end. Channel interference information can be obtained based on the interference factor for each channel measured using various channel RF parameters, such as channel active time, channel busy time, and channel transmit time. Retry rate information can indicate the percentage of transmitted packets that are retried in seconds.

[0142] The processor (230) may perform image processing including at least one of image enhancement, image restoration, image transformation, image analysis, or image understanding on the decoded frame as needed and output the image.

[0143] FIG. 8b is a block diagram illustrating a detailed configuration of an electronic device (200) according to one or more embodiments.

[0144] According to FIG. 8b, the electronic device (200') may include a communication interface (210), a memory (220), one or more processors (230), a display (240), a user interface (250), a camera (260), a speaker (270), and a sensor (280). Among the configurations illustrated in FIG. 8b, a detailed description of configurations that overlap with those illustrated in FIG. 8a will be omitted.

[0145] The display (240) may be implemented as a display including a self-luminous element or a display including a non-luminous element and a backlight. For example, it may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display (240) may also include a driving circuit, a backlight unit, etc., which may be implemented in a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. According to an example, a touch sensor that detects a touch operation in the form of a touch film, a touch sheet, a touch pad, etc. may be disposed on the front of the display (240) so as to be implemented so as to detect various types of touch inputs. For example, the display (240) can detect various types of touch inputs, such as a touch input by a user's hand, a touch input by an input device such as a stylus pen, and a touch input by a specific electrostatic material. Here, the input device can be implemented as a pen-type input device that can be referred to by various terms such as an electronic pen, a stylus pen, an S-pen, etc. According to an example, the display (240) can be implemented as a flat display, a curved display, a flexible display that can be folded or / and rolled, etc. According to an example, the processor (230) can display an acquired output image through the display (240).

[0146] For example, the processor (230) may control the display (240) to output a video frame based on a video signal restored from a compressed video stream and a sync signal corresponding to the video frame. For example, the processor (230) may generate a Vsync signal based on an identified frame refresh rate and output a video frame based on the generated Vsync signal.

[0147] The user interface (250) may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen that can also perform the display function and operation input function described above.

[0148] The camera (260) can be turned on and take pictures according to a preset event. The camera (260) can convert the captured image into an electrical signal and generate image data based on the converted signal. For example, the subject can be converted into an electrical image signal through a semiconductor optical element (CCD; Charge Coupled Device), and the converted image signal can be amplified and converted into a digital signal and then signal processed. For example, the camera (120) can be implemented as a general camera, a stereo camera, a depth camera, etc.

[0149] The speaker (270) may be configured to output various audio data as well as various notification sounds or voice messages. The processor (230) may control the speaker (270) to output feedback or various notifications in audio format according to various embodiments of the present disclosure.

[0150] The sensor (280) may include various types of sensors such as a touch sensor, a proximity sensor, an acceleration sensor (or a gravity sensor), a geomagnetic sensor, a gyro sensor, a pressure sensor, a position sensor, a distance sensor, a light sensor, etc.

[0151] In addition, the electronic device (200') may include a microphone (not shown), a tuner (not shown), and a demodulator (not shown) depending on the implementation example.

[0152] A microphone (not shown) is configured to receive user voice or other sounds and convert them into audio data. However, according to another embodiment, the electronic device (200') may receive user voice input via an external device through a communication interface (210).

[0153] A tuner (not shown) can receive RF broadcast signals by tuning to a channel selected by a user or all previously stored channels among RF (Radio Frequency) broadcast signals received through an antenna.

[0154] The demodulator (not shown) receives and demodulates a digital IF signal (DIF) converted from the tuner, and can also perform channel decoding, etc.

[0155] FIG. 9 is a flowchart for explaining a method of controlling an electronic device (200) according to one or more embodiments.

[0156] According to FIG. 9, the electronic device (200) can receive content in real time from the transmitting device (100) via a wireless network (S910). For example, the content may include at least one of video content and audio content. For example, the electronic device (200) can receive a compressed stream of at least one of the video content and audio content in a compressed form from the transmitting device (100) in a streaming format.

[0157] Next, the electronic device (200) can decode the received content (S920). For example, the electronic device (200) can decode the received compressed stream using a decoder. For example, the electronic device (200) can decode the received compressed stream based on compression information included in the header of the compressed stream. For example, the electronic device (200) can obtain a frame by decoding each compressed packet based on compression information included in the header of each compressed packet included in the received compressed stream. For example, the header area of ​​a compressed video packet corresponding to each of a plurality of video frames may include compression information such as an image size and an image compression rate.

[0158] Next, the electronic device (200) may transmit transmission delay information to the transmitting device (100) based on the decoding delay status of the content (S930). For example, the transmission delay information may include the number of delayed frames as delay information due to deterioration of the wireless network. For example, the number of delayed frames may include at least one of the number of audio delayed frames or the number of video delayed frames. However, the present invention is not limited thereto, and the transmission delay information may be level information that grades the number of delayed frames. In other words, the transmission delay information may be various types of information that may indicate a low-delay requirement of the content.

[0159] Next, the electronic device (200) may receive content including a plurality of frame-by-frame skip level information (S940). For example, the plurality of frame-by-frame skip level information may be information identified based on the characteristics of the content and transmission delay information received by the transmitting device (200). For example, the content characteristics may differ depending on whether the content is audio content or video content. For example, in the case of audio content, the content characteristics may include at least one of data continuity, content type, or sound size. In addition, in the case of video content, the content characteristics may include at least one of scene change, content type, motion size, or frame rate.

[0160] Next, the electronic device (200) can decode the received content based on skip level information for each of the plurality of frames (S950). For example, the skip levels for each of the plurality of frames can be information indicating whether skipping is possible during decoding, and can be included in the hair area of ​​the compressed packet corresponding to each frame. For example, if the skip level has a value of "1" if skippable and "0" if not skippable, the electronic device (200) can perform content decoding by skipping the frame with the skip level of "1".

[0161] Meanwhile, in Fig. 9, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0162] FIG. 10 is a flowchart for explaining a method of controlling an electronic device (200) according to one or more embodiments.

[0163] Among the operations illustrated in Fig. 10, detailed descriptions of operations that overlap with those illustrated in Fig. 9 will be omitted.

[0164] According to FIG. 10, the electronic device (200) can receive content in real time from the transmitting device (100) via a wireless network (S1005).

[0165] Next, the electronic device (200) can decode the received content (S1010).

[0166] Next, the electronic device (200) can transmit transmission delay information to the transmitting device (100) based on the decoding delay status of the content (S1015).

[0167] Next, the electronic device (200) can receive content including skip level information for a plurality of frames (S1020).

[0168] Next, the electronic device (200) can monitor the number of decoding delay frames to identify whether frame skipping is necessary (S1025). If the electronic device (200) determines that frame skipping is necessary (S1030:Y), it can identify skippable frames based on skip level information for each frame (S1035).

[0169] Next, the electronic device (200) can decode the received content by skipping the identified frame (S1040). For example, if the skip level has a value of "1" when skippable and "0" when non-skippable, the electronic device (200) can perform content decoding by skipping the frame with the skip level of "1."

[0170] Next, the electronic device (200) can identify whether additional skippable frames are required based on the number of decoding delay frames and the skip level information for each of the plurality of frames (S1045). If the electronic device (200) identifies whether additional skippable frames are required (S1045:Y), the electronic device (200) can transmit transmission delay information requesting additional skippable frames to the transmitting device (200) (S1050). For example, the electronic device (200) can compare the number of decoding delay frames and skippable frames in real time, and if the number of decoding delay frames is greater than the number of skippable frames, the electronic device (200) can transmit transmission delay information requesting additional skippable frames to the transmitting device (200).

[0171] Meanwhile, in Fig. 10, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0172] FIG. 11 is a flowchart for explaining a method of controlling an electronic device (200) according to one or more embodiments.

[0173] Among the operations illustrated in Fig. 11, detailed descriptions of operations that overlap with those illustrated in Figs. 9 and 10 will be omitted.

[0174] According to FIG. 10, the electronic device (200) can receive content in real time from the transmitting device (100) via a wireless network and decode the received content (S1010). For example, the electronic device (200) can receive a compressed stream containing both video content and audio content.

[0175] Next, the electronic device (200) can transmit transmission delay information to the transmitting device (100) based on the decoding delay status of the content (S1120). For example, the electronic device (200) can transmit transmission delay information including the number of video frame delay frames and the number of audio frame delay frames to the transmitting device (100).

[0176] Next, the electronic device (200) can receive content including skip level information for each frame (S1130). For example, the electronic device (200) can receive content including skip level information for each video frame and skip level information for each audio frame.

[0177] Next, the electronic device (200) can monitor the number of decoding delay frames to determine whether frame skipping is required (S1140). If the electronic device (200) determines that frame skipping is required (S1150:Y), the electronic device (200) can identify skippable frames based on skip level information for each frame (S1160). For example, the electronic device (200) can identify skippable video frames based on skip level information for each video frame, and can identify skippable audio frames based on skip level information for each audio frame.

[0178] Next, the electronic device (200) can identify audio frames and video frames to be finally skipped based on the plurality of frame-by-frame skip level information and the synchronization of audio content and video content (S1170). For example, the electronic device (200) must skip video frames and audio frames so that synchronization of the audio frames and video frames is possible. Accordingly, the electronic device (200) can identify audio frames and video frames as finally skippable only when both synchronized audio frames and video frames are skippable.

[0179] Next, the electronic device (200) can decode the received content by skipping the identified audio frame and the identified video frame (S1180).

[0180] Meanwhile, in Fig. 11, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0181] FIG. 12 is a drawing for explaining the operation between a transmitting device (100) and a receiving device (200) according to one or more embodiments.

[0182] According to FIG. 12, a transmitting device (100) and a receiving device (200) can communicate via a wireless network. In one example, the transmitting device (100) can be implemented as an electronic device (100) illustrated in FIG. 2, and the receiving device (200) can be implemented as an electronic device (200, 200') illustrated in FIGS. 8a and 8b.

[0183] According to one example, the transmitting device (810) may include an audio analysis module (12305), an audio skip module (1210), an audio encoder (1215), a video analysis module (1220), a video skip module (1225), a video encoder (1230), and a transmission / reception module (1240). Here, each module may be implemented with at least one software, at least one hardware, and / or a combination thereof. For example, at least one of the audio analysis module (12305), the audio skip control module (1210), the audio encoder (1215), the video analysis module (1220), the video skip control module (1225), the video encoder (1230), and the transmission / reception module (1240) may be implemented to utilize a predefined algorithm, a predefined formula, and / or a learned artificial intelligence model. The audio analysis module (12305), audio skip module (1210), audio encoder (1215), video analysis module (1220), video skip module (1225), video encoder (1230), and transmission / reception module (1240) may be included within the transmission device (100), but may be distributed to at least one external device according to an example.

[0184] According to one example, the receiving device (2O0) may include a transmitting / receiving module (1245), an audio decoder module (1250), an audio output module (1255), a video decoder module (1260), a video output module (1265), and a delay management module (1270). Here, each module may be implemented with at least one software, at least one hardware, and / or a combination thereof. For example, at least one of the transmitting / receiving module (1245), the audio decoder module (1250), the audio output module (1255), the video decoder module (1260), the video output module (1265), and the delay management module (1270) may be implemented to utilize a predefined algorithm, a predefined formula, and / or a learned artificial intelligence model. The transmitter / receiver module (1245), audio decoder module (1250), audio output module (1255), video decoder module (1260), video output module (1265), and delay management module (1270) may be included within the receiving device (200), but may be distributed to at least one external device, as an example.

[0185] In one example, the transmitting device (100) may analyze audio content using the audio analysis module (1205). For example, the audio analysis module (1205) may analyze at least one of data continuity, content type, or sound volume. Here, the content type may indicate not only the content provision form, such as real-time streaming content or OTT (content type) content, but also the content genre, such as game content or movie content.

[0186] In one example, the transmitting device (100) may determine audio encoding control options using the audio skip control module (1205). For example, the audio analysis module (1205) may identify whether skipping is possible for each audio frame based on transmission delay information transmitted from the transmitting / receiving module (270) and the analysis results of the audio analysis module (1205).

[0187] In one example, the transmitting device (100) may encode audio content using the audio encoder module (1215). For example, the audio encoder module (1215) may perform compression on an audio frame-by-frame basis to generate a plurality of compressed audio packets. In this case, the header area of ​​the plurality of audio compression packets may include compression information and skip level information.

[0188] In one example, the transmitting device (100) can analyze video content using the video analysis module (1220). For example, the video analysis module (1220) can analyze at least one of scene changes, content type, motion size, or frame rate. Here, the content type can indicate not only the content provision form, such as real-time streaming content or OTT (content type) content, but also the content genre, such as game content or movie content.

[0189] In one example, the transmitting device (100) may determine a video encoding control option using the video skip control module (1225). For example, the video skip control module (1225) may identify whether skipping is possible for each video frame based on transmission delay information transmitted from the transmitting / receiving module (270) and the analysis results of the video analysis module (1220).

[0190] In one example, the transmitting device (100) may encode video content using the video encoder module (1230). For example, the video encoder module (1230) may perform compression on a video frame-by-frame basis to generate a plurality of compressed video packets. In this case, the header area of ​​the plurality of video compression packets may include compression information and skip level information.

[0191] According to an example, the transmitting device (100) can transmit a compressed stream to the receiving device (200) using the transmitting / receiving module (1270). For example, the transmitting / receiving module (1270) can transmit a compressed stream including an encoded audio stream and an encoded video stream to the receiving device (200). In addition, the transmitting device (100) can transmit transmission delay information received from the receiving device (100) to the audio skip control module (1210) and the video skip control module (1225) using the transmitting / receiving module (1270).

[0192] In one example, the receiving device (200) may receive a compressed stream from the transmitting device (100) using the transceiver module (1245). In one example, the compressed stream may include an encoded audio stream and an encoded video stream.

[0193] In one example, the receiving device (200) may decode an audio stream using the audio decoder module (1250). For example, the audio decoder module (1250) may decode the audio stream by skipping audio frames, if necessary, based on transmission delay information received from the delay management module (1270). For example, the audio decoder module (1250) may decode the audio stream by skipping necessary audio frames based on a skip level corresponding to each audio frame.

[0194] According to an example, the receiving device (200) can output decoded audio content using the audio output module (1255). For example, the audio output module (1255) can output the audio content through a speaker (270).

[0195] In one example, the receiving device (200) can decode a video stream using the video decoder module (1260). For example, the video decoder module (1260) can decode the video stream by skipping video frames, if necessary, based on transmission delay information received from the delay management module (1270). For example, the video decoder module (1260) can decode the video stream by skipping necessary audio frames based on a skip level corresponding to each audio frame. In one example, the video decoder module (1260) can decode frames included in the encoded content in units of decoding blocks. However, the present invention is not limited thereto, and the video decoder module (1260) can decode frames included in the encoded content in units of decoding lines.

[0196] In one example, the receiving device (200) can output decoded video content using the video output module (1265). For example, the video output module (1265) can output the video content through the display (240).

[0197] According to one example, the receiving device (200) can monitor the decoding delay status of the audio decoder module (1250) and the video decoder module (1260) using the delay management module (1270) and generate transmission delay information based on the monitoring results. In addition, the receiving device (200) can transmit the transmission delay information generated using the delay management module (1270) to the audio decoder module (1250), the video decoder module (1260), and the transmission / reception module (1245). In this case, the transmission / reception module (1245) can transmit the transmission delay information to the transmission device (100).

[0198] According to an example, the receiving device (200) can manage synchronization (or sync alignment) of audio frames and video frames according to decoding skip of audio frames and video frames using the delay management module (1270). For example, when skippable audio frames and video frames are identified according to a skip level, the delay management module (1270) can determine the final skipped audio frame and video frame so that the corresponding audio frame and video frame are output in synchronization.

[0199] FIG. 13 is a diagram illustrating operation in unicast mode according to one or more embodiments.

[0200] For example, when the transmitting device (100) and the receiving device (200) are in a 1:1 transmission / reception mode (e.g., unicast mode), the transmitting device (100) can encode content by skipping skippable frames based on a plurality of identified frame-by-frame skip levels.

[0201] According to FIG. 13, the transmitting device (100) can analyze audio content to determine an initial skip level for each frame (S1305). Furthermore, the transmitting device (100) can analyze video content to determine an initial skip level for each frame (S1310). For example, the transmitting device (100) can determine an initial skip level for each frame based on at least one of data continuity and sound volume.

[0202] Next, the transmitting device (100) can adjust the skip level according to the content service classification characteristics (S1315). For example, the transmitting device (100) can adjust the skip level per frame based on the content provision type, such as real-time streaming content or OTT (content type) content, as well as the content genre, such as game content or movie content. In this case, the transmitting device (100) can adjust the skip level per frame by securing the maximum number of skippable frames.

[0203] Next, the transmitting device (100) can identify whether the delay level is greater than or equal to a threshold level based on the transmission delay information received from the receiving device (200) (S1320). For example, the transmitting device (100) can identify whether the number of decoding delay frames included in the transmission delay information is greater than or equal to a threshold number. For example, the threshold number can be determined to be a different value depending on the content provision form, such as real-time streaming content or OTT (content type) content, as well as the content genre, such as game content or movie content.

[0204] The transmitting device (100) can adjust the skip level and number of frames per frame when the delay level is greater than or equal to a threshold level based on transmission delay information (S1325).

[0205] Next, the transmitting device (100) can determine the final skip frame by considering audio and video sync (S1330).

[0206] Next, the transmitting device (100) can encode audio content except for the audio frame determined as the final skip frame (S1335). In addition, the transmitting device (100) can encode video content except for the video frame determined as the final skip frame (S1340).

[0207] Next, the transmitting device (100) can transmit a compressed stream containing encoded audio content and video content with skip applied to the receiving device (200) (S1345)

[0208] Meanwhile, the receiving device (200) can receive a compressed stream including encoded audio content and video content from the transmitting device (100) (S1350).

[0209] The receiving device (200) can decode audio content and video content (S1355, S1360). In this case, the receiving device (200) can monitor the number of decoding delay frames of the audio content and video content to generate transmission delay information and transmit the transmission delay information to the transmitting device (100).

[0210] As described above, if the current streaming mode is unicast mode, the transmitting device (100) can perform encoding by directly performing frame skipping based on the number of delayed frames included in the transmission delay information. Accordingly, the transmission amount can be reduced and the decoder's workload can also be reduced.

[0211] Meanwhile, in Fig. 13, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0212] FIG. 14 is a diagram for explaining operation in multicast mode according to one or more embodiments.

[0213] For example, when a transmitting device (100) and a receiving device (200) are in a 1:N transmission / reception mode (e.g., multicast / broadcast mode), the transmitting device (100) can encode content so that skippable frames are skippable when decoded by the receiving device (200) based on the identified skip levels per frame.

[0214] According to FIG. 14, the transmitting device (100) can analyze audio content to determine an initial skip level for each frame (S1405). Furthermore, the transmitting device (100) can analyze video content to determine an initial skip level for each frame (S1410). For example, the transmitting device (100) can determine an initial skip level for each frame based on at least one of data continuity and sound volume.

[0215] Next, the transmitting device (100) can adjust the skip level according to the content service classification characteristics (S1415). For example, the transmitting device (100) can adjust the skip level per frame based on the content provision type, such as real-time streaming content or OTT (content type) content, as well as the content genre, such as game content or movie content. In this case, the transmitting device (100) can adjust the skip level per frame by securing the maximum number of skippable frames.

[0216] Next, the transmitting device (100) can identify whether there is a frame skip request from the receiving device (200) (S1420).

[0217] When a frame skip request is received from a receiving device (200), the transmitting device (100) can adjust the skip level and the number of frames based on the frame skip request (S1425). For example, the transmitting device (100) can set the skip level for each frame so that the decoder can skip the number of delayed frames included in the frame skip request.

[0218] Next, the transmitting device (100) can determine the skip level for each final frame (S1430).

[0219] Next, the transmitting device (100) can encode audio content while considering skip frames during decoding (S1435). In addition, the transmitting device (100) can encode video content while considering skip frames during decoding (S1440). For example, the transmitting device (100) can encode video content so that skippable frames are excluded from the reference frame during decoding.

[0220] Next, the transmitting device (100) can transmit a compressed stream including encoded audio content and video content to the receiving device (200) (S1445). In this case, the compressed stream can include frame-by-frame skip level information.

[0221] Meanwhile, the receiving device (200) can receive a compressed stream containing encoded audio content and video content from the transmitting device (100) (S1450). Subsequently, the receiving device (200) can decode the audio content and video content.

[0222] The receiving device (200) can monitor the decoding status of audio content and video content to identify the number of frames requiring decoding skip (S1455).

[0223] Next, the receiving device (200) can identify whether the number of frames requiring skipping is greater than or equal to a threshold number (S1460).

[0224] The receiving device (200) can transmit a skip addition request to the transmitting device (100) when the number of frames requiring a skip is greater than or equal to a threshold number (S1465).

[0225] Next, the receiving device (200) can identify the final skip frame by considering the frame-by-frame skip level and audio / video sync (S1470).

[0226] Next, the receiving device (200) can decode audio content and video content by skipping some frames based on the skip level (S1474, S1480).

[0227] As described above, when the current streaming mode is a multicast mode, the transmitting device (100) transmits a skip level per frame so that the receiving device (200) can perform delayed multiple playback suitable for the local situation.

[0228] Meanwhile, in Fig. 14, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0229] FIGS. 15A to 15D are drawings for explaining a compression mode identification method according to one or more embodiments.

[0230] According to one or more embodiments, the transmitting device (100) can identify whether to compress frames after a frame skip occurs in an intra mode, taking into account image quality and coding efficiency, or to compress frames in an inter mode that only refers to frames that are not skipped frames.

[0231] For example, if the transmitting device (100) is identified as the intra mode, it can perform intra coding, and if it is identified as the inter mode, it can perform inter coding by referring only to frames excluding frames to be skipped. Here, intra coding is a method of individually performing compression for each frame as illustrated in FIG. 15a, and inter coding means a method of compressing multiple frames at once by utilizing similarities between adjacent frames as illustrated in FIG. 15b. An image compressed according to inter coding includes an inter frame as illustrated in FIG. 15c, and an inter frame compresses information of one or more frames, and may mean a B frame (Bidirectional predictive frame) and a P frame (Predicitve frame) excluding an I frame (Inter Frame). For example, assuming that the frames illustrated in FIG. 15c constitute one GOP, the I frame and P frame can be processed first during decoding, and then the B frame in between can be processed. Accordingly, the transmitting device (100) can encode to preferentially skip B frames within a single GOP. In addition, the transmitting device (100) can encode other GOPs by prioritizing them in the order of I frame > P frame > B frame.

[0232] In contrast, in intra mode, each frame is individually compressed as an intra frame, as illustrated in Figure 15d, so any frame can be skipped. Accordingly, intra coding can be performed regardless of the frames to be skipped.

[0233] According to the various embodiments described above, low-latency streaming services, such as real-time streaming, can minimize transmission delays while maintaining quality depending on network conditions and content characteristics. Furthermore, when delays occur, they can be restored to a minimal delay state as quickly as possible, prevent delay accumulation, and provide viewing quality optimized for the user's display specifications.

[0234] Meanwhile, the methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device.

[0235] Additionally, the various embodiments of the present disclosure described above can also be performed through an embedded server provided in an electronic device, or an external server of the electronic device.

[0236] Meanwhile, the methods according to the various embodiments of the present disclosure described above can be implemented in the form of applications that can be installed on existing electronic devices. Alternatively, the methods according to the various embodiments of the present disclosure described above can be performed using a deep learning-based artificial neural network (or deep artificial neural network), i.e., a learning network model. In one example, at least one of downscaling, decoding, encoding, and upscaling can be performed using a learned neural network model.

[0237] Meanwhile, according to a temporary example of the present disclosure, the various embodiments described above can be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored from the storage medium and operate according to the called instructions, and may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments. When an instruction is executed by a processor, the processor can perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0238] Furthermore, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0239] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0240] While the present disclosure has been specifically illustrated and described with reference to embodiments thereof, it will be understood that various changes in form and detail may be made therein without departing from the spirit and scope of the following claims.

Claims

1. In electronic devices, A communication interface capable of connecting to a wireless network; A memory storing one or more instructions; and comprising one or more processors; The one or more processors, by executing the one or more instructions, Transmitting content in real time to an external device via the wireless network through the above communication interface, Identifying a plurality of frame-by-frame skip levels included in the content based on transmission delay information received from the external device and characteristics of the content, Encode the content based on the identified multiple frame-by-frame skip levels, An electronic device that transmits the plurality of frame-by-frame skip level information and the encoded content to the external device through the wireless network via the communication interface.

2. In paragraph 1, The one or more processors, by executing the one or more instructions, When a skippable frame is identified among a plurality of frames included in the above content, the content is encoded so that the identified skippable frame is skippable when decoded by the external device, An electronic device that transmits the encoded content, in which the skip level information is included in one area of ​​each of the plurality of frames, to an external device via the wireless network through the communication interface.

3. In paragraph 2, The one or more processors, by executing the one or more instructions, An electronic device that, when a skippable frame is identified among a plurality of frames included in the above content, encodes the above content by excluding the identified skippable frame from the reference frame.

4. In paragraph 1, The one or more processors, by executing the one or more instructions, If the above content is audio content, identify multiple frame-by-frame skip levels based on at least one of data continuity, content type, or sound size, An electronic device that identifies a plurality of frame-by-frame skip levels based on at least one of scene change, content type, motion size, or frame rate when the content is video content.

5. In paragraph 1, The one or more processors, by executing the one or more instructions, If the above content is identified as the first type content, the plurality of frame-by-frame skip levels are identified so that the number of skippable frames is greater than or equal to a threshold number, An electronic device that identifies the plurality of frame-by-frame skip levels so that the number of skippable frames becomes less than a threshold number when the content is identified as second type content.

6. In paragraph 1, The one or more processors, by executing the one or more instructions, When the electronic device and the external device are in a 1:1 transmission / reception situation, the content is encoded by skipping skippable frames based on the identified multiple frame-by-frame skip levels, An electronic device, wherein when the electronic device and the external device are in a 1:N (N>1) transmission / reception situation, the content is encoded so that skippable frames are skippable when decoded by the external device based on the identified plurality of frame-by-frame skip levels.

7. In electronic devices, A communication interface capable of connecting to a wireless network; A memory storing one or more instructions; and comprising one or more processors; The one or more processors, by executing the one or more instructions, Receives content in real time from an external device through the wireless network via the above communication interface and decodes it, Transmitting transmission delay information to the external device based on the decoding delay status of the above content, An electronic device that decodes the received content including a plurality of frame-by-frame skip level information based on the identified plurality of frame-by-frame skip level information when the content including the plurality of frame-by-frame skip level information is received based on the transmission delay information and the characteristics of the content.

8. In paragraph 7, The one or more processors, by executing the one or more instructions, Monitor the number of decoding delay frames to identify if frame skipping is required, An electronic device that identifies a skippable frame based on the plurality of frame-by-frame skip level information when it is determined that frame skipping is necessary, and decodes content including the received plurality of frame-by-frame skip level information by skipping the identified frame.

9. In paragraph 8, The one or more processors, by executing the one or more instructions, An electronic device that transmits transmission delay information requesting additional skippable frames to an external device when it is identified that additional skippable frames are required based on the number of decoding delay frames and the plurality of frame-by-frame skip level information.

10. In paragraph 7, The one or more processors, by executing the one or more instructions, Based on the skip level information for each of the above multiple frames and the synchronization of audio content and video content, the audio frame and video frame to be finally skipped are identified, An electronic device for decoding content including skip level information per the received plurality of frames by skipping the identified audio frames and the identified video frames.

11. In a method for controlling an electronic device, A step of transmitting content to an external device in real time via a wireless network; A step of identifying a plurality of frame-by-frame skip levels included in the content based on transmission delay information received from the external device and characteristics of the content; A step of encoding the content based on the identified plurality of frame-by-frame skip levels; and A control method, comprising: a step of transmitting the plurality of frame-by-frame skip level information and the encoded content to the external device via the wireless network.

12. In paragraph 11, The steps for encoding the above content are: A step of encoding the content so that the identified skippable frame is skippable when decoding in the external device, when a skippable frame is identified among a plurality of frames included in the content; The step of transmitting the plurality of frame-by-frame skip level information and the encoded content to the external device comprises: A control method, comprising: a step of transmitting the encoded content, in which skip level information indicating whether skipping is possible for each of the plurality of frames is included in one area of ​​each of the plurality of frames, to the external device; 13. In paragraph 12, The steps for encoding the above content are: A control method, comprising: a step of encoding the content by excluding the identified skippable frame from the reference frame when a skippable frame is identified among a plurality of frames included in the content.

14. In paragraph 11, The step of identifying the above multiple frame-by-frame skip levels is: A step of identifying a plurality of frame-by-frame skip levels based on at least one of data continuity, content type, or sound size when the content is audio content; and A control method, comprising: a step of identifying a plurality of frame-by-frame skip levels based on at least one of scene change, content type, motion size, or frame rate when the content is video content.

15. In paragraph 11, The step of identifying the above multiple frame-by-frame skip levels is: If the above content is identified as the first type content, a step of identifying the plurality of frame-by-frame skip levels so that the number of skippable frames is greater than or equal to a threshold number; and A control method, comprising: a step of identifying the plurality of frame-by-frame skip levels so that the number of skippable frames becomes less than a threshold number, if the content is identified as second type content.

Images