Method of synchronizing audio and video and electronic device

Abstract

A method for synchronizing audio and video according to the present disclosure comprises the steps in which: a first electronic device receives video frame data and audio frame data from a second electronic device through wireless communication; the first electronic device plays back the video and audio at a first playback speed on the basis that an index of the video frame data is the same as an index of the audio frame data; and the first electronic device maintains the playback speed of the video at the first playback speed and plays back the audio at a second playback speed faster than the first playback speed on the basis that the index of the video frame data is different from the index of the audio frame data.

Description

Method and electronic device for synchronizing audio and video

[0001] The present disclosure relates to a method for synchronizing audio and video and an electronic device performing the same.

[0002] Recent advancements in video and audio processing technology have led to the production of large quantities of high-definition and high-quality content. Furthermore, advancements in communications technology have made it possible to transmit this content wirelessly.

[0003] Unlike when transmitting content using wired communication, when transmitting content using wireless communication, content transmission may frequently be delayed depending on the status of the communication network.

[0004] The present disclosure provides a method for synchronizing audio and video in a situation where content is transmitted / received via wireless communication and an electronic device for performing the same.

[0005] The present disclosure provides a method for synchronizing audio and video, which can prevent audio interruption in a situation where video frames are skipped due to poor wireless communication conditions, and an electronic device for performing the same.

[0006] The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0007] A method for synchronizing audio and video according to one embodiment of the present disclosure may include: a first electronic device receiving video frame data and audio frame data from a second electronic device via wireless communication; the first electronic device playing back video and audio at a first playback speed based on an index of the video frame data being the same as an index of the audio frame data; and the first electronic device maintaining a playback speed of the video at the first playback speed and playing back the audio at a second playback speed that is faster than the first playback speed based on an index of the video frame data being different from an index of the audio frame data.

[0008] An electronic device according to one embodiment of the present disclosure includes: a content playback device; a communication interface connected to a wireless network; a memory storing at least one command; and at least one processor connected to the communication interface and the memory, wherein the at least one processor, by executing the at least one command, receives video frame data and audio frame data from an external device through the communication interface, and controls the content playback device to play back video and audio at a first playback speed based on an index of the video frame data being the same as an index of the audio frame data, and controls the content playback device to maintain the playback speed of the video at the first playback speed and play back the audio at a second playback speed that is faster than the first playback speed based on an index of the video frame data being different from an index of the audio frame data.

[0009] FIG. 1 illustrates an example of a content provision system according to one embodiment.

[0010] FIG. 2 is a block diagram illustrating an example of a configuration of a content provision system according to one embodiment.

[0011] FIG. 3 is a conceptual diagram for explaining in more detail an example of the configuration of a content provision system according to one embodiment.

[0012] FIG. 4 is a flowchart illustrating a method for providing content by a content providing system according to one embodiment.

[0013] FIG. 5 is a flowchart illustrating an example of data processing performed by a second electronic device according to one embodiment.

[0014] FIG. 6 is a conceptual diagram illustrating an example of data processing performed by a second electronic device according to one embodiment.

[0015] FIG. 7 conceptually illustrates an example of how video frame data and audio frame data are transmitted from a second electronic device to a first electronic device according to one embodiment.

[0016] FIG. 8 is a flowchart illustrating an example of data processing performed by a first electronic device according to one embodiment.

[0017] FIG. 9 is a conceptual diagram illustrating an example of data processing performed by a first electronic device according to one embodiment.

[0018] FIG. 10A conceptually illustrates an example of data stored in a first electronic device when the first electronic device receives video frame data and audio frame data from a second electronic device without communication delay according to one embodiment.

[0019] FIG. 10b conceptually illustrates an example of data stored in a first electronic device when the first electronic device receives video frame data and audio frame data from a second electronic device in a communication delay situation according to one embodiment.

[0020] FIG. 11 illustrates an example of video and audio output by a first electronic device in a communication delay situation, according to one embodiment.

[0021] FIG. 12 illustrates an example of video and audio output by a first electronic device after a communication delay situation is resolved, according to one embodiment.

[0022] FIG. 13 illustrates an example of a user interface for setting video interruption, provided by a first electronic device according to one embodiment.

[0023] FIG. 14 illustrates an example of a user interface for setting audio speed provided by a first electronic device according to one embodiment.

[0024] The embodiments described in this specification and the configurations illustrated in the drawings are merely preferred examples of the disclosed invention, and there may be various modified examples that can replace the embodiments and drawings of this specification at the time of filing of this application.

[0025] The terminology used herein is for the purpose of describing embodiments only and is not intended to limit and / or restrict the disclosed invention.

[0026] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to cases where (1) only A is included, (2) only B is included, or (3) both A and B are included.

[0027] For example, in this specification, a singular expression may include a plural expression unless the context clearly indicates otherwise.

[0028] Additionally, terms such as “include” or “have” are intended to express the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but do not exclude the possibility of the additional presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0029] When a component is said to be “connected,” “coupled,” “supported,” or “in contact with” another component, this includes not only cases where the components are directly connected, coupled, supported, or in contact, but also cases where the components are indirectly connected, coupled, supported, or in contact through a third component.

[0030] When we say that a component is "on" another component, this includes not only cases where the component is in contact with the other component, but also cases where there is another component between the two components.

[0031] When it is said that a component (e.g., a first component) is “operatively or communicatively coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0032] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0033] In some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing the operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform the operations by executing at least one software program stored in a memory device.

[0034] Additionally, terms that include ordinal numbers, such as “first,” “second,” etc., are used to distinguish one component from another, and do not limit one component.

[0035] Additionally, terms such as "~part", "~device", "~block", "~absence", and "~module" may refer to a unit that processes at least one function or operation. For example, the terms may refer to at least one piece of hardware such as an FPGA (field-programmable gate array) / ASIC (application specific integrated circuit), at least one piece of software stored in memory, or at least one process processed by a processor.

[0036] Hereinafter, an embodiment of the disclosed invention will be described in detail with reference to the attached drawings. The same reference numbers or symbols used in the attached drawings may represent parts or components that perform substantially the same functions.

[0037] The operating principle and embodiments of the present disclosure are described below with reference to the attached drawings.

[0038] FIG. 1 illustrates an example of a content provision system according to one embodiment. FIG. 2 is a block diagram illustrating an example of the configuration of a content provision system according to one embodiment.

[0039] Referring to FIGS. 1 and 2, a content providing system according to one embodiment may include a first electronic device (1), a second electronic device (2), and a source device (3).

[0040] In one embodiment, the source device (3) may be a device that provides source content.

[0041] Source content may include source video data and source audio data.

[0042] The source video data may include source video frame data.

[0043] Source audio data may include source audio frame data.

[0044] The source device (3) can supply source frame data including source video frame data and source audio frame data.

[0045] The source device (3) may include devices such as a game console, PC, STB, BD Player, server, etc.

[0046] The source device (3) may include a port for coupling with a connector of the second electronic device (2). The connector may include HDMI, DisplayPort, etc.

[0047] A source device (3) can transmit content to a second electronic device (2) using wired communication (WR). The content can include content data, and the content data can include video frame data and audio frame data.

[0048] The source device (3) transmitting content to the second electronic device (2) may include the source device (3) transmitting source frame data including source video frame data and source audio frame data to the second electronic device (2).

[0049] The source device (3) may include at least one processor for controlling the operation of the source device (3) and at least one memory in which a program for controlling the operation of the source device (3) is stored.

[0050] The second electronic device (2) can provide a real-time video streaming service, and the real-time video streaming service can include a live broadcasting service, a cloud gaming service, a mirroring service, a video communication service, etc.

[0051] The second electronic device (2) can be implemented as various computing devices such as a workstation, a cloud, a data drive, a data station, etc.

[0052] For example, the second electronic device (2) may be implemented as a set-top box or a one connected box, but is not limited thereto, and may be applied to any device having image processing and communication functions.

[0053] The second electronic device (2) can perform functions such as managing a user account, registering the first electronic device (1) by linking it to the user account, and managing or controlling the registered first electronic device (1).

[0054] For example, a user can access a second electronic device (2) through a user interface provided by a first electronic device (1) and create a user account. The user account can be identified by an ID and password set by the user. The second electronic device (2) can register the first electronic device (1) to the user account according to a predetermined procedure. For example, the second electronic device (2) can register, manage, and control the first electronic device (1) by linking identification information (e.g., serial number or MAC address) of the first electronic device (1) to the user account.

[0055] The second electronic device (2) registering the first electronic device (1) may include the second electronic device (2) establishing wireless communication (wl) with the first electronic device (1).

[0056] The second electronic device (2) may include a communication interface (230) for establishing wired communication (wr) with the source device (3) and for establishing wireless communication (wl) with the first electronic device (1).

[0057] For this purpose, the communication interface (230) may include a wireless communication module and a wired communication module.

[0058] Wired communication methods may include HDMI, DisplayPort, etc. Wireless communication methods may include WiBro, Bluetooth (IEEE 802.15.1), Zigbee (IEEE 802.15.4), Wi-Fi Access Point, Wi-Fi Direct, NFC (Near Field Communication), Z-Wave, etc.

[0059] The communication interface (230) may support wireless communication such as Bluetooth, AP-based Wi-Fi (Wireless LAN network), Zigbee, Wireless LAN (Local Area Network), WAN (Wide Area Network), etc. However, it is not limited thereto, and may also support wired communication such as Ethernet, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), Optical, Coaxial, etc.

[0060] The second electronic device (2) can transmit content to the first electronic device (1) using wireless communication (wl). The content can include content data, and the content data can include video frame data and audio frame data. Here, the content data can correspond to content data transmitted from the source device (3) to the second electronic device (2).

[0061] In one embodiment, the second electronic device (2) can process content data received from the source device (3). For example, the second electronic device (2) can process source frame data received from the source device (3) and transmit the processed source frame data (hereinafter, 'frame data') to the first electronic device (1).

[0062] That is, the second electronic device (2) can process source frame data received from the source device (3) via wired communication (wr) and transmit the frame data to the first electronic device (1) via wireless communication (wl).

[0063] The second electronic device (2) may include at least one processor (210) that controls the operation of the second electronic device (2) and at least one memory (220) that stores a program for controlling the operation of the second electronic device (2).

[0064] At least one memory (220) can store data required for various embodiments. The memory (220) may be implemented in the form of a memory embedded in the second electronic device (2) or may be implemented in the form of a memory that can be detachably attached to the second electronic device (2), depending on the purpose of data storage. For example, data for driving the second electronic device (2) may be stored in a memory embedded in the second electronic device (2), and data for expanding the functions of the second electronic device (2) may be stored in a memory that can be detachably attached to the second electronic device (2). Meanwhile, in the case of the memory embedded in the second electronic device (2), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)). In addition, in the case of the memory that can be detachably attached to the electronic device (2), it may be implemented as a form of a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. Can be.

[0065] At least one processor (210) controls the overall operation of the second electronic device (2). Specifically, at least one processor (210) is connected to each component of the second electronic device (2) and can control the overall operation of the second electronic device (2). For example, at least one processor (210) is electrically connected to a memory (220) and can control the overall operation of the second electronic device (2). The processor (210) may be composed of one or more processors.

[0066] At least one processor (210) can perform operations of the second electronic device (2) according to various embodiments by executing at least one instruction stored in the memory (220).

[0067] At least one processor (210) may include one or more of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an APU (Accelerated Processing Unit), an MIC (Many Integrated Core), a DSP (Digital Signal Processor), an NPU (Neural Processing Unit), a hardware accelerator, or a machine learning accelerator. At least one processor (210) may control one or any combination of other components of the second electronic device (2), and may perform operations related to communication or data processing. At least one processor (210) may execute at least one program or instruction stored in the memory (220). For example, at least one processor (210) may execute at least one instruction stored in the memory (220), thereby performing a method according to at least one embodiment of the present disclosure.

[0068] When a method according to at least one embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to at least one embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0069] At least one processor (210) may be implemented as a single core processor including one core, or may be implemented as at least one multicore processor including multiple cores (e.g., homogeneous multicore or heterogeneous multicore). When at least one processor (210) is implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to at least one embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to at least one embodiment of the present disclosure.

[0070] When a method according to at least one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to at least one embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0071] In embodiments of the present disclosure, a processor may mean a system on a chip (SoC) in which at least one processor and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but embodiments of the present disclosure are not limited thereto. Hereinafter, for convenience of description, at least one processor (210) will be referred to as a processor (210).

[0072] At least one processor (210) can process source frame data received from a source device (3) and control a communication interface (230) to transmit the frame data to a first electronic device (1).

[0073] At least one processor (210) may include a preprocessing unit (210p, see FIG. 3) for preprocessing source frame data received from a source device (3), a video processor (210v, see FIG. 3) for processing source video frame data, and an audio processor (210a) for processing source audio frame data.

[0074] As previously explained, the preprocessing unit (210p, see FIG. 3), the video processor (210v, see FIG. 3), and the audio processor (210a) are conveniently classified to explain some of the functions of at least one processor (210), and the functions of the processing unit (210p, see FIG. 3), the video processor (210v, see FIG. 3), and the audio processor (210a) may be performed by processors having different cores, may be performed by one core among a plurality of cores included in a multi-core processor, or may be performed by a plurality of cores.

[0075] The first electronic device (1) can be implemented as a TV as shown in FIG. 1, but is not limited thereto, and can be applied to any device having communication and display functions, such as a personal computer, a monitor, a smart phone, a tablet PC, a wearable device, a notebook PC, an HMD (Head mounted Display), a NED (Near Eye Display), a LFD (large format display), a digital signage, a DID (Digital Information Display), a video wall, a projector display, etc.

[0076] The first electronic device (1) may include a communication interface (130) for establishing wireless communication (wl) with the second electronic device (2).

[0077] For this purpose, the communication interface (130) may include a wireless communication module.

[0078] For example, the communication interface (130) can support wireless communication such as Bluetooth, AP-based Wi-Fi (Wi-Fi, Wireless LAN network), Zigbee, Wireless LAN (Local Area Network), WAN (Wide Area Network), etc.

[0079] According to various embodiments, the communication interface (130) may further include a wired communication module for establishing wired communication with the source device (3).

[0080] Accordingly, the user can choose either to connect the first electronic device (1) to the source device (3) by wire or to connect it to the second electronic device (2) wirelessly.

[0081] A first electronic device (1) can receive content from a second electronic device (2) via wireless communication (wl). The content can include content data, and the content data can include video frame data and audio frame data. Here, the content data can correspond to content data transmitted from a source device (3) to the second electronic device (2).

[0082] As a result, the source device (3) can transmit content data to the second electronic device (2) via wired communication (wr), the second electronic device (2) can transmit content data to the first electronic device (1) via wireless communication (wl), and the first electronic device (1) can play content corresponding to the content data.

[0083] Since the source device (3) transmits content data to the second electronic device (2) via wired communication (wr), the content data can be transmitted to the second electronic device (2) without any communication delay. On the other hand, since the second electronic device (2) transmits content data to the first electronic device (1) via wireless communication (wl), if the wireless communication (wl) network condition is unstable, the transmission of the content data may be delayed due to communication delay.

[0084] The first electronic device (1) may include at least one processor (110) that controls the operation of the first electronic device (1) and at least one memory (120) that stores a program for controlling the operation of the first electronic device (1).

[0085] At least one memory (120) can store data required for various embodiments. The memory (120) may be implemented in the form of a memory embedded in the first electronic device (1) or may be implemented in the form of a memory detachable from the first electronic device (1) depending on the purpose of data storage. For example, data for driving the first electronic device (1) may be stored in a memory embedded in the first electronic device (1), and data for expanding functions of the first electronic device (1) may be stored in a memory detachable from the first electronic device (1). Meanwhile, in the case of the memory embedded in the first electronic device (1), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)). In addition, in the case of the memory that can be detachably attached to the electronic device (1), it may be implemented as a form of a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. Can be.

[0086] At least one processor (110) controls the overall operation of the first electronic device (1). Specifically, at least one processor (110) is connected to each component of the first electronic device (1) and can control the overall operation of the first electronic device (1). For example, at least one processor (110) is electrically connected to a memory (120) and can control the overall operation of the first electronic device (1). The processor (110) may be composed of one or more processors.

[0087] At least one processor (110) can perform operations of the first electronic device (1) according to various embodiments by executing at least one instruction stored in the memory (120).

[0088] At least one processor (110) may include one or more of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an APU (Accelerated Processing Unit), an MIC (Many Integrated Core), a DSP (Digital Signal Processor), an NPU (Neural Processing Unit), a hardware accelerator, or a machine learning accelerator. The at least one processor (110) may control one or any combination of other components of the first electronic device (1) and may perform operations related to communication or data processing. The at least one processor (110) may execute at least one program or instruction stored in the memory (120). For example, the at least one processor (110) may perform a method according to at least one embodiment of the present disclosure by executing at least one instruction stored in the memory (120).

[0089] When a method according to at least one embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to at least one embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0090] At least one processor (110) may be implemented as a single core processor including one core, or may be implemented as at least one multicore processor including multiple cores (e.g., homogeneous multicore or heterogeneous multicore). When at least one processor (110) is implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to at least one embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to at least one embodiment of the present disclosure.

[0091] When a method according to at least one embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to at least one embodiment, the first operation, the first operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0092] In embodiments of the present disclosure, a processor may mean a system on a chip (SoC) in which at least one processor and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but embodiments of the present disclosure are not limited thereto. Hereinafter, for convenience of description, at least one processor (110) will be referred to as a processor (110).

[0093] At least one processor (110) can process content data received from the second electronic device (2) and control a content player to play the content.

[0094] A content playback device (125) may include a display and a speaker. The display may play video of the content, and the speaker may play audio of the content. Playing video of the content may include outputting an image of the content. Playing audio of the content may include outputting sound of the content.

[0095] At least one processor (110) can control the display of the content playback device (125) to play the video based on processing the video frame data received from the second electronic device (2).

[0096] At least one processor (110) can control a speaker of a content playback device (125) to play audio based on processing audio frame data received from a second electronic device (2).

[0097] At least one processor (110) may include a preprocessing unit (110p, see FIG. 3) for preprocessing frame data received from the second electronic device (2), a video processor (110v, see FIG. 3) for processing video frame data, and an audio processor (110a) for processing audio frame data.

[0098] In one embodiment, at least one processor (110) may further include a sync control unit (110s, see FIG. 3) that controls the sync of video and audio played by the content playback device (125).

[0099] As described above, the preprocessing unit (110p, see FIG. 3), the video processor (110v, see FIG. 3), the audio processor (110a), and the sync control unit (110s, see FIG. 3) are conveniently classified to explain some of the functions of at least one processor (110), and the function of the processing unit (110p, see FIG. 3), the function of the video processor (110v, see FIG. 3), the function of the audio processor (110a), and the function of the sync control unit (110s, see FIG. 3) may be performed by processors having different cores, may be performed by one core among a plurality of cores included in a multi-core processor, or may be performed by a plurality of cores.

[0100] FIG. 3 is a conceptual diagram for explaining in more detail an example of the configuration of a content provision system according to one embodiment.

[0101] Referring to FIG. 3, a source device (3) can transmit source frame data to a second electronic device (2) via wired communication. The source frame data can include source audio frame data and source video frame data.

[0102] The preprocessing unit (210p) of the second electronic device (2) can preprocess source frame data. Preprocessing the source frame data may include converting the source frame data into a state that can be processed by the video processor (210v) of the second electronic device (2) and / or converting the source frame data into a state that can be processed by the audio processor (210a) of the second electronic device (2).

[0103] For example, preprocessing the source frame data may include classifying the source frame data into source video frame data and source audio frame data.

[0104] The video processor (210v) of the second electronic device (2) can process source video frame data. For example, the video processor (210v) can encode the source video frame data. To this end, the video processor (210v) can include a video encoder. Encoding the source video frame data can include compressing a plurality of source video frames frame by frame.

[0105] In one embodiment, the video processor (210v) may assign an index to source video frame data.

[0106] Assigning an index to the source video frame data may include assigning an index to a header of the source video frame data.

[0107] In one embodiment, the video processor (210v) may assign different indices to different video frame data.

[0108] For example, a first index may be assigned to first video frame data corresponding to a first video frame, and a second index may be assigned to second video frame data corresponding to a second video frame consecutive to the first video frame.

[0109] Accordingly, each header area of ​​source video frame data may be assigned a different index.

[0110] An index may contain numeric data, but the form of the index is not limited to this.

[0111] When the index includes numeric data, consecutive numbers may be assigned to consecutive video frames. For example, the first video frame data corresponding to the first video frame may be assigned an index '1', the second video frame data corresponding to the second video frame consecutive to the first video frame may be assigned an index '2', and the third video frame data corresponding to the third video frame consecutive to the second video frame may be assigned an index '3'.

[0112] According to various embodiments, the second electronic device (2) may initialize an index assigned to video frame data and / or audio frame data based on satisfaction of a preset condition.

[0113] As another example, the second electronic device (2) can initialize the index given to the video frame data and / or audio frame data at preset intervals.

[0114] Accordingly, based on the index assigned to the video frame data and / or audio frame data exceeding a predetermined value, the newly assigned index may start again from 1. In other words, the index value in the video frame data and / or audio frame data may not increase indefinitely but may be reset at predetermined intervals.

[0115] For convenience of explanation, the source video frame data processed by the video processor (210v) is referred to as video frame data below.

[0116] In one embodiment, the video frame data may be stored in memory (220) and then transmitted wirelessly to the first electronic device (1) via a communication interface (230).

[0117] The audio processor (210a) of the second electronic device (2) can process source audio frame data. For example, the audio processor (210a) can format the source audio frame data. For this purpose, the audio processor (210a) can include an audio formatter.

[0118] Formatting the source audio frame data may include converting the source audio frame data to a particular format or specification, or encoding the source audio frame data.

[0119] In one embodiment, the audio processor (210a) may assign an index to the source audio frame data.

[0120] Assigning an index to the source audio frame data may include assigning an index to a header of the source audio frame data.

[0121] In one embodiment, the audio processor (210a) may assign different indices to audio frame data corresponding to different video frames.

[0122] For example, a first index may be assigned to the first audio frame data corresponding to the first video frame, and a second index may be assigned to the second audio frame data corresponding to the second video frame consecutive to the first video frame.

[0123] According to various embodiments, there may be multiple audio frames corresponding to a single video frame. For example, an audio frame corresponding to a first video frame may include a first audio frame and a second audio frame.

[0124] In one embodiment, the audio processor (210a) may assign the same index to audio frame data corresponding to the same video frame.

[0125] For example, the audio processor (210a) may assign the same index to first audio frame data corresponding to the first audio frame and second audio frame data corresponding to the second audio frame.

[0126] Accordingly, each header area of ​​source audio frame data may be assigned a different index.

[0127] An index may contain numeric data, but the form of the index is not limited to this.

[0128] When the index includes numeric data, consecutive numbers may be assigned to consecutive audio frames. For example, audio frame data corresponding to at least one audio frame corresponding to a first video frame may be assigned an index '1', audio frame data corresponding to at least one audio frame corresponding to a second video frame consecutive to the first video frame may be assigned an index '2', and audio frame data corresponding to at least one audio frame corresponding to a third video frame consecutive to the second video frame may be assigned an index '3'.

[0129] For convenience of explanation, the source audio frame data processed by the audio processor (210a) is referred to as audio frame data below.

[0130] In one embodiment, audio frame data may be stored in memory (220) and then transmitted wirelessly to the first electronic device (1) via a communication interface (230).

[0131] Video frame data and audio frame data stored in the memory (220) can be transmitted to the first electronic device (1) in real time.

[0132] However, if the wireless communication network environment is not good, the transmission of video frame data and audio frame data from the second electronic device (2) to the first electronic device (1) may be delayed.

[0133] In particular, since high-definition video frame data has a large capacity, video frame data stored in the memory (220) may accumulate in a communication delay situation.

[0134] If the content transmitted from the source device (3) to the second electronic device (2) corresponds to content requiring real-time streaming, transmission of some of the video frame data stored in the memory (220) may be omitted, and transmission of recently stored video frame data may be resumed.

[0135] In one embodiment, the processor (210) may skip transmission of a portion of video frame data when a communication delay occurs and resume transmission of recently stored video frame data.

[0136] On the other hand, since the audio frame data is not large in size compared to the video frame data, it can be transmitted from the second electronic device (2) to the first electronic device (1) even if a communication delay occurs.

[0137] In one embodiment, the processor (210) may omit transmission of a portion of video frame data and maintain transmission of audio frame data when a communication delay occurs.

[0138] That is, when a communication delay occurs, the first electronic device (1) can temporarily receive only audio frame data without receiving video frame data.

[0139] In this way, the second electronic device (2) can transmit frame data to the first electronic device (1) via wireless communication. The frame data may include audio frame data and / or video frame data.

[0140] The communication interface (130) of the first electronic device (1) can receive frame data from the communication interface (230) of the second electronic device (2) using wireless communication.

[0141] The preprocessing unit (110p) of the first electronic device (1) can preprocess frame data. Preprocessing the frame data may include converting the frame data into a state that can be processed by the video processor (110v) of the first electronic device (1) and / or converting the frame data into a state that can be processed by the audio processor (110a) of the first electronic device (1).

[0142] For example, preprocessing frame data may include classifying the frame data into video frame data and audio frame data.

[0143] The video processor (110v) of the first electronic device (1) can process video frame data. For example, the video processor (110v) can decode the video frame data. To this end, the video processor (110v) can include a video decoder. Decoding the video frame data can include decompressing encoded video frame data. The encoded video frame data can include a plurality of video frames.

[0144] Decoding the video frame data may include reading metadata included in the encoded video frame data.

[0145] In one embodiment, the video processor (110v) can identify an index assigned to video frame data.

[0146] Identifying an index assigned to the video frame data may include reading a value stored in a header of the video frame data.

[0147] As previously explained, the header of the video frame data corresponding to each video frame of the plurality of video frames may be assigned a different index.

[0148] For example, a first video frame data corresponding to a first video frame may be assigned a first index, and a second video frame data corresponding to a second video frame consecutive to the first video frame may be assigned a second index.

[0149] Meanwhile, if consecutive video frames are not transmitted due to communication delay, the index values ​​assigned to multiple video frames may be discontinuous.

[0150] In one embodiment, video frame data may be stored in memory (120), and a video frame corresponding to the video frame data may be output through a content playback device (125).

[0151] Outputting a video frame corresponding to video frame data through a content playback device (125) may include displaying an image corresponding to the video frame data through a display of the content playback device (125).

[0152] The audio processor (110a) of the first electronic device (1) can process audio frame data. For example, the audio processor (110a) can parse audio frame data. To this end, the audio processor (110a) can include an audio parser.

[0153] Parsing audio frame data may include converting the audio frame data to a specific format or specification, or encoding the audio frame data. Parsing the audio frame data may include reading metadata contained in the formatted audio frame data.

[0154] In one embodiment, the audio processor (110a) can identify an index assigned to audio frame data.

[0155] Identifying an index assigned to audio frame data may include reading a value stored in a header of the audio frame data.

[0156] In one embodiment, the audio processor (110a) may assign different indices to audio frame data corresponding to different video frames.

[0157] As previously explained, the headers of audio frame data corresponding to each video frame of multiple video frames may be assigned different indices.

[0158] For example, at least one audio frame data corresponding to a first video frame may be assigned a first index, and at least one audio frame data corresponding to a second video frame consecutive to the first video frame may be assigned a second index.

[0159] Meanwhile, even if consecutive video frames are not transmitted from the second electronic device (2) to the first electronic device (1) due to a communication delay, audio frames having a relatively low capacity can be transmitted from the second electronic device (2) to the first electronic device (1).

[0160] That is, even if consecutive video frames are not transmitted due to communication delay, the index values ​​assigned to multiple audio frames can be consecutive.

[0161] In one embodiment, audio frame data may be stored in memory (120), and audio corresponding to the audio frame data may be output through a content playback device (125).

[0162] Outputting an audio frame corresponding to audio frame data through a content playback device (125) may include outputting a sound corresponding to the audio frame data through a speaker of the content playback device (125).

[0163] The sync control unit (110s) compares the index of the video frame data with the index of the audio frame data, and can synchronize the video and audio played by the content playback device (125) based on the comparison result.

[0164] In one embodiment, the sync control unit (110s) can control the playback speed of the video and / or the playback speed of the audio based on the result of comparing the index of the video frame data with the index of the audio frame data.

[0165] The content playback device (125) can play video and audio at a playback speed adjusted by the sync control unit (110s) based on video frame data and audio frame data.

[0166] FIG. 4 is a flowchart illustrating a method for providing content by a content providing system according to one embodiment.

[0167] Referring to FIG. 4, the source device (3) can transmit source frame data to the second electronic device (2) via wired communication (S1).

[0168] The second electronic device (2) can process source frame data (S2).

[0169] The operation (S2) of processing the source frame data may include encoding the source video frame data and assigning an index to the source video frame data.

[0170] The operation (S2) of processing source frame data may include formatting the source audio frame data and assigning an index to the source audio frame data.

[0171] The second electronic device (2) can process source frame data and transmit the processed data (frame data) to the first electronic device (1) via wireless communication (S3).

[0172] Frame data may include video frame data and audio frame data.

[0173] In one embodiment, the operation (S3) of the second electronic device (2) transmitting frame data to the first electronic device (1) may include an operation of the second electronic device (2) transmitting frame data containing only audio frame data when a wireless communication delay occurs.

[0174] That is, the second electronic device (2) can omit transmission of video frame data when a wireless communication delay occurs.

[0175] In one embodiment, the operation (S3) of the second electronic device (2) transmitting frame data to the first electronic device (1) may include an operation of the second electronic device (2) transmitting frame data including both video frame data and audio frame data when no wireless communication delay occurs.

[0176] The first electronic device (1) can process frame data received from the second electronic device (2) (S4).

[0177] The operation of processing frame data (S4) may include an operation of comparing an index of video frame data with an index of audio frame data.

[0178] The first electronic device (1) can play content based on video frame data and audio frame data (S5).

[0179] The operation (S5) of playing content based on video frame data and audio frame data may include an operation of playing video and audio at a first playback speed based on the index of the video frame data being the same as the index of the audio frame data.

[0180] At this time, the first playback speed can be defined as the basic speed, 1x speed.

[0181] The operation (S5) of playing back content based on video frame data and audio frame data may include an operation of maintaining the playback speed of the video at a first playback speed and playing back the audio at a second playback speed that is faster than the first playback speed based on the index of the video frame data being different from the index of the audio frame data.

[0182] The operation of playing back audio at a second playback speed that is faster than the first playback speed may include downsampling the audio frame data. Downsampling the audio frame data may include lowering the sampling rate of the audio frame data. In this case, the second playback speed may be preset to be faster than the first playback speed.

[0183] In one embodiment, the second playback speed may be a preset speed (e.g., 1.5x speed).

[0184] In one embodiment, the second playback speed may be set by the user.

[0185] In one embodiment, the first electronic device (1) can determine the second playback speed based on the difference between the index of the video frame data and the index of the audio frame data. In this case, the second playback speed can be variable.

[0186] For example, the first electronic device (1) can determine the second playback speed faster as the difference between the index of the video frame data and the index of the audio frame data becomes larger.

[0187] The first electronic device (1) can determine the second playback speed as a first speed corresponding to the first value when the difference between the index of the video frame data and the index of the audio frame data is a first value, and can determine the second playback speed as a second speed corresponding to the second value when the difference between the index of the video frame data and the index of the audio frame data is a second value greater than the first value. In this case, the second speed can be faster than the first speed.

[0188] According to the present disclosure, when a portion of video frame data fails to be transmitted due to a communication delay, audio is played at a double speed when the video is played, allowing a user to enjoy content without audio loss.

[0189] In one embodiment, the act of playing content based on video frame data and audio frame data may include the act of not playing both video and audio if reception of the video frame data is delayed.

[0190] That is, if the video doesn't play, the audio won't play either, and when the video resumes playing, the audio may play at a faster than default speed.

[0191] FIG. 5 is a flowchart illustrating an example of data processing performed by a second electronic device according to one embodiment. FIG. 6 is a conceptual diagram illustrating an example of data processing performed by a second electronic device according to one embodiment.

[0192] Referring to FIGS. 5 and 6, in operation S2, the second electronic device (2) can receive source frame data (SF) from the source device (3) (1000).

[0193] Source frame data (SF) may include source audio frame data (AFP) and source video frame data (VFP).

[0194] For example, source frame data (SF) corresponding to source frame 1 (Source #1) may include source video frame data (VFP) corresponding to video frame 1 (Video #1) and source audio frame data (AFP) corresponding to audio frame 1 (Audio #1).

[0195] In this way, source frame data (SF) corresponding to source frame 2, 3, or 4 (Source #2, Source #3, or Source #4) may include source video frame data (VFP) corresponding to video frame 2, 3, or 4 (Video #2, Video #3, or Video #4) and source audio frame data (AFP) corresponding to audio frame 2, 3, or 4 (Audio #2, Audio #3, or Audio #4).

[0196] In one embodiment, the operation (1000) of receiving source frame data (SF) may include receiving first source frame data (Source #1) including first source video frame data (Video #1) and first source audio frame data (Audio #1), and sequentially receiving second source frame data (Source #2) including second source video frame data (Video #2) and second source audio frame data (Audio #2).

[0197] As described above, according to various embodiments, the source frame data (SF) corresponding to source frame 1 (Source #1) may include source video frame data (VFP) corresponding to video frame 1 (Video #1) and source audio frame data (AFP) corresponding to multiple audio frames (e.g., Audio #1 and Audio #2).

[0198] In one embodiment, the operation (1000) of receiving source frame data (SF) may include receiving first source frame data (Source #1) including first source video frame data (Video #1) and first source audio frame data (Audio #1), and sequentially receiving second source frame data (Source #2) including second source video frame data (Video #2), second source audio frame data (Audio #2), and third source audio frame data (Audio #3).

[0199] In the present disclosure, for convenience of explanation, it is assumed that source frame data (SF) corresponding to source frame n (n is an integer, Source #n) includes source video frame data (VFP) corresponding to video frame n (Video #n) and source audio frame data (AFP) corresponding to audio frame n (Audio #n).

[0200] The preprocessing unit (210p) can preprocess source frame data (SF) and obtain source audio frame data (AFP) and source video frame data (VFP).

[0201] The preprocessing unit (210p) can transmit source audio frame data (AFP) to an audio processor (220a) and source video frame data (VFP) to a video processor (220v).

[0202] The second electronic device (2) can assign the same index (id) to the source audio frame data (AFP) and the source video frame data (VFP) included in the source frame data (SF) (1100).

[0203] The second electronic device (2) can encode source audio frame data (AFP) and source video frame data (VFP) (1200).

[0204] In the present disclosure, an operation (1200) of encoding source audio frame data (AFP) and source video frame data (VFP) may also include an operation (1100) of assigning the same index (id) to the source audio frame data (AFP) and the source video frame data (VFP).

[0205] In one embodiment, the audio processor (210a) may assign an index (id) to the source audio frame data (AFP). In one embodiment, the audio processor (210a) may format the source audio frame data (AFP).

[0206] In one embodiment, the video processor (210v) may assign an index (id) to the source video frame data (VFP). In one embodiment, the video processor (210v) may encode the source video frame data.

[0207] The audio processor (210a) and the video processor (210v) can assign the same index (id) to the source video frame data (VFP) and source audio frame data (AFP) corresponding to the same source frame.

[0208] In one embodiment, when the second electronic device (2) receives first source frame data (e.g., Source #1) including first source video frame data (e.g., Video #1) and first source audio frame data (e.g., Audio #1), and subsequently receives second source frame data (e.g., Source #2) including second source video frame data (e.g., Video #2) and second source audio frame data (e.g., Audio #2), the operation of assigning the same index (e.g., id) to the source video frame data (e.g., VFP) and the source audio frame data (e.g., AFP) corresponding to the same source frame may include assigning a first index (e.g., Index 1) to the first source video frame data (e.g., Video #1) and the first source audio frame data (e.g., Audio #1) and assigning a second index (e.g., Index 2) different from the first index (e.g., Index 1) to the second source video frame data (e.g., Video #2) and the second source audio frame data (e.g., Audio #2). there is.

[0209] For example, the source video frame data (VFP) corresponding to video frame n (Video #n) and the source audio frame data (AFP) corresponding to audio frame n (Audio #n) may be assigned the same index (id). In addition, the source video frame data (VFP) corresponding to video frame n (Video #n) and the source video frame data (VFP) corresponding to video frame m (where m is an integer different from n, Video #m) may be assigned different indexes (id). In addition, the source audio frame data (AFP) corresponding to audio frame n (Audio #n) and the source audio frame data (AFP) corresponding to audio frame m (Audio #m) may be assigned different indexes (id).

[0210] In one embodiment, when a second electronic device (2) receives first source frame data (SF) including first source video frame data (VFP) and first source audio frame data (AFP), and sequentially receives second source frame data (SF) including second source video frame data (VFP), second source audio frame data (AFP), and third source audio frame data (AFP), the operation of assigning the same index (id) to the source video frame data (VFP) and the source audio frame data (AFP) corresponding to the same source frame may include assigning a first index (id) to the first source video frame data (VFP) and the first source audio frame data (AFP), and assigning a second index (id) different from the first index (id) to the second source video frame data (VFP), the second source audio frame data (AFP), and the third source audio frame data (AFP).

[0211] That is, when one source frame data (SF) includes multiple audio frames, the source video frame data (VFP) corresponding to one video frame (e.g., Video #1) and the source audio frame data (AFP) corresponding to multiple audio frames (e.g., Audio #1, Audio #2) may be assigned the same index (id). In addition, different video frames may be assigned different indexes (ids). In addition, audio frames corresponding to different source frames may be assigned different indexes (ids).

[0212] Source audio frame data (AFP) may be processed by an audio processor (210a) to generate audio frame data (AFE) with an index (id).

[0213] Audio frame data (AFE) can be stored in audio memory (220a).

[0214] Audio frame data (AFE) can be temporarily stored in audio memory (220a) and transmitted to the first electronic device (1) via a communication interface (230).

[0215] Source video frame data (VFP) may be processed by a video processor (210v) to generate video frame data (VFE) with an index (id).

[0216] Video frame data (VFE) can be temporarily stored in video memory (220v) and transmitted to the first electronic device (1) via a communication interface (230).

[0217] FIG. 7 conceptually illustrates an example of how video frame data and audio frame data are transmitted from a second electronic device (2) to a first electronic device (1) according to one embodiment.

[0218] Referring to FIG. 7, the second electronic device (2) can omit transmission of a portion of the video frame data (VFE) stored in the video memory (220v) when a communication delay occurs.

[0219] The second electronic device (2) can control the communication interface to transmit only the recently stored video frame data (VFE) to ensure real-time content when video frame data (VFE) accumulates while being stored in the video memory (220v) due to communication delay. At this time, the transmitted video frame data (VFE) and the remaining untransmitted video frame data (VFE) can be deleted from the video memory (220v).

[0220] On the other hand, even if a communication delay occurs, the audio frame data (AFE) can be completely transmitted to the first electronic device (1) through the communication interface (230). At this time, the transmitted audio frame data (AFE) can be deleted from the audio memory (220a).

[0221] For example, if a communication delay occurs and transmission of a portion of the video frame data (VFE) (e.g., Video #1, Video #2, Video #3) is omitted, only the entire audio frame data (AFE) (e.g., Audio #1, Audio #2, Audio #3, Audio #4) and the most recently stored video frame data (VFE) (e.g., Video #4) may be transmitted to the first electronic device (1) via the communication interface (230).

[0222] According to the present disclosure, the real-time nature of content provided by the first electronic device (1) can be ensured by omitting transmission of video frame data (VFE) that cannot be transmitted in real time due to communication delay by the second electronic device (2).

[0223] Fig. 8 is a flowchart illustrating an example of data processing performed by a first electronic device (1) according to one embodiment. Fig. 9 is a conceptual diagram illustrating an example of data processing performed by a first electronic device (1) according to one embodiment.

[0224] Referring to FIGS. 8 and 9, in operation S4, the first electronic device (1) can receive frame data including video frame data (VFE) and audio frame data (AFE) from the second electronic device (2) (2000).

[0225] Video frame data (VFE) may be assigned an index (id) for each video frame. Audio frame data (AFE) may be assigned an index (id) for each audio frame. In this case, the video frame data (VFE) and audio frame data (AFE) may be encoded.

[0226] The preprocessing unit (110p) can preprocess frame data. The preprocessing unit (110p) can obtain video frame data (VFE) and audio frame data (AFE).

[0227] The preprocessing unit (110p) can transmit video frame data (VFE) to the video processor (110v) and audio frame data (AFE) to the audio processor (110a).

[0228] The first electronic device (1) can decode video frame data (VFE) and audio frame data (AFE) (2100).

[0229] The first electronic device (1) can compare the index (id) of video frame data (VFE) with the index (id) of audio frame data (AFE) (2200).

[0230] In the present disclosure, an operation (2100) of decoding video frame data (VFE) and audio frame data (AFE) may include an operation (2200) of comparing an index (id) of the video frame data (VFE) with an index (id) of the audio frame data (AFE).

[0231] Audio frame data (AFE) decoded (or parsed) by the audio processor (110a) may be stored in the audio memory (120a). Audio frame data (AFE) decoded (or parsed) by the audio processor (110a) may be temporarily stored in the audio memory (120a) and may be deleted when a predetermined condition (e.g., a certain period of time has elapsed) is satisfied.

[0232] Video frame data (VFE) decoded by the video processor (110v) may be stored in the video memory (120v). Video frame data (VFE) decoded by the video processor (110v) may be temporarily stored in the video memory (120v) and may be deleted when a predetermined condition (e.g., a certain period of time has elapsed) is satisfied.

[0233] At least one processor (110) can control a content playback device (125) to play audio based on audio frame data (AFE) stored in audio memory (120a).

[0234] At least one processor (110) can control a content playback device (125) to play a video based on video frame data (VFE) stored in a video memory (120v).

[0235] In one embodiment, the sync control unit (110s) can control the playback speed of the video and the playback speed of the audio. More specifically, the sync control unit (110s) can maintain the playback speed of the video at a first playback speed and play the audio at a first playback speed or a second playback speed according to predetermined conditions.

[0236] The first electronic device (1) can play back video and audio at a first playback speed (2300) based on the index (id) of the video frame data (VFE) being the same as the index (id) of the audio frame data (AFE) (example of 2200).

[0237] The first electronic device (1) may not play back both video and audio based on the interruption of reception of video frame data (VFE).

[0238] That is, the second electronic device (2) can suspend playback of the video and the audio until the video frame data (VFE) is received, based on the fact that only the audio frame data (AFE) is received among the video frame data (VFE) and the audio frame data (AFE).

[0239] At least one processor (110) can control a content control device to stop playback of video and audio based on exhaustion of video frame data (VFE) stored in a video memory (120v). Exhaustion of video frame data (VFE) stored in a video memory (120v) may include that the number of video frames included in the video frame data (VFE) stored in the video memory (120v) is less than or equal to a predetermined number.

[0240] The first electronic device (1) can compare the index (id) of the received video frame data (VFE) with the index (id) of the audio frame data (AFE) stored in the audio memory (120a) based on the resumption of reception of the video frame data (VFE). Since the received video frame data (VFE) can also be stored in the video memory (120v), the first electronic device (1) can compare the index (id) of the video frame data (VFE) stored in the video memory (120v) with the index (id) of the audio frame data (AFE) stored in the audio memory (120a) based on the resumption of reception of the video frame data (VFE).

[0241] The first electronic device (1) can play the video at a first playback speed and play the audio at a second playback speed (2400) based on whether the index (id) of the video frame data (VFE) is different from the index (id) of the audio frame data (AFE) (No of 2200).

[0242] The first electronic device (1) can restore the audio playback speed to the first playback speed (2300) based on the fact that the index (id) of the video frame data (VFE) becomes the same as the index (id) of the audio frame data (AFE) (example of 2200) after changing the audio playback speed to the second playback speed (after operation 2400).

[0243] FIG. 10A conceptually illustrates an example of data stored in a first electronic device when the first electronic device receives video frame data and audio frame data from a second electronic device without communication delay according to one embodiment.

[0244] Referring to FIG. 10a, it can be confirmed that the index (id) of the video frame data (VFE) stored in the video memory (120v) and the index (id) of the audio frame data (AFE) stored in the audio memory (120a) match.

[0245] The sync control unit (110s) can determine that there is no problem with the synchronization of video and audio in response to identifying that the index (id) of the video frame data (VFE) stored in the video memory (120v) and the index (id) of the audio frame data (AFE) stored in the audio memory (120a) are the same.

[0246] The sync control unit (110s) can maintain the video playback speed and the audio playback speed at the first playback speed based on the index (id) of the video frame data (VFE) and the index (id) of the audio frame data (AFE) being the same.

[0247] FIG. 10b conceptually illustrates an example of data stored in a first electronic device when the first electronic device receives video frame data (VFE) and audio frame data (AFE) from a second electronic device in a communication delay situation according to one embodiment.

[0248] Referring to FIG. 10b, it can be confirmed that the index (id) of the video frame data (VFE) stored in the video memory (120v) and the index (id) of the audio frame data (AFE) stored in the audio memory (120a) are different.

[0249] The fact that the index (id) of the video frame data (VFE) stored in the video memory (120v) and the index (id) of the audio frame data (AFE) stored in the audio memory (120a) are different is an indicator that can confirm that a part of the video frame data (VFE) was not transmitted from the second electronic device (2) to the first electronic device (1) due to a communication delay.

[0250] The sync control unit (110s) can determine that there is a problem with the synchronization of video and audio in response to identifying that the index (id) of the video frame data (VFE) stored in the video memory (120v) and the index (id) of the audio frame data (AFE) stored in the audio memory (120a) are different.

[0251] The sync control unit (110s) can play the video at a first playback speed but change the audio playback speed to a second playback speed based on the difference between the index (id) of the video frame data (VFE) and the index (id) of the audio frame data (AFE).

[0252] As the playback speed of audio increases, it can be expected that the index (id) of the video frame data (VFE) to be played will become the same as the index (id) of the audio frame data (AFE).

[0253] For example, assuming that the second playback speed is twice the first playback speed, two or more audio frames may be played for each video frame played. Accordingly, a point may come when the index (id) of the video frame data (VFE) and the index (id) of the audio frame data (AFE) become identical.

[0254] The sync control unit (110s) can restore the audio playback speed to the first playback speed based on the index (id) of the video frame data (VFE) and the index (id) of the audio frame data (AFE) becoming identical, thereby synchronizing the video and audio.

[0255] According to the present disclosure, in order to ensure real-time content, audio is played in full even if a portion of the video is not played, thereby allowing a user to check audio in situations where the content is temporarily paused.

[0256] FIG. 11 illustrates an example of video and audio output by a first electronic device in a communication delay situation, according to one embodiment.

[0257] The first electronic device (1) can play video and audio based on video frame data and audio frame data received from the second electronic device (2).

[0258] As a delay occurs in wireless communication between the second electronic device (2) and the first electronic device (1), the second electronic device (2) can transmit only audio frame data to the first electronic device (1) without transmitting video frame data.

[0259] Referring to FIG. 11, the first electronic device (1) can suspend playback of audio and video based on receiving only audio frame data among the video frame data and the audio frame data.

[0260] In one embodiment, the first electronic device (1) can output a visual indication (K1) indicating a reception delay of the video frame data based on the fact that only the audio frame data is received among the video frame data and the audio frame data.

[0261] The user can check for visual indications (K1) indicating interruptions in audio and delays in receiving video frame data, and to confirm that there is an abnormality in the state of the wireless communication.

[0262] When the delay in wireless communication is resolved, the second electronic device (2) transmits video frame data to the first electronic device (1), and the video frame data that was not transmitted is consequently not transmitted to the first electronic device (1).

[0263] The first electronic device (1) can resume playback of video and audio based on the resumption of reception of video frame data from the second electronic device (2).

[0264] FIG. 12 illustrates an example of video and audio output by a first electronic device after a communication delay situation is resolved, according to one embodiment.

[0265] Referring to FIG. 12, the first electronic device (1) can output video corresponding to the video frame data when reception of the video frame data is resumed.

[0266] When reception of video frame data resumes, the index of the video frame data and the index of the audio frame data may be different.

[0267] For example, as illustrated in FIG. 12, the index of the video frame data may correspond to index 4 corresponding to video frame 4, while the index of the audio frame data may correspond to index 1 corresponding to audio frame 1.

[0268] The first electronic device (1) can maintain the playback speed of the video at the first playback speed and adjust the playback speed of the audio to the second playback speed until the index of the video frame data and the index of the audio frame data become the same.

[0269] That is, in the example shown in Fig. 12, while the image (K3) corresponding to video frame 4 is output at the first playback speed, the audio corresponding to audio frame 1 to audio frame 4 can be played at the second playback speed.

[0270] Assuming that the index of the audio frame data and the index of the video frame data become the same as index 5 as the audio is played back at the second playback speed, the first electronic device (1) can change the playback speed of the audio back to the first playback speed.

[0271] As a result, video and audio can be synchronized by having the playback speeds matched. Simultaneously, audio playback is accelerated during segments where video and audio playback was temporarily paused, allowing users to indirectly infer the content of segments of video lost due to communication delays without having to replay the video.

[0272] On the other hand, depending on the user's preference, the fast playback of audio may be uncomfortable and a secondary playback speed adjustment may be required.

[0273] FIG. 13 illustrates an example of a user interface for setting video interruption, provided by a first electronic device according to one embodiment.

[0274] Referring to FIG. 13, the first electronic device (1) can provide an interface for setting various functions that can be applied in a situation where video is interrupted due to communication delay.

[0275] For example, the first electronic device (1) may provide a first interface element (f1) for selecting a function that allows video and audio to be played together when the video is interrupted, but allows audio in the skipped section of the video to be played at a higher speed.

[0276] According to various embodiments, the first electronic device (1) may perform the operations described above only when the first interface element (f1) is selected by the user.

[0277] As another example, the first electronic device (1) may provide a second interface element (f2) for selecting a function that allows video and audio to be played together when the video is interrupted, but also skips audio for the portion of the video that was skipped.

[0278] According to various embodiments, the first electronic device (1) may not perform the operations described above when the second interface element (f2) is selected by the user.

[0279] That is, the first electronic device (1) may not perform an operation of changing the playback speed of the audio to the second playback speed when the second interface element (f2) is selected by the user.

[0280] As another example, the first electronic device (1) may provide a third interface element (f3) for selecting a function to play only audio when video is interrupted.

[0281] According to various embodiments, when the third interface element (f3) is selected by the user, the first electronic device (1) may pause only the video playback and play the audio at the first playback speed based on the fact that only the audio frame data is received among the video frame data and the audio frame data.

[0282] When the first electronic device (1) pauses only the playback of the video while playing the audio at the first playback speed, when the reception of the video frame data resumes, the index of the video frame data and the index of the audio frame data may be different.

[0283] For example, if the first electronic device (1) pauses only the playback of the video while playing the audio at the first playback speed, when the reception of the video frame data resumes, the index value of the video frame data may be greater than the index value of the audio frame data, or the index value of the audio frame data may be greater than the index value of the video frame data.

[0284] The first electronic device (1) can change the playback speed of the audio to a second playback speed in response to the index value of the video frame data being greater than the index value of the audio frame data until the index of the video frame data and the index of the audio frame data become the same.

[0285] Conversely, the first electronic device (1) may, in response to the index value of the audio frame data being greater than the index value of the video frame data, change the playback speed of the video to a second playback speed or pause playback of the audio until the index of the video frame data and the index of the audio frame data become equal.

[0286] FIG. 14 illustrates an example of a user interface for setting audio speed provided by a first electronic device according to one embodiment.

[0287] Referring to FIG. 14, according to various embodiments, the first electronic device (1) may provide interface elements (f4, f5) for setting the second playback speed.

[0288] For example, the first electronic device (1) may provide a fourth interface element (f4) for automatically selecting the second playback speed.

[0289] According to various embodiments, the first electronic device (1) may determine, when the fourth interface element (f4) is selected by the user, the second playback speed by setting the second playback speed to a predetermined default value or by determining the second playback speed to be faster the greater the difference between the index of the video frame data and the index of the audio frame data.

[0290] As another example, the first electronic device (1) may provide a fifth interface element (f5) for manually setting the second playback speed.

[0291] According to various embodiments, the first electronic device (1) may enable the user to set a second playback speed when the fifth interface element (f5) is selected by the user. At this time, the user interface provided by the first electronic device (1) may be designed so that the second playback speed can be selected as faster than the default speed of 1x. In other words, the user interface provided by the first electronic device (1) may be designed so that the second playback speed cannot be selected as slower than the default speed of 1x.

[0292] User input corresponding to selection of an interface element (e.g., f1, f2, f3, f4, f5) may be received by an input interface device provided in the first electronic device (1) or may be received via a remote controller.

[0293] According to the present disclosure, a user can change settings related to the audio speed function according to his / her preference.

[0294] A method for synchronizing audio and video according to one embodiment of the present disclosure may include: receiving, by a first electronic device (1), video frame data (VFE) and audio frame data (AFE) from a second electronic device (2) via wireless communication; reproducing video and audio at a first playback speed based on the index (id) of the video frame data (VFE) being the same as the index (id) of the audio frame data (AFE); and reproducing, by the first electronic device (1), a playback speed of the video at the first playback speed and a playback speed of the audio at a second playback speed that is faster than the first playback speed based on the index (id) of the video frame data (VFE) being different from the index (id) of the audio frame data (AFE).

[0295] The method for synchronizing the audio and video may further include: the second electronic device (2) receiving, from a source device (3), source frame data (SF) including source video frame data (VFP) and source audio frame data (AFP) through wired communication; and the second electronic device (2) assigning the same index (id) to the source video frame data (VFP) and the source audio frame data (AFP) included in the source frame data (SF) and transmitting the same index (id) to the first electronic device (1) through wireless communication.

[0296] Receiving the source frame data (SF) may include receiving first source frame data (SF) including first source video frame data (VFP) and first source audio frame data (AFP), and sequentially receiving second source frame data (SF) including second source video frame data (VFP) and second source audio frame data (AFP); and assigning the same index (id) to the source video frame data (VFP) and the source audio frame data (AFP) included in the source frame data (SF) may include assigning a first index (id) to the first source video frame data (VFP) and the first source audio frame data (AFP); and assigning a second index (id) different from the first index (id) to the second source video frame data (VFP) and the second source audio frame data (AFP).

[0297] Receiving the source frame data (SF) may include receiving first source frame data (SF) including first source video frame data (VFP) and first source audio frame data (AFP), and sequentially receiving second source frame data (SF) including second source video frame data (VFP), second source audio frame data (AFP), and third source audio frame data (AFP); and assigning the same index (id) to the source video frame data (VFP) and the source audio frame data (AFP) included in the source frame data (SF) may include assigning a first index (id) to the first source video frame data (VFP) and the first source audio frame data (AFP); and assigning a second index (id) different from the first index (id) to the second source video frame data (VFP), the second source audio frame data (AFP), and the third source audio frame data (AFP).

[0298] Maintaining the playback speed of the video at the first playback speed and playing the audio at the second playback speed may include down sampling the audio frame data (AFE).

[0299] The method for synchronizing the audio and video may further include the first electronic device (1) pausing playback of the video and the audio until the video frame data (VFE) is received, based on the fact that only the audio frame data (AFE) is received among the video frame data (VFE) and the audio frame data (AFE).

[0300] The method for synchronizing the audio and video may further include the first electronic device (1) temporarily storing the video frame data (VFE) and the audio frame data (AFE) received from the second electronic device (2) in a memory (120).

[0301] The method for synchronizing the audio and video may further include the first electronic device (1) comparing the index (id) of the received video frame data (VFE) with the index (id) of the audio frame data (AFE) stored in the memory (120) based on the resumption of reception of the video frame data (VFE).

[0302] Playing the video and the audio at the first playback speed may include, after changing the playback speed of the audio to the second playback speed, restoring the playback speed of the audio to the first playback speed when the index (id) of the video frame data (VFE) becomes the same as the index (id) of the audio frame data (AFE).

[0303] The index (id) of the video frame data (VFE) may be a value stored in the header of the video frame data (VFE) by the second electronic device (2), and the index (id) of the audio frame data (AFE) may be a value stored in the header of the audio frame data (AFE) by the second electronic device (2).

[0304] The method for synchronizing the audio and video may further include the first electronic device (1) comparing a value stored in a header of the video frame data (VFE) with a value stored in a header of the audio frame data (AFE).

[0305] The method for synchronizing the audio and video may further include the second electronic device (2) transmitting only the audio frame data (AFE) while omitting transmission of the video frame data (VFE) due to a delay in the wireless communication.

[0306] An electronic device (1) according to one embodiment of the present disclosure comprises: a content playback device (125); a communication interface (130) connected to a wireless network; a memory (120) storing at least one command; And at least one processor (110) connected to the communication interface (130) and the memory (120); wherein the at least one processor (110) receives video frame data (VFE) and audio frame data (AFE) from an external device (2) through the communication interface (130) by executing the at least one command, and controls the content playback device (125) to play back video and audio at a first playback speed based on the index (id) of the video frame data (VFE) being the same as the index (id) of the audio frame data (AFE), and controls the content playback device (125) to maintain the playback speed of the video at the first playback speed and play back the audio at a second playback speed that is faster than the first playback speed based on the index (id) of the video frame data (VFE) being different from the index (id) of the audio frame data (AFE).

[0307] The at least one processor (110) can reproduce the audio at the second reproduction speed by down-sampling the audio frame data (AFE) by executing the at least one command.

[0308] The at least one processor (110) may control the content playback device (125) to temporarily suspend playback of the video and the audio until the video frame data (VFE) is received, based on the fact that only the audio frame data (AFE) is received among the video frame data (VFE) and the audio frame data (AFE), by executing the at least one command.

[0309] The at least one processor (110) can temporarily store the video frame data (VFE) and the audio frame data (AFE) received from the external device in the memory (120) by executing the at least one command.

[0310] The at least one processor (110) may compare the index (id) of the received video frame data (VFE) with the index (id) of the audio frame data (AFE) stored in the memory (120) based on the resumption of reception of the video frame data (VFE) by executing the at least one command.

[0311] The at least one processor (110) can control the content playback device (125) to restore the playback speed of the audio to the first playback speed when the index (id) of the video frame data (VFE) becomes the same as the index (id) of the audio frame data (AFE) after changing the playback speed of the audio to the second playback speed by executing the at least one command.

[0312] The at least one processor (110) can compare a value stored in a header of the video frame data (VFE) with a value stored in a header of the audio frame data (AFE) by executing the at least one command.

[0313] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing computer-executable instructions. The instructions may be stored in the form of program code, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0314] Computer-readable storage media include all types of storage media that store instructions that can be deciphered by a computer. Examples include read-only memory (ROM), random access memory (RAM), magnetic tape, magnetic disks, flash memory, and optical data storage devices.

[0315] Additionally, a computer-readable recording medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" simply means a tangible device that does not contain signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is permanently stored in the storage medium and cases where data is temporarily stored. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0316] According to one embodiment, the method according to various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable recording medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) may be temporarily stored or temporarily generated on a machine-readable recording medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0317] The disclosed embodiments have been described with reference to the attached drawings as described above. Those skilled in the art will understand that the present invention can be implemented in forms other than the disclosed embodiments without altering the technical spirit or essential features of the present invention. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

1. A first electronic device receives video frame data and audio frame data from a second electronic device via wireless communication; The first electronic device plays back the video and audio at a first playback speed based on the index of the video frame data being the same as the index of the audio frame data; A method for synchronizing audio and video, wherein the first electronic device maintains the playback speed of the video at the first playback speed and plays the audio at a second playback speed that is faster than the first playback speed based on the index of the video frame data being different from the index of the audio frame data.

2. In paragraph 1, The second electronic device receives source frame data including source video frame data and source audio frame data from a source device via wired communication; A method for synchronizing audio and video, further comprising: the second electronic device assigning the same index to the source video frame data and the source audio frame data included in the source frame data and transmitting them to the first electronic device via wireless communication; 3. In paragraph 2, Receiving the above source frame data, Receiving first source frame data including first source video frame data and first source audio frame data, and sequentially receiving second source frame data including second source video frame data and second source audio frame data; Assigning the same index to the source video frame data and the source audio frame data included in the source frame data, Assigning a first index to the first source video frame data and the first source audio frame data; A method for synchronizing audio and video, comprising: assigning a second index different from the first index to the second source video frame data and the second source audio frame data.

4. In paragraph 2, Receiving the above source frame data, Receiving first source frame data including first source video frame data and first source audio frame data, and sequentially receiving second source frame data including second source video frame data, second source audio frame data, and third source audio frame data; Assigning the same index to the source video frame data and the source audio frame data included in the source frame data, Assigning a first index to the first source video frame data and the first source audio frame data; A method for synchronizing audio and video, comprising: assigning a second index different from the first index to the second source video frame data, the second source audio frame data, and the third source audio frame data.

5. In paragraph 1, The playback speed of the above video is maintained at the first playback speed, and the playback speed of the above audio is played at the second playback speed. A method for synchronizing audio and video, comprising: down sampling the above audio frame data.

6. In paragraph 1, A method for synchronizing audio and video, further comprising: wherein the first electronic device suspends playback of the video and the audio until the video frame data is received, based on receiving only the audio frame data among the video frame data and the audio frame data.

7. In paragraph 6, A method for synchronizing audio and video, further comprising: the first electronic device temporarily storing the video frame data and the audio frame data received from the second electronic device in a memory.

8. In paragraph 7, A method for synchronizing audio and video, further comprising: comparing an index of the received video frame data with an index of the audio frame data stored in the memory based on the resumption of reception of the video frame data by the first electronic device; 9. In paragraph 1, Playing the above video and the above audio at the first playback speed, A method for synchronizing audio and video, comprising: after changing the playback speed of the audio to the second playback speed, restoring the playback speed of the audio to the first playback speed when the index of the video frame data becomes the same as the index of the audio frame data; 10. In paragraph 1, The index of the above video frame data is, A value stored in the header of the video frame data by the second electronic device, The index of the above audio frame data is, A method for synchronizing audio and video, wherein the value is stored in the header of the audio frame data by the second electronic device.

11. In paragraph 10, A method for synchronizing audio and video, further comprising: comparing a value stored in a header of the video frame data with a value stored in a header of the audio frame data; 12. In paragraph 1, A method for synchronizing audio and video, further comprising: the second electronic device transmitting only the audio frame data while omitting transmission of the video frame data due to a delay in the wireless communication.

13. In electronic devices, Content playback device; A communication interface connected to a wireless network; a memory storing at least one instruction; and At least one processor connected to the communication interface and the memory; The at least one processor, by executing the at least one instruction, Receive video frame data and audio frame data from an external device through the above communication interface, Control the content playback device to play the video and audio at a first playback speed based on the index of the video frame data being the same as the index of the audio frame data, An electronic device that controls the content playback device so that the playback speed of the video is maintained at the first playback speed and the playback speed of the audio is played at a second playback speed that is faster than the first playback speed based on the index of the video frame data being different from the index of the audio frame data.

14. In paragraph 13, The at least one processor, by executing the at least one instruction, An electronic device that reproduces the audio at the second playback speed by downsampling the audio frame data.

15. In paragraph 13, The at least one processor, by executing the at least one instruction, An electronic device that controls the content playback device to suspend playback of the video and the audio until the video frame data is received, based on the fact that only the audio frame data is received among the video frame data and the audio frame data.

Images