Method for controlling streaming based on input and electronic device therefor

Abstract

An electronic device is disclosed comprising a wireless communication circuit, a processor, and a memory operatively connected to the processor. The electronic device may be configured to connect to an external electronic device using the wireless communication circuit, transmit audio data and video data for multimedia playback on the external electronic device using the wireless communication circuit, determine whether an input for the multimedia is detected, and, if an input for the multimedia is detected, transmit information instructing the external electronic device to stop synchronization between the audio data and the video data. In addition to this, various embodiments identified through this document are possible.

Description

Technology Field

[0001] The various embodiments disclosed in this document relate to a method for controlling streaming based on input and an electronic device for performing the method. Background Technology

[0002] Electronic devices, such as smartphones, can provide communication services with external electronic devices. For example, the electronic device can provide communication services with a mouse, keyboard, display device, printer, camera, and / or TV. As one example, the electronic device can be connected to a TV via wired and / or wireless communication and can support the output of applications running on the electronic device through the TV. As another example, the electronic device can be connected to a display device connected to a vehicle's dashboard via wired and / or wireless communication and can support the output of applications running on the electronic device through the display device.

[0003] In this regard, the electronic device may utilize mirroring or a connected communication link to output the execution screen of an application running on the electronic device through an external electronic device. For example, the electronic device may execute an application installed on the electronic device and, by using mirroring or a communication link, transmit data corresponding to the execution screen of the application to an external electronic device, thereby supporting the output of the application's execution screen on the external electronic device. In this case, the electronic device may configure the application's execution screen using resources included in the application (e.g., image files) that are tailored to the characteristics of the electronic device, such as the size of the display included in the electronic device, the resolution of the display, the display orientation, or the input processing method. The problem to be solved

[0004] An electronic device may transmit multimedia data, including audio data and / or video data, to an external display device. By transmitting multimedia data, the electronic device may mirror the screen of the electronic device to the external display device. By transmitting multimedia data, the electronic device may enable a virtual execution environment to be output to the external display device. For example, the electronic device may transmit a virtual screen (e.g., video data) and virtual audio information to the external display device. For example, the electronic device may transmit multimedia data to the external display device that executes a specified function in response to input to the virtual execution environment and displays the result of the execution of the specified function.

[0005] An external display device can receive multimedia data and perform synchronization of the audio and video data of the received multimedia data. For example, the external display device can output the audio and video data by synchronizing them by comparing the time information (e.g., timestamp) of the received audio data with the time information of the video data. The external display device can synchronize the audio and video data by buffering the audio and video data and comparing the buffered data.

[0006] To synchronize audio and video data, latency may occur between the electronic device and the external display output device. For example, even if audio data is received first, the external display device may wait for video data corresponding to the time information of the audio data to be received for synchronization. Conversely, even if video data is received first, the external display device may wait for audio data corresponding to the time information of the received video data to be received for synchronization. When a user performs input on the currently output multimedia data, the user may experience input lag due to this latency. When the user interacts with applications associated with the multimedia data, the user experience may be degraded.

[0007] The various embodiments disclosed in this document may provide electronic devices and methods for solving the problems described above. means of solving the problem

[0008] An electronic device according to one embodiment disclosed in this document comprises a wireless communication circuit, a processor, and a memory operatively connected to the processor, and the memory may store instructions such that, at execution, the processor connects to an external electronic device using the wireless communication circuit, transmits audio data and video data for multimedia playback on the external electronic device using the wireless communication circuit, determines whether an input for the multimedia is detected, and, if an input for the multimedia is detected, transmits information to the external electronic device instructing it to stop synchronization between the audio data and the video data.

[0009] A method for controlling synchronization of an electronic device according to an embodiment disclosed in this document comprises: connecting to an external electronic device; transmitting audio data and video data for multimedia playback in the external electronic device to the external electronic device; and controlling the synchronization of the external electronic device during the transmission of the audio data and the video data. The operation of controlling synchronization may include: determining whether an input for the multimedia is detected; and, when an input for the multimedia is detected, transmitting information to the external electronic device instructing it to stop the synchronization between the audio data and the video data. Effects of the invention

[0010] According to various embodiments disclosed in this document, an electronic device can reduce end-to-end latency by controlling synchronization based on input.

[0011] An electronic device according to one embodiment disclosed in this document can reduce input delay by reducing end-to-end delay.

[0012] In addition, various effects that can be identified directly or indirectly through this document may be provided. Brief explanation of the drawing

[0013] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments. FIG. 2 illustrates an input reception environment of an electronic device according to one example. FIG. 3 illustrates the structure of an electronic device and an external electronic device according to one embodiment. FIG. 4 illustrates a block diagram of an electronic device and an external electronic device according to one embodiment. FIG. 5 illustrates a user interface for connecting an external electronic device by an electronic device according to one embodiment. FIG. 6 illustrates an example of a user interface for connecting an electronic device by an external electronic device according to one embodiment. FIG. 7 illustrates an example of a user interface for connecting an electronic device by an external electronic device according to one embodiment. FIG. 8 is a signal flow diagram of a method for connecting an electronic device and an external electronic device according to one embodiment. FIG. 9 is a flowchart of a method for connecting an external electronic device and an electronic device according to one embodiment. FIG. 10 illustrates a source device environment according to one example. FIG. 11 illustrates a multiple electronic device environment according to one example. FIG. 12 illustrates a flowchart of a synchronization control method for an electronic device according to one embodiment. FIG. 13 illustrates a flowchart of a synchronization control method for an electronic device according to one embodiment. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Specific details for implementing the invention

[0014] Hereinafter, various embodiments of the present invention are described with reference to the accompanying drawings. However, this is not intended to limit the present invention to specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present invention.

[0016] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input module (150), sound output module (155), display module (160), audio module (170), sensor module (176), interface (177), connection terminal (178), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., connection terminal (178)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components (e.g., sensor module (176), camera module (180), or antenna module (197)) may be integrated into a single component (e.g., display module (160)).

[0017] The processor (120) can control at least one other component (e.g., hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can store commands or data received from other components (e.g., sensor module (176) or communication module (190)) in volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., central processing unit or application processor) or an auxiliary processor (123) that can operate independently or together with it (e.g., graphics processing unit, neural processing unit (NPU), image signal processor, sensor hub processor, or communication processor). For example, if the electronic device (101) includes a main processor (121) and an auxiliary processor (123), the auxiliary processor (123) may be configured to use lower power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0018] The auxiliary processor (123) may control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display module (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)). According to one embodiment, the auxiliary processor (123) (e.g., neural network processing unit) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. Such learning may be performed, for example, on the electronic device (101) itself where the artificial intelligence is performed, or through a separate server (e.g., server (108)). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model may include a plurality of artificial neural network layers.An artificial neural network may be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to the hardware structure, the artificial intelligence model may include a software structure, either additionally or substantially.

[0019] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, input data or output data for software (e.g., program (140)) and related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0020] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0021] The input module (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input module (150) may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0022] The sound output module (155) can output a sound signal to the outside of the electronic device (101). The sound output module (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as multimedia playback or recording playback. The receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0023] The display module (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0024] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150) or output sound through the sound output module (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) connected directly or wirelessly to the electronic device (101).

[0025] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0026] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0027] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0028] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0029] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0030] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0031] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0032] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0033] The wireless communication module (192) can support 5G networks and next-generation communication technologies following 4G networks, for example, new radio access technology. NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module (192) can support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The wireless communication module (192) can support various technologies for securing performance in the high-frequency band, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large-scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), external electronic device (e.g., electronic device (104)), or network system (e.g., second network (199)). According to one embodiment, the wireless communication module (192) can support a Peak data rate (e.g., 20 Gbps or more) for realizing eMBB, loss coverage (e.g., 164 dB or less) for realizing mMTC, or U-plane latency (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) for realizing URLLC.

[0034] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as part of the antenna module (197).

[0035] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high frequency band (e.g., mmWave band), and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top surface or side surface) of the printed circuit board and capable of transmitting or receiving a signal of the specified high frequency band.

[0036] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0037] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, or 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device (101) may provide ultra-low latency services using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server using machine learning and / or neural networks. According to one embodiment, the external electronic device (104) or the server (108) may be included within a second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0038] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0039] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0040] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0041] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0042] According to one embodiment, the method according to the various embodiments disclosed herein may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0043] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0044] FIG. 2 illustrates an input reception environment of an electronic device according to one example.

[0045] An electronic device (201) (e.g., the electronic device (101) of FIG. 1) can communicate with an external electronic device (202). The electronic device (201) can be configured to act as a source device to transmit multimedia data to at least one client device (e.g., the external electronic device (202)). The electronic device (201) can be configured to transmit multimedia data to the external electronic device (202) via a wired connection and / or a wireless connection. For example, the wireless connection may be based on a short-range wireless protocol (e.g., a WiFi-based protocol).

[0046] The external electronic device (202) is an electronic device including a display and can display received multimedia data on the display. The external electronic device (202) can output audio data and / or video data included in the multimedia data. The external electronic device (202) can perform synchronized output of audio data and video data.

[0047] The electronic device (201) may transmit multimedia data to mirror the screen of the electronic device (201) to an external electronic device (202) or to provide a virtual execution environment to the external electronic device (202). For example, the screen displayed on the external electronic device (202) (e.g., mirrored screen and / or virtual execution environment) may include an execution screen of at least one application. The multimedia data may include the execution screen of the at least one application. In the example of FIG. 2, the external electronic device (202) may display a first window (210) and a second window (220). For example, the first window (210) may correspond to an execution screen of a first application running on the electronic device (201), and the second window (220) may correspond to an execution screen of a second application running on the electronic device (201).

[0048] A user of the electronic device (201) can perform an interaction with a screen displayed on an external electronic device (202). The user can perform any input for the interaction. The electronic device (201) can obtain user input using various methods.

[0049] For example, the electronic device (201) may obtain user input from input to the electronic device (201). Touch input by a hand (282) on the electronic device (201) may be user input. Input using a stylus (281) on the electronic device (201) may be user input. Input via an external input device (e.g., keyboard (283) and / or mouse (284)) connected to the electronic device (201) via wired or wireless connection may be user input. In one example, the electronic device (201) may operate as a trackpad for an execution screen displayed on an external electronic device (202). In one example, the electronic device (201) may display an input interface (e.g., keyboard interface) and obtain user input from input to the input interface.

[0050] For example, an electronic device (201) can obtain user input from an input to an external electronic device (202). For example, the external electronic device (202) may be connected wired and / or wirelessly to at least one input device (e.g., keyboard (283), mouse (284), and / or remote controller (not shown)). When the external electronic device (202) receives input from at least one input device, it may transmit information about the user input to the electronic device (201). The electronic device (201) can obtain user input by receiving information about the user input from the external electronic device (202).

[0051] The electronic device (201) can execute a specified function based on user input and transmit multimedia data corresponding to the execution of the specified function to an external electronic device (202). For example, if the user input is an input corresponding to the movement of a pointer (299), the electronic device (201) can transmit multimedia data corresponding to the movement of the pointer (299) to an external electronic device (202). The user can obtain visual feedback corresponding to their input by confirming the movement of the pointer (299) through the external electronic device (202).

[0052] An external electronic device (202) can perform synchronization of audio data and video data included in multimedia data. For example, the external electronic device (202) can synchronize audio data and video data by creating a buffer for synchronization (e.g., a jitter buffer) and comparing time information (e.g., a timestamp) of audio data and video data stored in the buffer. When performing synchronization of audio data and video data, end-to-end delay may be increased. In the disclosure below, the electronic device (201) can improve the user experience by controlling the synchronization of the external electronic device (202) based on user input.

[0053] FIG. 3 illustrates the structure of an electronic device and an external electronic device according to one embodiment.

[0054] The electronic device (201) may include a processor (320) (e.g., processor (120) of FIG. 1), memory (330) (e.g., memory (130) of FIG. 1), a display (360) (e.g., display module (160) of FIG. 1), and / or a communication circuit (390) (e.g., communication module (190) of FIG. 1). The processor (320) is operatively connected to other components of the electronic device (201) and can control various operations of the electronic device (201). The processor (320) can perform various operations of the electronic device (201) by executing one or more instructions stored in the memory (330). Hereinafter, operations described as being performed by the electronic device (201) may be referred to as being performed by the processor (320). The memory (330) is operatively connected to at least the processor (320) and can store instructions. The memory (330) can store various information. For example, the memory (330) can store information (e.g., Bluetooth address information) of an external electronic device (202) for sharing multimedia data with the electronic device (201). The communication circuit (390) can support communication based on a plurality of communication protocols. For example, the plurality of communication protocols may include Bluetooth (e.g., Bluetooth legacy and / or Bluetooth low energy), Wi-Fi (e.g., Wi-Fi Direct), NFC (near field communication), and / or cellular.

[0055] The external electronic device (202) may include a processor (321) (e.g., processor (120) of FIG. 1), memory (331) (e.g., memory (130) of FIG. 1), a display (361) (e.g., display module (160) of FIG. 1), and / or a communication circuit (391) (e.g., communication module (190) of FIG. 1). The external electronic device (202) may be a device configured to output multimedia data as a device separate from the electronic device (201). The processor (321) may perform various operations of the external electronic device (202) by executing one or more instructions stored in the memory (331). In the following, the operations described as being performed by the external electronic device (202) may be referred to as being performed by the processor (321). The memory (331) may be operatively connected to at least the processor (321) and may store instructions. The memory (332) can store various information. For example, the memory (332) can store identification information of the electronic device (201). The display (360) can be operatively connected to the processor (320) and can visually display information according to the control commands of the processor (320). The display (360) can receive or provide at least one piece of information to the user. For example, the display (360) can receive or provide information to the user for providing a mirroring and / or virtual execution environment. The communication circuit (390) can support communication based on a plurality of communication protocols. For example, the plurality of communication protocols may include Bluetooth (e.g., Bluetooth legacy and / or Bluetooth low energy), Wi-Fi (e.g., Wi-Fi Direct), NFC (near field communication), and / or cellular.

[0056] An electronic device (201) can be connected to an external electronic device (202) using a communication circuit (390). The electronic device (201) can recognize the external electronic device (202) by transmitting a scanning signal using the communication circuit (390) and receiving a response signal thereto. The electronic device (201) can store information of the external electronic device (202) included in the response signal (e.g., address information of the external electronic device (202)) in memory (330). The electronic device (201) can transmit a connection request including an identifier of the electronic device (201) to the external electronic device (202) and connect with the external electronic device (202) based on the response to the connection request. For example, the identifier of the electronic device (201) may be a unique identifier of the electronic device (201) associated with an application for sharing multimedia data (e.g., Samsung DeX). After connecting with the external electronic device (202), the electronic device (201) can transmit multimedia data to the external electronic device (202).

[0057] An external electronic device (202) can be connected to an electronic device (201) using a communication circuit (391). The external electronic device (202) can recognize the electronic device (201) by transmitting a signal based on a second protocol (e.g., Bluetooth protocol) based on user input and receiving a response signal based on the second protocol from the electronic device (201). In one example, when the electronic device (201) receives a signal based on the second protocol from the external electronic device (202), it can compare the information (e.g., address) of the external electronic device (202) with the information stored in the memory (330) and transmit a response signal if it has previously been connected to the external electronic device (202). The external electronic device (202) can provide information of the recognized electronic device (201) to a display (361) and transmit a message based on the second protocol to the electronic device (201) based on user input. For example, the message may include information for a connection based on a first protocol (e.g., Wi-Fi protocol). After connecting with the electronic device (201), the external electronic device (202) can receive multimedia data from the electronic device (201) and output the received multimedia data.

[0058] The electronic device (201) can control video and audio synchronization of the external electronic device (202) based on user input. As described above in relation to FIG. 2, the electronic device (201) can obtain user input from input to the electronic device (201) or the external electronic device (202). For example, the electronic device (201) can transmit a signal to the external electronic device (202) instructing the external electronic device (202) to stop synchronization when user input is detected during multimedia data sharing with the external electronic device (202). When the external electronic device (202) stops synchronization, the end-to-end delay between the electronic device (201) and the external electronic device (202) can be reduced by about 50%. The user may experience almost no input lag by immediately confirming visual feedback through the external electronic device (202) regarding the user input. After transmitting a synchronization interruption signal, the electronic device (201) may transmit a signal to an external electronic device (202) instructing to resume synchronization if no input is detected within a specified time from the last detected input.

[0059] The memory (330) may store instructions that cause the processor (320) to perform the operations described below during execution. The instructions may cause the processor (320) to connect with an external electronic device (202) using a communication circuit (390) during execution, transmit audio data and video data for multimedia playback on the external electronic device (202) using the communication circuit (390), determine whether an input for the multimedia is detected, and, if an input for the multimedia is detected, transmit information instructing the external electronic device (202) to stop synchronization between the audio data and the video data. For example, the multimedia may correspond to mirroring of the display screen of the electronic device or a virtual execution environment provided by the electronic device. The input for the multimedia may include an input for the execution environment of the mirrored electronic device (201) or an input for the virtual execution environment.

[0060] The above instructions may cause the processor (320) to detect an input to the multimedia based on an input to the electronic device)2-1_ or an input to the external electronic device (202) when executed.

[0061] The above instructions may cause the processor (320) to start a timer when an input for the multimedia is detected during execution. The above instructions may cause the processor (320) to restart the timer when a subsequent input for the multimedia is detected while the timer is running, and to transmit a signal instructing the external electronic device to resume synchronization between the audio data and the video data when the timer expires.

[0062] The above instructions may cause the processor, when executed, to receive a first advertisement signal based on a second protocol from the external electronic device (202), and if the address of the external electronic device indicated by the first advertisement signal corresponds to an address stored in the memory, to transmit a second advertisement signal corresponding to the first advertisement signal. The above instructions may cause the processor (320), when executed, to receive a connection request from the external electronic device using the second protocol, to connect with the external electronic device (202) using the first protocol via the communication circuit (390).

[0063] The above instructions may cause the processor (320) to transmit the audio data and the video data based on RTP (real-time transport protocol) upon execution. Each of the audio data and the video data may include time information in a header.

[0064] The components of the electronic device (201) and the external electronic device (202) shown in FIG. 3 are exemplary and the examples in this document are not limited thereto. For example, the electronic device (201) and / or the external electronic device (202) may further include components not shown in FIG. 3.

[0065] FIG. 4 illustrates a block diagram of an electronic device and an external electronic device according to one embodiment.

[0066] The configurations of the electronic device (201) and the external electronic device (202) illustrated in FIG. 4 may, for example, be software modules. As another example, at least some of the configurations may be modules implemented in hardware.

[0067] In one example, the electronic device (201) may include an audio and video encoder (410) (hereinafter, encoder (410)), a communication module (420), a synchronization control module (430), and / or an input detection module (440).

[0068] The encoder (410) can encode multimedia data to be shared with an external electronic device (202) according to a specified audio codec and / or video codec. The encoder (410) can transmit the encoded multimedia data to a communication module (420).

[0069] The communication module (420) can transmit multimedia data encoded according to a specified protocol (e.g., real-time transport protocol) to an external electronic device (202). For example, the communication module (420) can packetize video data and audio data according to a specified protocol and transmit the packetized data to an external electronic device (202) through a network interface. The communication module (420) can include time information (e.g., a timestamp) indicating the sampling time in the packet header.

[0070] The input detection module (440) can detect user input associated with currently shared multimedia data. For example, the user input may be an input that causes a visual change in the shared multimedia data. In the case of user input unrelated to currently shared multimedia data, the input detection module (440) may ignore the user input. As described above in relation to FIG. 2, the input detection module (440) can detect user input from an input to an electronic device (201), an external input device, or an external electronic device (202). When user input is detected, the input detection module (440) may transmit a signal indicating user input detection to the synchronization control module (430).

[0071] The synchronization control module (430) can control the synchronization of the external electronic device (202). For example, when the synchronization control module (430) receives a signal instructing the detection of user input, it can transmit a signal instructing the interruption of synchronization to the external electronic device (202) using the communication module (420). After transmitting the signal instructing the interruption of synchronization, the synchronization control module (430) can start a timer of a specified time length. The synchronization control module (430) can be configured to update the timer whenever an input is detected. When the timer expires, the synchronization control module (430) can transmit a signal instructing the resumption of synchronization to the external electronic device (202) using the communication module (420).

[0072] A communication module (421) can receive a signal from an electronic device (201). The communication module (421) can transfer multimedia data received from the electronic device (201) to a data buffer (451). The data buffer (451) (e.g., a jitter buffer) may store the received multimedia data in a specified size. A video decoder (461) can decode video data in the data buffer (451), and an audio decoder (462) can decode audio data in the data buffer (451).

[0073] The synchronization module (471) can compare time information (e.g., timestamps) of video data and audio data within the data buffer (451) and synchronize the video data and audio data based on the time information. The synchronization module (471) can synchronize the audio data and video data by causing the audio data and video data having corresponding time information to be output simultaneously. The synchronization module (471) can transmit the synchronized video data and audio data to the video renderer (481) and the audio renderer (482), respectively. The video renderer (481) can output video data by rendering the received video data, and the audio renderer (482) can output audio data by rendering the audio data.

[0074] The synchronization control module (431) can control the synchronization of audio data and video data of an external electronic device (202). When the connection of the electronic device (201) is initiated, the synchronization control module (431) can initiate the synchronization of audio data and video data. When a signal instructing the electronic device (201) to stop synchronization is received, the synchronization control module (431) can cause the audio data and video data stored in the data buffer (451) to be output sequentially without performing synchronization. When a signal instructing the electronic device (201) to resume synchronization is received, the synchronization control module (431) can cause the synchronization module (471) to synchronize the video data and audio data.

[0075] FIG. 5 illustrates a user interface for connecting an external electronic device by an electronic device according to one embodiment.

[0076] Referring to reference numeral 501, according to one example, an electronic device (201) may display a control UI (510) for controlling the functions of the electronic device (201) on a display (360). The control UI (510) may include a sharing menu (511) for activating a function or application for sharing multimedia data (e.g., mirroring and / or providing a virtual execution environment). Based on user input to the sharing menu (511), the electronic device (201) may provide a sharing UI for sharing multimedia data (e.g., the UI of reference numeral 503) through the display (360).

[0077] Referring to reference number 503, the electronic device (201) may provide a guide UI (520) and an external electronic device list (530). The guide UI (520) may include, for example, guidance information for functions or applications for sharing multimedia data. The external electronic device list (530) may include information (531) of external electronic devices recognized by the electronic device (201). For example, the electronic device (201) may display only the information of external electronic devices that have previously been connected among the recognized external electronic devices in the external electronic device list (530). Based on a selection input for one of the external electronic devices displayed in the external electronic device list (530), the electronic device (201) may connect with the selected external electronic device.

[0078] The electronic device (201) disclosed in FIG. 5 has a bar-type or plate-type appearance, but the present invention is not limited thereto. For example, the illustrated electronic device (201) may be part of a rollable electronic device or a foldable electronic device. "Rollable electronic device" may mean an electronic device in which a display (e.g., the display (360) in FIG. 3) is capable of bending deformation so that at least a portion can be wound or rolled or stored inside a housing (not shown). Depending on the user's needs, the rollable electronic device may be used to expand the screen display area by unfolding the display or by exposing a larger area of ​​the display to the outside. "Foldable electronic device" may mean an electronic device in which two different areas of the display can be folded to face each other or in opposite directions. Generally, in a portable state, the display of a foldable electronic device is folded so that two different regions face each other or in opposite directions, and in an actual usage state, the user can unfold the display so that the two different regions form a substantially flat shape. In some embodiments, the electronic device (201) according to the various embodiments disclosed in this document may be interpreted to include not only portable electronic devices such as smartphones, but also various other electronic devices such as laptop computers or home appliances.

[0079] FIG. 6 illustrates an example of a user interface for connecting an electronic device by an external electronic device according to one embodiment.

[0080] Referring to reference number 601, for example, an external electronic device (202) may display a selection UI (610) on a display (361) for selecting a remote access source device of the external electronic device (202). For example, the selection UI (610) may include a remote access menu (611) for remote access. Based on user input to the remote access menu (611), the external electronic device (202) may provide a UI for selecting a remote access source device (e.g., the UI of reference number 603).

[0081] Referring to reference number 603, the external electronic device (202) may display a list of remote access source devices. When input is received for a designated mirroring and / or virtual environment provision menu (613) among the list of remote access sources, the external electronic device (202) may display a UI such as that of FIG. 7. The mirroring and / or virtual environment provision menu (613) may be a user interface for providing a function to share multimedia data to mirror the screen of a remote access source device (e.g., the electronic device (201) of FIG. 2 or FIG. 3) to the external electronic device (202) or to provide a virtual execution environment to the external electronic device (202). The multimedia data may include video data and audio data corresponding to the virtual execution environment or mirroring.

[0082] FIG. 7 illustrates an example of a user interface for connecting an electronic device by an external electronic device according to one embodiment.

[0083] Referring to reference number 701, the external electronic device (202) may display a list of recognized remote access source devices. In the example of FIG. 7, the external electronic device (202) may display information about a first source device (e.g., Galaxy Note 20) and a second source device (e.g., Galaxy S10). When input to the connection menu (711) for the first source device is received, the external electronic device (202) may attempt to connect to the first source device. When input to the connection menu (712) for the second source device is received, the external electronic device (202) may attempt to connect to the second source device.

[0084] FIG. 8 is a signal flow diagram of a method for connecting an electronic device and an external electronic device according to one embodiment.

[0085] In one embodiment, the operations illustrated in FIG. 8 may be performed by a processor of an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2 and 3) (e.g., the processor (120) of FIG. 1 or the processor (320) of FIG. 3)) and a processor of an external electronic device (e.g., the external electronic device (202) of FIG. 3) (e.g., the processor (321) of FIG. 3).

[0086] In operation 805, the electronic device (201) can transmit (e.g., broadcast) a scanning signal. For example, the electronic device (201) can transmit the scanning signal based on a first protocol (e.g., Wi-Fi or Wi-Fi Direct). In one example, the electronic device (201) can transmit the scanning signal based on an input to the menu (511) of FIG. 5.

[0087] In operation 810, the external electronic device (202) may transmit a scanning response signal for the received scanning signal. In one example, the scanning response signal may include information of the external electronic device (202). The information of the external electronic device (202) may include, for example, a Bluetooth MAC (media access control) address. The information of the external electronic device (202) may further include the MAC address of the external electronic device (202) (e.g., Wi-Fi MAC address, Bluetooth MAC address, and / or BLE MAC address), name, and / or icon information. The electronic device (201) may store the received information of the external electronic device (202) in memory (e.g., memory (330) of FIG. 3). For example, in operation 815, the electronic device (201) may store the address of the external electronic device (202). Through operations 805 and 810, the electronic device (201) can obtain information (e.g., MAC address, name, and / or icon information) of an external electronic device (202). Operations 805 and 810 may be referred to as provisioning (801).

[0088] In operation 820, the electronic device (201) can display information about an external electronic device (202). For example, the electronic device (201) can display a list of external electronic devices (202) that have received a response signal (e.g., the list of external electronic devices (530) of FIG. 5) on a display (e.g., the display (360) of FIG. 3).

[0089] In operation 825, the electronic device (201) may receive an input selecting an external electronic device (202) from a list of external electronic devices. Based on the selection input of the external electronic device (202), in operation 830, the electronic device (201) may transmit a connection request signal to the external electronic device (202). Based on the selection input of the external electronic device (202), the electronic device (201) may generate unique identifier information for providing a mirroring screen and / or virtual execution environment associated with the electronic device (201). The connection request signal may include the unique identifier information of the electronic device (201). For example, the unique identifier information may be an identifier of the electronic device (201) associated with an application for providing a mirroring and / or virtual execution environment. Based on the received connection request signal, the external electronic device (202) may store the unique identifier information of the electronic device (201) in memory (e.g., memory (332) of FIG. 3). For example, in operation 835, the external electronic device (202) may store unique identifier information of the electronic device (201). The unique identifier information of the electronic device (201) may be information for displaying information related to the electronic device (201) in the external electronic device (202) (e.g., see UI of reference number 701) if the external electronic device (202) and the electronic device (201) have a history of previous connection. In operation 840, the electronic device (201) may receive a connection request response signal from the external electronic device (202). Based on the connection request response signal, the electronic device (201) may establish a connection based on the first protocol and transmit multimedia data to the external electronic device (202).

[0090] As needed, at least one of the actions described above may be omitted or added, and the order of the actions may also be changed.

[0091] FIG. 9 is a flowchart of a method for connecting an external electronic device and an electronic device according to one embodiment.

[0092] In one embodiment, the operations illustrated in FIG. 9 may be performed by a processor of an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2 and 3) (e.g., the processor (120) of FIG. 1 or the processor (320) of FIG. 3)) and a processor of an external electronic device (e.g., the external electronic device (202) of FIG. 3) (e.g., the processor (321) of FIG. 3).

[0093] In the example of FIG. 9, the external electronic device (202) and the electronic device (201) may be assumed to have been connected at least once according to the connection method of FIG. 8. For example, the external electronic device (202) and the electronic device (201) may be assumed to have performed at least one provisioning (e.g., 801 of FIG. 8). The electronic device (201) may store information of the external electronic device (202) (e.g., MAC address, icon information, and / or name).

[0094] In operation 905, the external electronic device (202) can receive a designated input. For example, the designated input may be an input that enables the remote access function of the external electronic device (202). The external electronic device (202) can receive an input for the menu (613) of FIG. 6 as the designated input.

[0095] In operation 910, the external electronic device (202) may transmit (e.g., broadcast) a first advertisement based on a second protocol (e.g., Bluetooth or low-power Bluetooth) when a specified input is received. The first advertisement may include information indicating that multimedia data sharing (e.g., mirroring and / or providing a virtual environment) by the external electronic device (202) is possible.

[0096] In operation 915, the electronic device (201) may transmit a second advertisement based on a second protocol in response to the reception of a first advertisement. For example, the second advertisement may store unique identifier information of the electronic device (201). In one example, the electronic device (201) may be configured to obtain address information of an external electronic device (202) from the received first advertisement and transmit the second advertisement only when the obtained address information of the external electronic device (202) corresponds to (matches) information stored in the electronic device (201). If the received first advertisement contains information of an external electronic device (e.g., MAC address) that is not stored in the electronic device (201), the electronic device (201) may not transmit the second advertisement. For example, the electronic device (201) may receive an advertisement from an external electronic device unrelated to the user (e.g., a television in a neighbor's house). In this case, to prevent unintended connections, the electronic device (201) may ignore advertisements from external electronic devices that do not contain information. For example, an attacker may attempt to hack the electronic device (201) by transmitting advertisements. The electronic device (201) may defend against external attacks by ignoring advertisements from external electronic devices that do not contain information.

[0097] In operation 920, the external electronic device (202) can transmit a scan request to the electronic device (201) based on the second protocol. In operation 925, the electronic device (201) can transmit a scan response to the external electronic device (202).

[0098] In operation 930, the external electronic device (202) can display information of the received electronic device (201). For example, the external electronic device (202) can display information of the electronic device (201) in the list of external source devices described in relation to FIG. 7 based on the unique identifier information of the electronic device (201) described in relation to operation 835 of FIG. 8.

[0099] In operation 935, the external electronic device (202) may receive a selection input from the electronic device (201). In response to the reception of the selection input, in operation 940, the external electronic device (202) may transmit a connection request to the electronic device (201). For example, the connection request may be transmitted based on a second protocol and may include information for a connection based on a first protocol (e.g., the first protocol associated address of the external electronic device (202)). In operation 945, the electronic device (201) may transmit a connection request response signal to the external electronic device (202). After receiving the connection request response signal, the electronic device (201) and the external electronic device (202) may form a connection based on the first protocol and share multimedia data through the formed connection.

[0100] In the example of FIG. 9, if the function associated with the second protocol of the electronic device (201) is disabled, the electronic device (201) may not receive the first advertisement. For example, the electronic device (201) may exceptionally keep the function of the second protocol enabled even in a mode that disables the function of the second protocol (e.g., airplane mode, kids mode, or safety mode, etc.).

[0101] As needed, at least one of the actions described above may be omitted or added, and the order of the actions may also be changed.

[0102] FIG. 10 illustrates a source device environment according to one example.

[0103] Although various examples have been described with reference to the electronic device (201) and the external electronic device (202) in relation to FIGS. 2 to 9, the embodiments of this document can also be applied in an environment where multiple source devices (e.g., the electronic device (201) of FIG. 2 or FIG. 3) exist.

[0104] Referring to FIG. 10, there may be a plurality of source devices (UE1, UE2, UE3, UE4, and UE5). When receiving an advertisement (e.g., the first advertisement (910) of FIG. 9) from an external electronic device (202), each of the plurality of source devices (UE1, UE2, UE3, UE4, and UE5) may determine whether it has ever been connected to the external electronic device (202). For example, if a source device (e.g., the electronic device (201) of FIG. 2 or FIG. 3) has ever been connected to the external electronic device (202), the source device may store the address of the external electronic device (202) (e.g., operation 815 of FIG. 8). The source device may be configured to transmit a response signal (e.g., the second advertisement (915) of FIG. 9) only when the address of the external electronic device (202) included in the advertisement signal corresponds to the stored address.

[0105] In the example of FIG. 10, UE1 and UE2 are source devices that have been connected to the external electronic device (202), while UE3, UE4, and UE5 may not have been connected to the external electronic device (202). UE1 and UE2 may transmit an advertisement signal in response to receiving an advertisement from the external electronic device (202). UE3, UE4, and UE5 may ignore the advertisement signal from the external electronic device (202) and not transmit an advertisement signal. By not transmitting an advertisement signal, interference between the multiple source devices (UE1, UE2, UE3, UE4, and UE5) may be reduced.

[0106] The external electronic device (202) can display a selection menu (1010) for UE1 and a selection menu (1020) for UE2 to which an advertisement signal has been received.

[0107] With reference to FIG. 10, the external electronic device (202) is described as displaying a selection menu (1010 and 1020) based on the reception of an advertisement signal, but the embodiments of this document are not limited thereto. According to one embodiment, the external electronic device (202) may display a selection menu for an electronic device that has a connection history. For example, even if no advertisement signal is received from UE2, the external electronic device (202) may display a selection menu (1020) for a UE2 that has a connection history. When an input selecting a UE2 for which no advertisement signal has been received is received, the external electronic device (202) may attempt to connect to the UE2. While attempting to connect, the external electronic device (202) may display information about the progress of the connection. The external electronic device (202) may attempt to connect to the UE2 until a connection is established, or attempt to connect to the UE2 for a specified period of time or a specified number of times. In one example, the external electronic device (202) can temporarily disable the selection menu (1020) for the UE2 if the connection to the UE2 fails.

[0108] FIG. 11 illustrates a plurality of electronic device environments according to one example.

[0109] Referring to FIG. 11, there may be a plurality of source devices (UE1 and UE2) and a plurality of client devices (1101 and 1102). A source device may refer to a device that provides a service for sharing multimedia data, i.e., an electronic device (201) of FIG. 2 or FIG. 3. A client device may refer to an external electronic device (202) of FIG. 3 that receives a service for sharing multimedia data. For example, a first client device (1101) and a second client device (1102) may transmit an advertisement signal (e.g., a first advertisement (910) of FIG. 9). UE1 is a device that has been connected to both the first client device (1101) and the second client device (1102), and can transmit an advertisement signal (e.g., the second advertisement (915) of FIG. 9) in response to an advertisement signal from the first client device (1101) and the second client device (1102). UE2 is a device that has been connected only to the first client device (1101), and can respond only to an advertisement signal from the first client device (1101).

[0110] The first client device (1101) can display the selection menu (1111) of UE1 based on the advertisement signal of UE1 and the selection menu (1112) of UE2 based on the advertisement signal of UE2. The second client device (1102) can display the selection menu (1111) of UE1 based on the advertisement signal of UE1.

[0111] For example, UE1 may receive connection requests from a first client device (1101) and a second client device (1102). In this case, UE1 may be connected to the client device that first sent the connection request. While UE1 is connected to one client device, it may not be connected to another client device (e.g., the second client device (1102)). After the connection with the existing client device (e.g., the first client device (1101)) is terminated, UE1 may be connected to another client device (e.g., the second client device (1102)).

[0112] FIG. 12 illustrates a flowchart (1200) of a synchronization control method for an electronic device according to one embodiment.

[0113] In one embodiment, the operations illustrated in FIG. 12 may be performed by a processor of an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2 and 3) (e.g., the processor (120) of FIG. 1 or the processor (320) of FIG. 3)) and a processor of an external electronic device (e.g., the external electronic device (202) of FIG. 3) (e.g., the processor (321) of FIG. 3).

[0114] In operation 1205, the electronic device (201) can be connected to an external electronic device (202). For example, the electronic device (201) can be connected based on a first protocol. The electronic device (201) can recognize and connect to the external electronic device (202) according to the connection method described above in relation to FIG. 8 or FIG. 9.

[0115] In operation 1210, the electronic device (201) can transmit video and audio data (e.g., multimedia data) to an external electronic device (202). As described above in relation to FIGS. 2 and 3, the multimedia data may include video data and audio data corresponding to a virtual execution environment or mirroring.

[0116] In operation 1215, the electronic device (201) can control the synchronization of the external electronic device (202) based on whether there is input. For example, when user input associated with multimedia data is received, the electronic device (201) can transmit a signal to the external electronic device (202) instructing the interruption of synchronization of video data and audio data of the external electronic device (202). Operation 1215 may be referenced in relation to the synchronization control method of FIG. 13.

[0117] FIG. 13 illustrates a flowchart of a synchronization control method (1300) of an electronic device according to one embodiment.

[0118] In one embodiment, the operations illustrated in FIG. 13 may be performed by a processor of an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (201) of FIG. 2 and 3) (e.g., the processor (120) of FIG. 1 or the processor (320) of FIG. 3)) and a processor of an external electronic device (e.g., the external electronic device (202) of FIG. 3) (e.g., the processor (321) of FIG. 3).

[0119] For example, the electronic device (201) can perform the synchronization control method of FIG. 13 during multimedia data sharing with an external electronic device (202).

[0120] In operation 1305, the electronic device (201) can determine whether input is detected. The electronic device (201) can detect input by receiving input from the electronic device (201), an external electronic device (202), or an external input device. For example, the electronic device (201) can detect input by receiving input from a mouse, keyboard, touchpad, touch sensor, or stylus pen of the electronic device (201) connected to the electronic device (201) or the external electronic device (202). If no input is detected (operation 1305-N), the electronic device (201) can continue to monitor whether input is received. The electronic device (201) can stop monitoring input when the multimedia sharing service ends.

[0121] When an input is detected (operation 1305-Y), in operation 1310, the electronic device (201) may transmit a signal to the external electronic device (202) instructing it to stop synchronization. When the external electronic device (202) receives the signal instructing it to stop synchronization, it may stop the synchronization of video data and audio data.

[0122] In operation 1315, the electronic device (201) can start a timer. In operation 1320, the electronic device (201) can determine whether a subsequent input is detected. If a subsequent input is detected (operation 1320-Y), the electronic device (201) can start the timer again (e.g., update). That is, the timer can be continuously updated whenever an input is received before the timer expires. If no subsequent input is detected (operation 1320-N), in operation 1325, the electronic device (201) can determine whether the timer has expired. If the timer has not expired (operation 1325-N), the electronic device (201) can continue to monitor for subsequent inputs. If the timer has expired (operation 1325-Y), in operation 1330, the electronic device (201) can transmit a signal to an external electronic device (202) instructing it to start synchronization. The external electronic device (202) can resume synchronization when a signal indicating the start of synchronization is received.

[0123] In operation 1335, the electronic device (201) can determine whether the service for multimedia data sharing has ended. If the service has ended (operation 1335-Y), the electronic device (201) can stop input-based synchronization control. If the service has not ended (operation 1335-N), the electronic device (201) can continue to monitor the input.

Claims

Claim 1 An electronic device comprising: a wireless communication circuit; a processor; and a memory operatively connected to the processor, wherein the memory stores instructions such that, at execution, the processor: connects to an external electronic device using the wireless communication circuit; transmits audio data and video data for multimedia playback on the external electronic device using the wireless communication circuit; determines whether an input for the multimedia is detected; and, if an input for the multimedia is detected, transmits information to the external electronic device instructing it to stop synchronization between the audio data and the video data, and wherein, at execution, the instructions cause the processor to transmit the audio data and the video data based on RTP (real-time transport protocol). Claim 2 In claim 1, the electronic device, wherein the multimedia corresponds to mirroring of the display screen of the electronic device or a virtual execution environment provided by the electronic device. Claim 3 An electronic device according to claim 2, wherein the input to the multimedia includes an input to the execution environment of the mirrored electronic device or an input to the virtual execution environment. Claim 4 An electronic device according to any one of claims 1 to 3, wherein the instructions cause the processor to detect an input to the multimedia based on an input to the electronic device or an input to the external electronic device at the time of execution. Claim 5 An electronic device according to any one of claims 1 to 3, wherein, when the instructions are executed, the processor starts a timer when an input for the multimedia is detected. Claim 6 An electronic device according to claim 5, wherein the instructions cause the processor, when executed, to restart the timer when a subsequent input for the multimedia is detected while the timer is running, and when the timer expires, to transmit a signal instructing the external electronic device to resume synchronization between the audio data and the video data. Claim 7 An electronic device according to any one of claims 1 to 3, wherein, at the time of execution of the instructions, the processor receives a first advertisement signal based on a second protocol from the external electronic device, and if the address of the external electronic device indicated by the first advertisement signal corresponds to an address stored in the memory, the processor transmits a second advertisement signal corresponding to the first advertisement signal. Claim 8 An electronic device according to claim 7, wherein, when the instructions are executed, the processor connects to the external electronic device using the first protocol via the wireless communication circuit when it receives a connection request from the external electronic device using the second protocol. Claim 9 delete Claim 10 An electronic device according to claim 1, wherein each of the audio data and the video data includes time information in a header. Claim 11 A method for controlling the synchronization of an electronic device, comprising: connecting to an external electronic device; transmitting audio data and video data for multimedia playback on the external electronic device to the external electronic device based on RTP (real-time transport protocol); and controlling the synchronization of the external electronic device during the transmission of the audio data and the video data, wherein the operation of controlling the synchronization comprises: determining whether an input for the multimedia is detected; and, when an input for the multimedia is detected, transmitting information to the external electronic device instructing it to stop the synchronization between the audio data and the video data. Claim 12 In claim 11, the multimedia corresponds to mirroring of the display screen of the electronic device or a virtual execution environment provided by the electronic device. Claim 13 In claim 12, the input to the multimedia comprises an input to the execution environment of the mirrored electronic device or an input to the virtual execution environment. Claim 14 A method according to any one of claims 11 to 13, wherein the operation of determining whether an input to the multimedia is detected includes the operation of detecting an input to the multimedia based on an input to the electronic device or an input to the external electronic device. Claim 15 A method according to any one of claims 11 to 13, further comprising the operation of starting a timer when an input for the multimedia is detected. Claim 16 A method according to claim 15, further comprising: an operation to restart the timer when a subsequent input for the multimedia is detected while the timer is running; and an operation to transmit a signal to the external electronic device to resume synchronization between the audio data and the video data when the timer expires. Claim 17 A method according to any one of claims 11 to 13, wherein the operation of connecting to the external electronic device comprises: the operation of receiving a first advertisement signal based on a second protocol from the external electronic device; and the operation of transmitting a second advertisement signal corresponding to the first advertisement signal if the address of the external electronic device indicated by the first advertisement signal corresponds to an address stored in the memory of the electronic device. Claim 18 In claim 17, the operation of connecting to the external electronic device further includes, when a connection request is received from the external electronic device using the second protocol, connecting to the external electronic device using the first protocol by using the wireless communication circuit of the electronic device. Claim 19 delete Claim 20 A method according to claim 11, wherein each of the audio data and the video data includes time information in a header.

Images