Electronic device for providing lip sync self-diagnosis, and operation method thereof

Abstract

The present disclosure provides an electronic device for performing lip sync self-diagnosis. The electronic device can: in response to a lip sync self-diagnosis request by a user input, identify a current-audio output device from among one or more external speakers connected through a speaker and a communication circuit; activate a microphone of the current-audio output device; collect respective pieces of sensor data obtained by detecting an illuminance change of a display and a sound of the current-audio output device while first test data and second test data are alternately output a predetermined number of times by means of the display and the current-audio output device; calculate a delay value between video and audio on the basis of the sensor data obtained by detecting the illuminance change and the sensor data obtained by detecting the sound; and reflect the delay value in an output table of the current-audio output device.

Description

Electronic device providing lip-sync self-diagnosis and method of operation thereof

[0001] The present disclosure relates to an electronic device that provides lip-sync self-diagnosis and a method of operating the same.

[0002] With recent technological advancements, TVs can provide a wide variety of content. TVs can receive content from external servers via wired, satellite, or wireless communication and stream it back. Depending on the communication environment or the processing speed of high-capacity content, lip-sync discrepancies may occur.

[0003] A TV can play audio through various audio output devices depending on user requirements. For example, a TV can be connected to a soundbar, home theater system, Bluetooth speaker, or headset. The TV and audio output devices can be connected via wired or wireless connections, and signal delays or differences in data processing speeds may occur depending on the communication environment. Similarly, lip-sync discrepancies may occur in video and audio playback depending on which audio output device the TV is connected to.

[0004] Some TVs and sound equipment provide lip sync delay settings, allowing you to manually synchronize audio and video. However, since the user manually adjusts the degree of audio delay, achieving accurate lip sync alignment may be difficult.

[0005] An electronic device according to one embodiment of the present disclosure comprises: a color sensor; a microphone; a speaker; a display; a communication circuit; and a memory for storing at least one program. The device includes at least one processor that is electrically connected to the memory and executes at least one instruction of a program stored in the memory, wherein the at least one processor identifies a current audio output device among one or more external speakers connected through the speaker and the communication circuit in response to a lip-sync self-diagnosis request by user input, activates the microphone of the current audio output device, and collects sensor data that detects a change in illumination of the display and a sound of the current audio output device while alternately outputting first test data and second test data a predetermined number of times through the display and the current audio output device, calculates a delay value between video and audio based on the sensor data that detects the change in illumination and the sensor data that detects the sound, and is configured to reflect the delay value in the output table of the current audio output device, wherein the first test data includes a first test sound and a first screen, the second test data includes only a second screen, and the difference in brightness between the first screen and the second screen may be greater than a predetermined threshold.

[0006] According to one embodiment, the at least one processor can determine the difference between a first time point at which the first test data is output from sensor data that detects the change in illumination and a second time point corresponding to the first test sound from sensor data that detects the output sound as a delay value of video or audio.

[0007] According to one embodiment, the at least one processor may calculate the average of video or audio delay values ​​accumulated a predetermined number of times and reflect the average of the delay values ​​in the output table of the current audio output device.

[0008] According to one embodiment, the color sensor may be positioned to detect the output direction of the display. The at least one processor may detect a change in illumination of the display using the color sensor.

[0009] According to one embodiment, the communication circuit may be connected to the current audio output device based on short-range wireless communication. The at least one processor may receive sensor data acquired by the microphone of the current audio output device through the communication circuit.

[0010] According to one embodiment, the at least one processor can detect the sound of the speaker using the microphone in response to the current audio output device corresponding to the speaker.

[0011] According to one embodiment, the communication circuit may be connected to the current audio output device based on short-range wireless communication. The at least one processor may receive sensor data acquired by the ultrasonic sensor of the current audio output device through the communication circuit when the first test sound is an ultrasonic sound in the inaudible frequency range.

[0012] According to one embodiment, the communication circuit may be connected to an external microphone based on short-range wireless communication. The at least one processor may receive sensor data from the external microphone that detects a sound signal output from the current audio output device through the communication circuit.

[0013] According to one embodiment, the first test sound may be either a beep or an ultrasonic wave in the inaudible frequency range.

[0014] According to one embodiment, the at least one processor may transmit the first test data and the second test data to the external electronic device in response to receiving a mirroring request from the external electronic device through the communication circuit, and output the received first test image or the second test image to the display and the current audio output device in response to receiving the first test data and the second test data alternately through the communication circuit.

[0015] According to one embodiment, the at least one processor may output a screen containing information regarding lip-sync self-diagnosis for the current audio output device.

[0016] A lip-sync self-diagnosis method for an electronic device according to another embodiment of the present disclosure comprises: an operation of identifying a current audio output device among a speaker of the electronic device and one or more external speakers in response to a lip-sync self-diagnosis request by user input; an operation of activating a microphone of the current audio output device; an operation of collecting sensor data that detects a change in illumination of the display and a sound of the current audio output device while alternately outputting first test data and second test data a predetermined number of times through the display of the electronic device and the current audio output device; an operation of calculating a delay value between video and audio based on the sensor data that detects the change in illumination and the sensor data that detects the output sound; and an operation of reflecting the delay value in an output table of the current audio output device; wherein the first test data includes a first test sound and a first screen, the second test data includes only a second screen, and the difference in brightness between the first screen and the second screen may be greater than a predetermined threshold.

[0017] In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0018] FIG. 1 is a drawing showing an example of an electronic device according to one embodiment of the present disclosure.

[0019] FIG. 2 illustrates a block diagram of an electronic device according to one embodiment of the present disclosure.

[0020] FIG. 3 is a flowchart illustrating the lip-sync self-diagnosis operation of an electronic device according to one embodiment of the present invention.

[0021] FIG. 4 is an example of a test image according to one embodiment of the present disclosure.

[0022] FIG. 5 is a comparison graph of sensor data acquired by a color sensor and a microphone while outputting a test image according to one embodiment of the present disclosure.

[0023] FIG. 6 is an example of a mirroring operation between an electronic device and an external electronic device according to one embodiment of the present disclosure.

[0024] FIG. 7 is a flowchart illustrating the operation of performing lip-sync self-diagnosis in mirroring mode according to one embodiment of the present disclosure.

[0025] FIG. 8 is an example of an electronic device according to one embodiment of the present disclosure performing lip-sync self-diagnosis using an external microphone.

[0026] FIG. 9 is an example of a screen for a lip-sync self-diagnosis function of an electronic device according to one embodiment of the present disclosure.

[0027] FIG. 10 is an example of a screen for a lip-sync self-diagnosis function in mirroring mode of an electronic device according to one embodiment of the present disclosure.

[0028] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0029] An embodiment of the present disclosure will be described below with reference to the attached drawings.

[0030] FIG. 1 is a drawing showing an example of an electronic device according to one embodiment of the present disclosure.

[0031] An electronic device (101) according to one embodiment can play video through a display and an audio output device. The audio output device (102) may be, for example, a speaker of the electronic device (101), a soundbar (102a), a Bluetooth speaker (102b), a headset (102c), or a home theater system (102d). The audio output devices (102) illustrated in FIG. 1 are exemplary and may be implemented in various forms.

[0032] An electronic device (101) according to one embodiment may be connected to a plurality of audio output devices via wired or wireless connection, and may play the sound of a video through any one of the plurality of audio output devices by the user's selection. When the electronic device (101) and an external audio output device are connected based on wireless communication, a lip-sync problem may occur in which the timing of the display output of the electronic device (101) and the timing of the voice output of the audio output device (102) do not match depending on the wireless communication environment. The electronic device (101) may provide a user function to match the lip-sync through lip-sync self-diagnosis for the current audio output device. In one embodiment, the lip-sync self-diagnosis function may include an operation to play a test video to calculate the delay between the video output timing and the audio output timing, and to adjust the audio delay. If the user experiences discomfort while watching due to lip-sync mismatch, the user can automatically match the lip-sync by executing the lip-sync self-diagnosis function of the electronic device (101).

[0033] FIG. 2 illustrates a block diagram of an electronic device according to one embodiment of the present disclosure.

[0034] According to one embodiment, an electronic device (101) may receive content from a media device (201) and use an external audio output device (102a, 102b, 102c, 102d) to play video. The electronic device (101) may be, for example, a TV or a display device. The media device (201) may be a server or set-top box of a content provider (e.g., Netflix, Wavve). The set-top box may be, for example, a cable set-top box, an IPTV set-top box, or a smart set-top box. The external audio output device (102a, 102b, 102c, 102d) may be various types of speakers connected to the TV to provide effective sound. For example, the external audio output device (102) may include a soundbar (102a), a Bluetooth speaker (102b), a headset (102c), or a home theater system (102d). The electronic device (101) can be simultaneously connected wirelessly or wired to multiple external audio output devices (102), and one or more audio output devices (102) can be selected for content playback.

[0035] According to one embodiment, the electronic device (101) may include a processor (110), memory (120), communication circuit (130), connection terminal (140), display (150), color sensor (160), speaker (170), or microphone (180).

[0036] The processor (110) can, for example, execute software (e.g., a program) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (110) and perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (110) can store commands or data received from other components (e.g., a color sensor (160) or a communication circuit (130)) in volatile memory, process the commands or data stored in volatile memory, and store the resulting data in non-volatile memory. According to one embodiment, the processor (110) may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor. For example, if the electronic device (101) includes a main processor and an auxiliary processor, the auxiliary processor may be configured to use less power than the main processor or to be specialized for a designated function. The auxiliary processor may be implemented separately from the main processor or as part thereof.

[0037] The memory (120) may store various data used by at least one component of the electronic device (101) (e.g., processor (110) or color sensor (160)). The data may include, for example, input data or output data for software (e.g., program) and related commands. The memory (120) may include volatile memory or non-volatile memory. The memory (120) may include test data for lip-sync self-diagnosis. The memory (120) may include various types of test videos. For example, the memory (120) may include test videos by size, such as 2k 60hz, 4k 60hz, 4k 120hz, and 8k 60hz.

[0038] The communication circuit (130) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., a media device (201) or an audio output device (102)), and the performance of communication through the established communication channel. The communication circuit (130) may include one or more communication processors that operate independently of the processor (110) (e.g., an application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication circuit (130) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (e.g., a LAN (local area network) communication module, or a power line communication module). The corresponding communication module among these communication modules can communicate with a media device (201) or an audio output device (102) through a first network (e.g., a short-range communication network such as Bluetooth, WiFi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module can identify or authenticate an electronic device (101) within a communication network, such as the first network or the second network, using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in a subscriber identification module.

[0039] The wireless communication module can support 5G networks following 4G networks and next-generation communication technologies, such as new radio access technology (NR access technology). NR access technology can support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low-latency (ultra-reliable and low-latency communications (URLLC)). The wireless communication module can support high-frequency bands (e.g., mmWave band) to achieve high data transmission rates, for example. The wireless communication module can support various technologies to secure performance in high-frequency bands, such as beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beamforming, or large-scale antenna. The wireless communication module can support various requirements specified in the electronic device (101), external electronic device (e.g., media device (201)), or network system. According to one embodiment, the wireless communication module can support a Peak data rate for eMBB realization (e.g., 20 Gbps or more), loss coverage for mMTC realization (e.g., 164 dB or less), or U-plane latency for URLLC realization (e.g., downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less).

[0040] The connection terminal (140) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., a media device (201) or an audio output device (102)). According to one embodiment, the connection terminal (140) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0041] The display (150) can visually provide information to an external (e.g., user) of the electronic device (101). The display (150) may include, for example, a display panel, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display (150) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of the force generated by said touch.

[0042] The color sensor (160) can detect the brightness, color, and intensity of light and perceive the corresponding color and illuminance (brightness). The color sensor (160) can be positioned to detect the output direction of the display (150). The color sensor (160) can detect light output through the display (150) and recognize changes in illuminance.

[0043] The speaker (170) is an audio output device that allows sound to be heard along with video. The speaker (170) may include a speaker driver, an amplifier, and a sound processor. The sound processor may store and manage a sound output table of the speaker (170). The sound output table may include a value that adjusts the degree of audio delay for lip-sync matching with video.

[0044] The microphone (180) is an input device (sensor) that detects sound and can provide a function to recognize voice.

[0045] FIG. 3 is a flowchart illustrating a lip-sync self-diagnosis operation of an electronic device according to one embodiment of the present disclosure.

[0046] According to one embodiment, the electronic device (101) can perform lip-sync self-diagnosis using a test video and perform lip-sync matching based on the accurate audio delay difference for the current audio output device.

[0047] In operation 310, the electronic device (101) according to one embodiment may receive a lip-sync self-diagnosis request. The electronic device (101) may receive user input to execute a lip-sync self-diagnosis function.

[0048] In operation 320, an electronic device (101) according to one embodiment may identify an output speaker designated as the current audio output device and request that the microphone of the identified output speaker be activated. The electronic device (101) may determine which device is designated as the current audio output device among a built-in speaker or one or more external audio output devices connected via wired / communication. The electronic device (101) may determine a speaker designated as the default value (e.g., a soundbar) as the current audio output device. The electronic device (101) may determine an audio output device selected by user input as the current output speaker. For example, the electronic device (101) may determine an audio device selected by user input as the output speaker according to an audio device selection function. The electronic device (101) may determine an external audio output device with a wireless communication connection enabled as the current output speaker. For example, in response to the Bluetooth speaker completing a short-range wireless communication connection with the electronic device and maintaining the Bluetooth communication connection between the two devices, the electronic device (101) may determine the Bluetooth speaker as the current output speaker. The electronic device (101) may display information about the identified current audio output device on a lip-sync self-diagnosis screen. For example, the electronic device (101) may output a message on the display screen saying, "Proceeding with lip-sync self-diagnosis for the soundbar."

[0049] According to one embodiment, if the identified output speaker is an external audio output device, the electronic device (101) may request the external audio output device to activate the microphone for lip-sync self-diagnosis. The external audio output device may activate the microphone in response to the request for microphone activation from the electronic device (101) and become ready to detect sound.

[0050] In operation 330, an electronic device (101) according to one embodiment may alternately output first test data and second test data a predetermined number of times. The electronic device (101) may perform lip-sync self-diagnosis using a test video, and the test video may alternately include first test data containing sound and second test data not containing sound at a predetermined time interval. The first test data and second test data of the test video may have a large color difference contrast, for example, a white screen and a black screen. The first test data may consist of frames with high brightness (or color) values ​​and include test sound, whereas the second test data may consist of frames with very low brightness (or color) values ​​and may not include any sound, including test sound. In order to accurately compare the video output time and audio output time at the time when the first test data is played, the second test data may be set so that the color sensor value detected by the first test data output and the microphone reception sound can be clearly contrasted. A fixed time interval between the first test data and the second test data can be pre-set to reflect the maximum delay. The number of times the first test data and the second test data are alternately output can be set considering the total execution time of the lip-sync self-diagnosis. The test video may be included in the memory of the electronic device (101) (e.g., memory (120) of FIG. 2). The memory (120) may include various types of test videos. For example, the memory (120) may include test videos in sizes of 2k 60Hz, 4k 60Hz, 4k 120Hz, and 8k 60Hz. In one embodiment, the test video may be determined according to the hardware specifications of the display (e.g., display (150) of FIG. 2). Or, in one embodiment, the test video may vary depending on the content that the user wishes to view.For example, if the electronic device (101) streams and plays 4k 120hz content through a streaming service (e.g., Netflix), it can perform lip-sync self-diagnosis using a test video of 4k 120hz size.

[0051] In one embodiment, the test video may use the test video of the above embodiment for accurate lip-sync self-diagnosis, and as an embodiment that does not interfere with user viewing conditions, the start screen and start sound corresponding to the execution of content that the user wishes to watch may be used as the test video. For example, the electronic device (101) at the time of execution (T) of the first application s Data regarding sound and screen that occur accurately in ) is stored and managed, and the execution of the first application is induced to perform lip-sync self-diagnosis at the user's request, so that the difference between sound output and screen output can be calculated and determined as a delay between video and audio.

[0052] In one embodiment, the test sound may be a beep for general audio testing, or an ultrasonic sound in an inaudible frequency range (e.g., 23 kHz) for user convenience.

[0053] An electronic device (101) according to one embodiment can check the degree of discrepancy between video and audio while outputting first test data and second test data alternately.

[0054] In operation 341, an electronic device (101) according to one embodiment can detect changes in illumination of a display (150) using a color sensor (e.g., the color sensor (160) of FIG. 2) while a test image is being played. The color sensor (160) may be positioned adjacent to the display (150) of the electronic device (101) so as to detect the screen output from the display (150). The color sensor (160) can sense a high illumination value when a first test data with high brightness is output (e.g., when it is a white screen) and can sense a low illumination value when a second test data with low brightness is output (e.g., when it is a black screen). The sensor data detected by the color sensor (160) can be transmitted to the processor (110) of the electronic device (101) via an I2C data line.

[0055] In operation 342, an electronic device (101) according to one embodiment may receive a sound signal detected by a microphone of an output speaker while a test video is being played through a communication circuit (e.g., communication circuit (130) of FIG. 2). The output speaker may include a microphone and may activate the microphone at the request of the electronic device (101) and detect sound while the test video is being played. The output speaker may transmit the detected sound signal and log information to the electronic device (101). The log information may include a timestamp for the detected sound signal. Operations 341 and 342 may be performed simultaneously and may be processed in parallel.

[0056] In operation 350, the electronic device (101) according to one embodiment can calculate the delay between video and audio based on the time difference between the illuminance change and the sound signal. The electronic device (101) can calculate the delay between video and audio based on the time point (T) when the first test data is output from the sensor data detected by the color sensor (160). lux1 ) and the point in time when the second test data is output (T lux2Check ) and likewise, the point in time (T) when the first test data is output from the sensor data received from the output speaker. sound1 ) and the point in time when the second test data is output (T sound2 ...can be verified. The processor (110) can identify the output time of the first test data and the output time of the second test data from the data change received from the color sensor (160) via the I2C line. Since the first test data and the second test data have a large color difference, the difference between the illuminance value detected when the first test data is output and the illuminance value detected when the second test data is output can have a clear difference. For example, if the first test data is a white screen and the second test data is a black screen, the electronic device (101) can view the case where the illuminance value has the highest value in the log information (e.g., I2C data) of the color sensor (160) that detected the illuminance change as the time when the first test data is output, and the case where the illuminance value has the lowest value as the time when the second test data is output. The electronic device (101) can store and refer to reference data as a difference value confirmed in an initial test regarding the extent of the difference between the illuminance value from the output of the first test data and the illuminance value from the output of the second test data, taking into account the surrounding environment. The electronic device (101) can determine whether to switch between the output of the first test data and the second test data by referring to the reference data.

[0057] The first test data may include a test sound, and the second test data may not include a sound. Various types of sounds may be applied to the test sound. For example, the test sound may be a beep, an ultrasonic sound in an inaudible frequency band, or a specific sound selected by user settings. If the test sound is an ultrasonic sound, the electronic device (101) may request the ultrasonic sensor to be activated instead of the microphone of the output speaker and receive sensor data that senses the ultrasonic sound from the ultrasonic sensor.

[0058] An electronic device (101) according to one embodiment detects a change in illumination by the video output of the first test data at a first time point (T lux1 ) and the second time point (T) when a sound signal is detected by the audio output of the first test data souns1 The difference can be calculated and determined as the difference (delay) between video output and audio output.

[0059] An electronic device (101) according to one embodiment can perform cross-output of test data multiple times to obtain multiple delay values ​​between video and audio and calculate a more accurate delay value. For example, while the first test data and the second test data are cross-outputted a predetermined number of times, the electronic device (101) can determine the average of the calculated delay values ​​between video and audio as the final delay.

[0060] In operation 360, the electronic device (101) according to one embodiment can achieve lip-sync matching by reflecting a delay in the output table of the output speaker, which is the current audio output device. The electronic device (101) can match the lip-sync of the video and audio by adjusting the delay value between the video and audio in the audio output table based on the video output. For example, if the video output is faster than the audio output, the electronic device (101) can adjust the delay value in the audio output table forward (faster), and if the opposite is true, adjust the delay value in the audio output table backward (slower). By adjusting the output table for the current audio output device, the audio output of another external audio output device connected to the electronic device (101) or the speaker of the electronic device (101) can be maintained as is. The electronic device (101) can optimize lip-sync alignment for the current audio output device by performing lip-sync automatic self-diagnosis to adjust lip-sync mismatch that occurs depending on the playback content or the communication environment with the external audio output device.

[0061] FIG. 4 is an example of a test image according to one embodiment of the present disclosure.

[0062] A test video according to one embodiment may alternately output first test data and second test data for lip-sync verification. The first test data and the second test data may each include two types of screens with a large difference in brightness. Only one of the first test data and the second test data may include a test sound. In one embodiment, the test data including a screen with high brightness (bright) may include a test sound, and the test data including a screen with low brightness (dark) may not include a test sound or any sound at all.

[0063] Referring to FIG. 4, the first test data (410) consists of a white screen and includes a test sound. The second test data (420) consists of a black screen and does not include a sound. According to one embodiment, the electronic device (101) can output the first test data and the second test data a predetermined number of times. For example, if the predetermined number of times is 10 times, the electronic device (101) 0 <n<=10 동안, Tn시간에 제1 테스트 데이터(410)를 출력하고, Tn+1시간에 제2 테스트 데이터(420)를 출력할 수 있다.

[0064] In one embodiment, the second test data (420) can be replaced by turning off the power to the display (150) panel.

[0065] FIG. 5 is a comparison graph of sensor data acquired by a color sensor and a microphone while outputting a test image according to one embodiment of the present disclosure.

[0066] According to one embodiment, the electronic device (101) can play a test video for lip-sync self-diagnosis, detect changes in screen illumination using a color sensor, and detect output sound using a microphone of an audio output device. This will be explained together with reference to FIG. 4.

[0067] Referring to FIG. 5, the first sensor data acquired by the color sensor can be represented as an illuminance graph (lux / color graph) showing changes in illuminance over time. The second sensor data acquired by the microphone can be represented as a sound graph showing changes in sound signals over time. By matching the two graphs based on the time (T0) at which the test video is played, a delay value between the video and the audio can be calculated.

[0068] Referring to the top of FIG. 5, the first time point (T1), when the magnitude of the illuminance detected in the illuminance graph is the largest, can be seen as the time point when the first test data (410), which includes a white screen with high brightness, is output. The second time point (T2), when the magnitude of the illuminance is the smallest, can be seen as the time point when the second test data (420), which includes a black screen with the lowest brightness, is output. Since the first test data (410) and the second test data (420) are repeatedly alternately output, the third time point (T3), when the illuminance of the same magnitude as the first time point (T1) is detected, can be seen as the time point when the first test data (410) is output again.

[0069] Referring to the bottom of Fig. 5, the fourth time point (T) at which the sound intensity detected in the sound graph is greatest 11 ) can be seen as the point in time when the first test data (410) containing the test sound is output. The fifth point in time (T) when the sound is quietest. 12 ) can be seen as the point in time when the second test data (420) that does not include sound is output. Since the first test data (410) and the second test data (420) are repeatedly alternately output, the first point in time (T 11 The 6th time point (T) where a sound of the same magnitude as ) was detected 13 ) can be seen as the point in time when the first test data (410) is output again.

[0070] According to one embodiment, the electronic device (101) has a first time point (T1) at which a white screen of the first test data (410) is detected and a fourth time point (T1) at which a test sound of the first test data (420) is detected. 11 By comparing ), the delay value between the video and audio can be calculated. For example, the delay value is the fourth time point (T 11 It can be the difference obtained by subtracting the first time point (T1) from ). The first test data (410) and the second test data (420) are repeatedly alternately output at the sixth time point (T 13The difference obtained by subtracting the third time point (T3) from ) can be the second delay value. Multiple delay values ​​can be calculated as many times as the first test data (410) and the second test data (420) are repeatedly output. The electronic device (101) can determine the final delay as the average of the multiple delay values.

[0071] FIG. 6 is an example of a mirroring operation between an electronic device and an external electronic device according to one embodiment of the present disclosure.

[0072] An electronic device (101) according to one embodiment can receive and play content by means of a mirroring operation of an external electronic device (103). The electronic device (101) can match the lip-sync between video and audio by performing a lip-sync self-diagnosis for video playback by mirroring with the external electronic device (103). Referring to FIG. 6, the electronic device (101) can duplicate and display the playback video from the external electronic device (103) exactly as it is through a mirroring function. The electronic device (101) can output the playback video of the external electronic device (103) in real time via wireless communication (e.g., Wi-Fi network) using the display of the electronic device (101) (e.g., the display (150) of FIG. 2) and the speaker (102) set as an audio output device. A lip-sync mismatch may occur depending on the wireless communication environment or content processing speed between the electronic device (101) and the external electronic device (103). The electronic device (101) can resolve lip-sync mismatch between video and audio through lip-sync self-diagnosis even in the case of mirroring operation.

[0073] FIG. 7 is a flowchart illustrating the operation of performing lip-sync self-diagnosis in mirroring mode according to one embodiment of the present disclosure.

[0074] According to one embodiment, the electronic device (101) can perform lip-sync self-diagnosis in mirroring mode. At least a portion of the lip-sync self-diagnosis operation of FIG. 7 may correspond to the lip-sync self-diagnosis operation in the embodiment of FIG. 3.

[0075] In operation 710, the electronic device (101) according to one embodiment can check the mirroring status in response to receiving a lip-sync self-diagnosis request. The electronic device (101) can determine whether it is currently in mirroring mode. If the electronic device (101) is in mirroring mode, it can identify a data transmission device (e.g., the electronic device (103) of FIG. 6) that provides the mirroring screen. If the electronic device (101) is not in mirroring mode, it can perform a lip-sync self-diagnosis according to the lip-sync self-diagnosis operation of FIG. 3.

[0076] In operation 720, an electronic device (101) according to one embodiment may transmit a first test data and a second test data to a data transmission device (103) using a communication circuit (e.g., the communication circuit (130) of FIG. 2). The first test data and the second test data may each include two types of screens with a large difference in brightness. Only one of the first test data and the second test data may include a test sound. In one embodiment, the test data including a screen with a large brightness (bright) may include a test sound, and the test data including a screen with a small brightness (dark) may not include a test sound or any sound at all.

[0077] An electronic device (101) according to one embodiment may be connected to a data transmission device (103) based on short-range wireless communication. For example, the communication circuit (130) of the electronic device (101) may transmit first test data and second test data to the data transmission device (103) based on a Wi-Fi wireless communication protocol. If the data transmission device (103) has a history of performing lip-sync self-diagnosis in a previous mirroring mode, it may already have the first test data and second test data stored, in which case it may not transmit the test data.

[0078] In operation 730, the electronic device (101) according to one embodiment may identify an output speaker designated as the current audio output device (102) and request that the microphone of the identified output speaker be activated. The operation may include at least a portion of operation 320 of FIG. 3.

[0079] An electronic device (101) according to one embodiment may display information about an identified current audio output device (102) and information about a data transmission device (103) connected in mirroring mode together on a lip-sync self-diagnosis screen. For example, the electronic device (101) may output a message such as "Starting lip-sync self-diagnosis for mirrored video of Galaxy Flip and soundbar output."

[0080] In operation 740, the electronic device (101) according to one embodiment may cross-output the first test data and the second test data using a display (e.g., the display (150) of FIG. 2) and a current audio output device (102) in response to cross-receiving the first test data and the second test data from a data transmission device (103). The operation may include at least a portion of operation 330 of FIG. 3.

[0081] In operation 751, an electronic device (101) according to one embodiment may detect a change in illumination of a display (150) using a color sensor (e.g., the color sensor (160) of FIG. 2) while first test data and second test data are alternately output. The color sensor (160) may be positioned adjacent to the display (150) of the electronic device (101) so as to detect a screen output from the display (150). The color sensor (160) may sense a high illumination value when first test data with high brightness is output (e.g., when it is a white screen) and sense a low illumination value when second test data with low brightness is output (e.g., when it is a black screen). Sensor data detected by the color sensor (160) may be transmitted to the processor (110) of the electronic device (101) via an I2C data line. The operation may include at least a portion of operation 341 of FIG. 3.

[0082] In operation 752, an electronic device (101) according to one embodiment may receive a sound signal detected by a microphone of an output speaker through a communication circuit (130). The electronic device (101) may be connected to an audio output device (102) based on short-range wireless communication. For example, the communication circuit (130) of the electronic device (101) may receive sensor data acquired by a microphone from the audio output device (102) based on a Wi-Fi wireless communication protocol. The operation may include at least a portion of operation 342 of FIG. 3. Operations 751 and 752 may be performed simultaneously and may be processed in parallel.

[0083] In operation 760, an electronic device (101) according to one embodiment can calculate a delay between video and audio based on the time difference between the illuminance change and the sound signal. The electronic device (101) can calculate the difference in the time at which the first test data is output as a delay value (delay) between video and audio by comparing the illuminance change data obtained by the color sensor (160) with the sound detection data received from the current audio output device (102). The electronic device (101) may cause lip-sync mismatch due to the influence of the communication environment connected to the data transmission device (103) and the audio output device (102). The delay value calculated in the above operation may reflect both the delay that occurs when the electronic device (101) receives the test data from the data transmission device (103) and the delay that occurs when the electronic device (101) transmits the audio signal to the current audio output device (102) and outputs it. The above operation may include at least a part of operation 350 of FIG. 3.

[0084] In operation 770, the electronic device (101) according to one embodiment can achieve lip-sync matching by reflecting a delay in the output table of the output speaker, which is the current audio output device. By adjusting the output table for the current audio output device, the electronic device (101) can maintain the audio output of another external audio output device connected to the electronic device (101) or the speaker of the electronic device (101). The operation may include at least a portion of operation 360 of FIG. 3.

[0085] FIG. 8 is an example of an electronic device according to one embodiment of the present disclosure performing lip-sync self-diagnosis using an external microphone.

[0086] An electronic device (801) according to one embodiment (e.g., the electronic device (101) of FIG. 1) may be connected to a plurality of external audio output devices (e.g., the audio output device (102) of FIG. 1). Referring to FIG. 8, the plurality of audio output devices (102) may be, for example, a soundbar (820), a first speaker (830), a second speaker (840), and a third speaker (850) of a home theater system. The electronic device (101) may perform lip-sync self-diagnosis for the third speaker (850) when the current audio output device is the third speaker (850) of the home theater system.

[0087] In one embodiment, if the third speaker (850) of the home theater system does not include a microphone, the electronic device (101) can detect the sound output from the third speaker (850) using a separate electronic device (e.g., remote control (802)) and receive it through a communication circuit. The separate electronic device may be a remote control (802) that is connected to the electronic device (101) via communication, as shown in the example of FIG. 8, and may be another electronic device of the user of the electronic device (101) (e.g., mobile terminal, watch-type wearable electronic device, ring-type wearable electronic device).

[0088] An electronic device (101) according to one embodiment identifies an output speaker currently designated as an audio output device, and if the identified output speaker does not include a microphone, it may determine a separate electronic device to be used as a microphone capable of detecting the sound of the output speaker. For example, if the identified output speaker is a third speaker (850) of a home theater system and the third speaker (850) does not include a microphone, the electronic device (101) may determine to use an electronic device located adjacent to the third speaker (850) as a microphone. The electronic device (101) may identify whether the user of the electronic device (101) is wearing a wearable electronic device, and if the user is wearing a wearable electronic device, it may determine to use the wearable electronic device as a microphone.

[0089] If the electronic device (101) according to one embodiment decides to use a separate electronic device as a microphone, it can activate the microphone and request that the detected sensor data be transmitted while the test data is cross-output. If the electronic device (101) determines a microphone to detect the output sound of the third speaker (850), it can provide information about the determined microphone through a lip-sync self-diagnosis screen. For example, the electronic device (101) can output a message saying, "Please place the remote control close to the speaker for lip-sync self-diagnosis of the third speaker of the home theater system."

[0090] In one embodiment, if the electronic device (101) cannot automatically determine a separate electronic device to be used as a microphone, it may output a UI for selecting a device to be used as a microphone through a lip-sync self-diagnosis screen. For example, the electronic device (101) may list a remote control, a user's mobile device, and a user's wearable electronic device, and provide a message to select one of them. The electronic device (101) may request the separate electronic device acting as a microphone selected by user input to activate the microphone, and may request the transmission of sensor data detected while test data is cross-output.

[0091] In one embodiment, the electronic device (101) can perform lip-sync self-diagnosis for the first speaker (830) when the current audio output device is the first speaker (830) of the home theater system. If the first speaker (830) does not include a microphone, the electronic device (101) can determine a separate electronic device to be used as a microphone for detecting the output sound of the first speaker (830). The electronic device (101) can sense the output sound of the first speaker (830) using the microphone (180) when the first speaker (830) is located within the sound detection range of the electronic device (101)'s microphone (e.g., the microphone (180) in FIG. 2). The sound data detected by the microphone (108) of the electronic device (101) can be transmitted to the processor (110) of the electronic device (101) via an I2C data line.

[0092] FIG. 9 is an example of a screen for a lip-sync self-diagnosis function of an electronic device according to one embodiment of the present disclosure.

[0093] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (101) of FIG. 2) can output a lip-sync self-diagnosis screen through a display (e.g., the display (150) of FIG. 2).

[0094] Referring to FIG. 9, the electronic device (101) may display a "lip-sync self-diagnosis" function that can be selected on a sound settings menu screen (910). In response to the selection of the "lip-sync self-diagnosis" function by user input, the electronic device (101) may display a speaker selection screen (920) for lip-sync self-diagnosis. The electronic device (101) may output "current output device" information for the output speaker designated as the current audio output device, thereby allowing the user to explicitly confirm the target of the lip-sync self-diagnosis. The electronic device (101) may display a screen (930) to start the lip-sync self-diagnosis in response to the input of the next button while the "soundbar" displayed as the current output device is selected. When "Bluetooth Speaker" is selected on the speaker selection screen (920) by user input, the electronic device (101) performs a lip-sync self-diagnosis operation for the "Bluetooth Speaker" and can indicate on the lip-sync self-diagnosis start screen (930) that the lip-sync self-diagnosis is for the "Bluetooth Speaker" output. On the lip-sync self-diagnosis start screen (930), the electronic device (101) can provide information about the target of the lip-sync self-diagnosis, information for performing an accurate lip-sync self-diagnosis (e.g., a dark surrounding environment), and information on the time required for the self-diagnosis. The electronic device (101) can provide the user with a "Please dim the lights" message so that the color sensor can more accurately detect the difference in illumination of the display. For user convenience, the electronic device (101) can display the estimated time required for the lip-sync self-diagnosis. The electronic device (101) may display a completion screen (940) indicating that lip-sync matching is complete when the lip-sync self-diagnosis is completed. The electronic device (101) may clearly display the target of the lip-sync self-diagnosis (e.g., soundbar) on the completion screen and include a notice that lip-sync matching is complete (e.g., output is optimized).The lip-sync self-diagnosis screen of FIG. 9 is exemplary, and the electronic device (101) may require user input necessary for lip-sync self-diagnosis operation in various embodiments or display various information.

[0095] FIG. 10 is an example of a screen for a lip-sync self-diagnosis function in mirroring mode of an electronic device according to one embodiment of the present disclosure.

[0096] An electronic device according to one embodiment (e.g., the electronic device (101) of FIG. 2 or the electronic device (101) of FIG. 6) can perform lip-sync self-diagnosis in mirroring mode. This can be explained together with reference to FIG. 7.

[0097] An electronic device (101) according to one embodiment can identify a data transmission device and a current audio output device when in a mirroring state. The electronic device (101) can provide a user with a lip-sync self-diagnosis start screen (1010) containing information about the identified data transmission device and the current audio output device. For example, referring to FIG. 10, if the data transmission device is "Galaxy Flip" and the audio output device is "Sound Bar," the electronic device (101) can output a message saying "Starting lip-sync self-diagnosis for mirrored video of Galaxy Flip and sound bar output." When the lip-sync self-diagnosis is completed, the electronic device (101) can clearly indicate the target of the lip-sync self-diagnosis on a completion screen (1020), similar to the start screen (1010), and provide guidance that lip-sync matching is completed. For example, the electronic device (101) can output a message saying "Lip-sync self-diagnosis is complete, lip-sync matching for Galaxy Flip and soundbar is complete."

[0098] The embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" each may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0099] The term “module” as used in the embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0100] One embodiment of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0101] According to one embodiment, the method according to the embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0102] According to one embodiment, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to one embodiment, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to one embodiment, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device, Color sensor; mike; speaker; display; Communication circuit; Memory for storing at least one program; and It includes at least one processor electrically connected to the memory and executing at least one instruction of a program stored in the memory, The above-mentioned at least one processor is, In response to a lip-sync self-diagnosis request by user input, the current audio output device is identified among one or more external speakers connected through the speaker and the communication circuit, and Activate the microphone of the current audio output device mentioned above, and While the first test data and the second test data are alternately output a predetermined number of times through the display and the current audio output device, respective sensor data detecting changes in the illumination of the display and the sound of the current audio output device are collected, and Based on the sensor data detecting the above-mentioned change in illumination and the sensor data detecting the above-mentioned sound, a delay value between video and audio is calculated, and The output table of the current audio output device is set to reflect the delay value, and The above first test data includes a first test sound and a first screen, and The above second test data includes only the second screen, and An electronic device that makes the difference in brightness between the first screen and the second screen greater than a predetermined threshold.

2. In Paragraph 1, The above-mentioned at least one processor is, An electronic device that determines the difference between a first time point at which the first test data is output from sensor data that detects the change in illumination and a second time point corresponding to the first test sound from sensor data that detects the output sound as a delay value of video or audio.

3. In Paragraph 1, The above-mentioned at least one processor is, Calculate the average of the video or audio delay values ​​accumulated for the above-determined number of times, and An electronic device that reflects the average of the delay values ​​in the output table of the current audio output device.

4. In Paragraph 1, The color sensor is positioned to detect the output direction of the display, and The above at least one processor is an electronic device that detects a change in illumination of the display using the color sensor.

5. In Paragraph 1, The above communication circuit is connected to the current audio output device based on short-range wireless communication, and The above at least one processor is an electronic device that receives sensor data acquired by the microphone of the current audio output device through the communication circuit.

6. In Paragraph 1, The above-mentioned at least one processor is an electronic device that detects the sound of the speaker using the microphone in response to the current audio output device corresponding to the speaker.

7. In Paragraph 1, The above communication circuit is connected to the current audio output device based on short-range wireless communication, and The above-mentioned at least one processor is an electronic device that receives sensor data acquired by the ultrasonic sensor of the current audio output device through the communication circuit when the first test sound is an ultrasonic sound in the inaudible frequency range.

8. In Paragraph 1, The above communication circuit is connected to an external microphone based on short-range wireless communication, and The above-mentioned at least one processor is an electronic device that receives sensor data from the external microphone that detects a sound signal output from the current audio output device through the communication circuit.

9. In Paragraph 1, The above first test sound is an electronic device in which either a beep or an ultrasonic wave in the inaudible frequency range.

10. In Paragraph 1, The above at least one processor, in response to receiving a mirroring request from an external electronic device through the communication circuit, transmits the first test data and the second test data to the external electronic device, and An electronic device that outputs the received first test image or the second test image to the display and the current audio output device in response to receiving the first test data and the second test data alternately through the communication circuit.

11. In Paragraph 1, An electronic device wherein the above-mentioned at least one processor outputs a screen containing information regarding lip-sync self-diagnosis for the current audio output device.

12. In a method for self-diagnosing lip-sync of an electronic device, An operation to identify the current audio output device among the speaker of the electronic device and one or more external speakers in response to a lip-sync self-diagnosis request by user input; The operation of activating the microphone of the above-mentioned current audio output device; An operation of collecting respective sensor data that detects changes in illumination of the display and sound of the current audio output device while alternately outputting first test data and second test data a predetermined number of times through the display of the electronic device and the current audio output device; An operation to calculate a delay value between video and audio based on sensor data detecting the above-mentioned change in illumination and sensor data detecting the above-mentioned output sound; It includes an operation to reflect the delay value in the output table of the current audio output device; The above first test data includes a first test sound and a first screen, and The above second test data includes only the second screen, and A lip-sync self-diagnosis method in which the difference in brightness between the first screen and the second screen is greater than a predetermined threshold.

13. In Paragraph 12, The operation of calculating the delay value between the above video and audio is, A lip-sync self-diagnosis method that determines the difference between a first time point at which the first test data is output from sensor data detecting the change in illumination and a second time point corresponding to the first test sound from sensor data detecting the output sound as a delay value of video or audio.

14. In Paragraph 12, The operation of calculating the delay value between the above video and audio is, Calculate the average of video or audio delay values ​​accumulated for the above-determined number of times, and A lip-sync self-diagnosis method that reflects the average of the delay values ​​in the output table of the current audio output device.

15. In Paragraph 12, The operation of collecting respective sensor data that detects changes in illumination of the above-mentioned display and sound of the above-mentioned current audio output device is, A lip-sync self-diagnosis method that detects a change in illumination of the display by using a color sensor arranged to detect the output direction of the display of the electronic device.

Images