Electronic device for outputting audio and operation method therefor

Abstract

Disclosed is an electronic device comprising a memory storing instructions, and a processor. The instructions may cause the electronic device to: acquire, from first media data, a first audio data block corresponding to the end point of the first media data; identify a final playback position of the first audio data block on the basis of the end point; acquire a second audio data block for use as a search target from second media data on the basis of the start point, corresponding to the end point, of the second media data; search a second audio data block for audio data corresponding to the final playback position; determine the playback start position of the second audio data block on the basis of the audio data searched for; and output audio data following the playback start position among the second media data after the audio playback up to the final playback position is completed.

Description

Electronic device for outputting audio and method of operation thereof

[0001] Embodiments of the present disclosure relate to an electronic device for outputting audio data and a method of operating the same.

[0002] As electronic communication technologies advance, various functions are being integrated into communication devices or electronic devices, and electronic devices are also being implemented to perform interoperability functions that allow them to communicate with other electronic devices. For example, portable electronic devices (e.g., mobile terminals, tablets, or wearable electronic devices) include both communication and audio playback capabilities. In addition to audio sources stored during manufacturing, they can also play various audio sources associated with applications through the installation of additional applications, and output audio sounds.

[0003] An electronic device and an operating method thereof according to embodiments of the present disclosure can output audio data obtained through demuxing and decoding.

[0004] An electronic device and its operating method according to embodiments of the present disclosure can seamlessly play audio of separated media files.

[0005] An electronic device and its operating method according to embodiments of the present disclosure can determine a playback start position of another media file by searching for the same audio data of another media data based on audio data corresponding to a last playback position of one media data.

[0006] An electronic device according to one embodiment of the present disclosure may include a memory storing instructions, a speaker, and at least one processor operatively connected to the memory and the speaker. The instructions, when executed by the at least one processor, may cause the electronic device to obtain a first audio data block corresponding to an end time of the first media data from first media data. The instructions, when executed by the at least one processor, may cause the electronic device to identify a last playback position of the first audio data block based on the end time. The instructions, when executed by the at least one processor, may cause the electronic device to obtain a second audio data block to be used as a search target from the second media data based on a start time of second media data corresponding to the end time. The instructions, when executed by the at least one processor, may cause the electronic device to search for audio data corresponding to the last playback position in the second audio data block. The above instructions, when executed by the at least one processor, may cause the electronic device to determine a playback start position of the second audio data block based on the searched audio data. The above instructions, when executed by the at least one processor, may cause the electronic device to output audio data from the playback start position of the second media data to the speaker after audio playback of the first media data up to the last playback position of the first audio data block is completed.

[0007] An operating method of an electronic device according to one embodiment of the present disclosure may include an operation of obtaining a first audio data block corresponding to an end time of the first media data from first media data. The method may include an operation of identifying a last playback position of the first audio data block based on the end time. The method may include an operation of obtaining a second audio data block to be used as a search target from the second media data based on a start time of second media data corresponding to the end time. The method may include an operation of searching for audio data corresponding to the last playback position in the second audio data block. The method may include an operation of determining a playback start position of the second audio data block based on the searched audio data. The method may include an operation of outputting audio data from the playback start position of the second media data to a speaker after audio playback of the first media data up to the last playback position of the first audio data block is completed.

[0008] According to one embodiment of the present disclosure, a non-transitory computer-readable storage medium storing one or more programs may include instructions that, when executed by at least one processor of an electronic device, cause the electronic device to: obtain a first audio data block corresponding to an end time point of the first media data from first media data, identify a last playback position of the first audio data block based on the end time point, obtain a second audio data block to be used as a search target from the second media data based on a start time point of second media data corresponding to the end time point, search for audio data corresponding to the last playback position in the second audio data block, determine a playback start position of the second audio data block based on the searched audio data, and output audio data from the playback start position among the second media data to a speaker after audio playback of the first media data up to the last playback position of the first audio data block is completed.

[0009] The above and other aspects, features and advantages of specific embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0010] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0011] FIG. 2 is a block diagram showing the configuration of an electronic device that outputs audio according to one embodiment of the present disclosure.

[0012] FIG. 3 is a diagram for explaining split playback of a media file according to one embodiment of the present disclosure.

[0013] FIG. 4 is a diagram for explaining a mismatch of audio data due to the absence of a reference frame according to one embodiment of the present disclosure.

[0014] FIG. 5 is a diagram illustrating audio interruption due to inaccuracy of a playback start position according to one embodiment of the present disclosure.

[0015] FIG. 6 is a diagram illustrating playback of media files including overlapping shooting sections according to one embodiment of the present disclosure.

[0016] FIG. 7 is a block diagram illustrating a configuration for continuous audio playback according to one embodiment of the present disclosure.

[0017] FIG. 8 is a flowchart illustrating a method for continuous playback of audio according to one embodiment of the present disclosure.

[0018] FIG. 9 is a flowchart illustrating a procedure for determining a final playback position according to one embodiment of the present disclosure.

[0019] FIG. 10 is a flowchart illustrating a procedure for determining a last playback position for an audio data block according to one embodiment of the present disclosure.

[0020] FIG. 11 is a diagram illustrating an example of determining a final playback position for media data with an end point set according to one embodiment of the present disclosure.

[0021] FIG. 12 is a diagram illustrating an example of determining a last playback position for media data for which an end point is not set according to one embodiment of the present disclosure.

[0022] FIG. 13 is a flowchart illustrating a procedure for determining audio data to be searched according to one embodiment of the present disclosure.

[0023] FIG. 14 is a drawing for explaining setting of a demuxing start position according to one embodiment of the present disclosure.

[0024] FIG. 15 is a drawing for explaining a demuxing operation from a demuxing start position according to one embodiment of the present disclosure.

[0025] FIG. 16 is a diagram for explaining determination of a search target audio data block according to one embodiment of the present disclosure.

[0026] FIG. 17 is a flowchart illustrating a procedure for searching audio data according to one embodiment of the present disclosure.

[0027] FIG. 18 is a diagram for explaining determination of a search target channel according to one embodiment of the present disclosure.

[0028] FIG. 19 is a diagram illustrating determination of search reference data based on the last playback position according to one embodiment of the present disclosure.

[0029] FIG. 20 is a diagram illustrating determination of search reference data based on the last position according to one embodiment of the present disclosure.

[0030] FIG. 21 is a diagram for explaining audio data search according to one embodiment of the present disclosure.

[0031] FIG. 22 is a diagram for explaining determination of a playback start position based on a search result according to one embodiment of the present disclosure.

[0032] FIG. 23 is a diagram for explaining determination of a playback start position based on a search result according to one embodiment of the present disclosure.

[0033] FIG. 24 is a diagram for explaining an operation according to a failure to search audio data according to one embodiment of the present disclosure.

[0034] FIG. 25 is a diagram for explaining an operation of outputting video frames according to a changed timestamp of audio data according to one embodiment of the present disclosure.

[0035] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to one embodiment.

[0036] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0037] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0038] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, in the electronic device (101) itself where artificial intelligence is performed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0039] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0040] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0041] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0042] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0043] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0044] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0045] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0046] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0047] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0048] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0049] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0050] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0051] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0052] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0053] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0054] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0055] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0056] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0057] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or application-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0058] FIG. 2 is a block diagram showing the configuration of an electronic device that outputs audio according to one embodiment of the present disclosure.

[0059] Referring to FIG. 2, an electronic device (e.g., electronic device (101)) may include at least one processor (200) (e.g., processor (120) of FIG. 1) and a speaker (250) (e.g., audio output module (155) of FIG. 1). The processor (200) may include at least one of a controller (210), a demuxer (220), or a decoder (230). At least one of the controller (210), the demuxer (220), or the decoder (230) may be implemented as software executed by the processor (200), or may be implemented as a hardware module. In one embodiment, at least one of a memory (215) or an audio buffer (240) may be included in the processor (200), or may be implemented as a separate memory (e.g., memory (130) of FIG. 1).

[0060] The demuxer (220) can receive media data (e.g., at least one media file) including audio frames and video frames, and demux the media data to separate it into audio frames and video frames. The audio frames can be encoded (e.g., compressed) audio frames. The audio frames can be transmitted to the decoder (230). The decoder (230) can decode (e.g., decompress) the audio frames according to a specified codec (e.g., MPEG (moving picture experts group)) to generate audio data including PCM (pulse code modulation) data. The audio buffer (240) can store the audio data until it is output to the speaker (250). The audio buffer (240) can store the audio data generated by the decoder (230) in units of audio data blocks of a specified size.

[0061] The controller (210) can control the operation of the demuxer (220), the decoder (230) and / or the audio buffer (240). According to the disclosed embodiments, the controller (210) can determine frames input to the demuxer (220), manage a timestamp indicating a playback start time of PCM data output from the decoder (230), back up (e.g., copy and store) at least a portion of the PCM data stored in the audio buffer (240) to the memory (215), and determine PCM data to be output from the audio buffer (240) based on a controlled playback start position (e.g., data position or time position) according to the disclosed embodiments.

[0062] The controller (210) may perform audio rendering (not shown) such as volume adjustment, resampling, etc. of sound output to the speaker (250) before outputting the PCM data stored in the audio buffer (240) to the speaker (250). In the embodiments of the present disclosure, outputting audio data (e.g., PCM) to the speaker (250) may include an operation of performing audio rendering such as volume adjustment or resampling on the audio data.

[0063] The memory (215) can store at least one of metadata related to at least one media data to be played back under the control of the controller (210) (e.g., at least one of a sampling rate, a channel count, or a sample byte size), a start time or an end time of the media data, audio data read from an audio buffer (240) (e.g., at least one audio data block), or a timestamp of the audio data.

[0064] An electronic device (101) (e.g., a processor (200) of FIG. 2) may be configured to continuously play one or more media data (e.g., first media data and second media data). The electronic device (101) (e.g., a processor (200) of FIG. 2) may read the one or more media data from a memory (e.g., a memory (130) of FIG. 1), or may receive the one or more media data from an external electronic device (e.g., an electronic device (102) of FIG. 1, an electronic device (104) of FIG. 1, or a server (108) of FIG. 1)) via a communication module (190). In one embodiment, the one or more media data may be received via an audio streaming service.

[0065] An electronic device (101) (e.g., a processor (200) of FIG. 2) may obtain audio data (e.g., PCM data) for at least one audio frame of first media data through a demuxer (220) and a decoder (230), store the audio data in an audio buffer (240), and output audio data before a specified end point of the first media data and audio data up to the end point to a speaker (250). The electronic device (101) (e.g., a processor (200) of FIG. 2) may request the demuxer (220) to output at least one audio frame corresponding to a specified start point of the second media data in order to continuously play the second media data. An electronic device (101) (e.g., a processor (200)) can obtain audio data (e.g., PCM data) by decoding at least one audio frame output from a demuxer (220) through a decoder (230), and output audio data corresponding to the start point of second media data and subsequent audio data to a speaker (250).

[0066] FIG. 3 is a diagram for explaining split playback of a media file according to one embodiment of the present disclosure.

[0067] Referring to FIG. 3, the electronic device (101) (e.g., the processor (200) of FIG. 2) may, according to one embodiment, divide a media file (310) (e.g., media.mp4) having a length of 10,000 ms into first media data (312) and second media data (314), and may segment and process the first media data (312) and the second media data (314) through different demuxing and decoding processes (e.g., the demuxer (220) and the decoder (230)).

[0068] According to one embodiment, the electronic device (101) (e.g., the processor (200)) may divide the media file (310) into first media data (312) corresponding to a section of 0 to 5,000 ms and second media data (314) corresponding to a section of 5,000 to 10,000 ms based on a specified point in time (e.g., 5,000 ms) and play the divided data. Accordingly, the specified point in time (e.g., 5,000 ms) may be set as end_time indicating an end point of the first media data (312) and start_time indicating a start point of the second media data (314), respectively. According to one embodiment, the electronic device (101) (e.g., the processor (200)) may control the demuxer (220) and the decoder (230) so that the second media data (314) may be played continuously following the first media data (312).

[0069] According to one embodiment, since the first media data (312) and the second media data (314) are configured in frame units by the demuxer (220), the audio frames at the back of the first media data (312) may at least partially overlap with the audio frames at the front of the second media data (314). However, depending on the decoding method of the audio frames, the audio data at the end of the first media data (312) and the audio data at the start of the second media data (314) may not be continuous, which may cause the audio output to the speaker (250) to be interrupted.

[0070] For example, the decoder (230) may improve decoding performance by referencing one or more previous audio frames for decoding each audio frame, but at least one preceding audio frame of the second media data (314) may be decoded without the reference audio frames (e.g., previous audio frames), which may result in a loss of continuity with respect to the first media data (312) and thus a discontinuity in audio output to the speaker (250).

[0071] FIG. 4 is a diagram for explaining a mismatch of audio data due to the absence of a reference frame according to one embodiment of the present disclosure.

[0072] Referring to FIG. 4, an audio buffer (e.g., audio buffer (240) of FIG. 2) can store audio data (e.g., PCM data) generated by decoding a specified number (e.g., one or more) of audio frames of the first media data (312) in units of audio data blocks. The last audio data block (402) of the first media data (e.g., second media data (312) of FIG. 3) stored in the audio buffer (240) after demuxing and decoding can have, for example, TS=4,991 ms and a size of 4,096 bytes. The last audio data block (402) can include audio data generated as a result of decoding at least one audio frame of, for example, TS=4,991 ms, and PCM data corresponding to each audio frame can be generated through decoding with reference to previous audio frames (not shown) in the first media data (312). An electronic device (101) (e.g., a processor (200) of FIG. 2) can output audio data (404) up to the last playback position (406) corresponding to the end_time (e.g., 5,000 ms) of the first media data (e.g., the first media data (312) of FIG. 3) among the last audio data blocks (402) to a speaker (e.g., a speaker (250) of FIG. 2).

[0073] According to one embodiment, the electronic device (101) (e.g., the processor (200)) may operate a demuxer (e.g., the demuxer (220) of FIG. 2) and a decoder (e.g., the decoder (230) of FIG. 2) on the second media data (314) to sequentially play the second media data (e.g., the second media data (314) of FIG. 3) following the first media data (312). The demuxer (220) may demux from the audio frame of TS=4,991 ms closest to the start_time (e.g., 5,000 ms) of the second media data (314) and output the audio frames after demuxing to the decoder (230). Through the demuxer (220) and decoder (230), the first audio data block (412) of the second media data (314) corresponding to the start_time (e.g., 5,000 ms) of the second media data (314) can be stored in the audio buffer (240). Accordingly, the first audio data block (412) of the second media data (314) can include audio data generated from at least one audio frame of TS=4,991 ms, similar to the last audio data block (402) of the first media data (312).

[0074] The first audio data block (412) of the second media data (314) may include the initial decoding result of the second media data (314) through a separate decoding process from the first media data (312). Since a decoder (e.g., decoder (230) of FIG. 2) decodes the front audio frames of the second media data (314) without any previous audio frames to refer to, the PCM data of the first audio data block (412) of the second media data (314) may be inconsistent (420) with the PCM data of the last audio data block (402) of the first media data (312). Due to the above PCM data mismatch (420), when the electronic device (101) (e.g., processor (200)) plays audio data (404) prior to the last playback position (406) corresponding to the end_time (e.g., 5,000 ms) of the last audio data block (402), and then plays audio data (414) after the playback start position (416) corresponding to the start_time (e.g., 5,000 ms) of the first audio data block (412), audio continuity may not be guaranteed and audio interruption may occur.

[0075] FIG. 5 is a diagram illustrating audio interruption due to inaccuracy of a playback start position according to one embodiment of the present disclosure.

[0076] Referring to FIG. 5, the electronic device (101) (e.g., the processor (200) of FIG. 2) may output audio data (506) prior to the end_time of the first media data (312) to a speaker (e.g., the speaker (250) of FIG. 2) for audio playback of the first media data (e.g., the first media data (312) of FIG. 3). The last audio data block (502) of the first media data (312) that passes through a demuxer (e.g., the demuxer (220) of FIG. 2) and a decoder (e.g., the decoder (230) of FIG. 2) and is stored in an audio buffer (e.g., the audio buffer (240) of FIG. 2) may have, for example, a size of TS=4,991 ms and 4,096 bytes. An electronic device (101) (e.g., a processor (200)) can output audio data (506) up to the last playback position P1 (504) corresponding to end_time (e.g., 5,000 ms) among the last audio data block (502) of the first media data (312) to the speaker (250).

[0077] An electronic device (101) (e.g., processor (200)) can operate a demuxer (220) and a decoder (230) to output audio data corresponding to the start_time of second media data (314) for continuous playback of second media data (e.g., second media data (314) of FIG. 3). The electronic device (101) (e.g., processor (200)) can control the demuxer (220) to output at least one audio frame corresponding to the start_time (=5,000 ms) among the media files (310).

[0078] In the example of (a), due to inaccuracy in the location identification of the data to be demuxed, the demuxer (220) may not accurately find at least one audio frame corresponding to the 4,991 ms position closest to the start_time among the second media data (314) and may output at least one audio frame starting at an incorrect position (e.g., at the 4,996 ms position) to the decoder (230). As a result, the first audio data block (512) of the second media data (314) may include audio data (e.g., PCM data) of TS=4,996 ms.

[0079] The electronic device (101) (e.g., processor (200)) may consider that the first audio data block (512) includes audio data of TS=4,991 ms, and output audio data from the actual playback position P3 (516) based on the last playback position (504) of the first media data (312) to the speaker (250). However, the desired playback position P2 (514) including audio data corresponding to the last playback position (504) of the first media data (312) is actually before the actual playback position P3 (516), and thus, audio data in the section P2 to P3 may be omitted.

[0080] In the example of (b), due to inaccuracy in the location confirmation of the data to be demuxed, the demuxer (220) may not accurately find the audio frame corresponding to the audio data at the position 4,991 ms closest to the start_time among the second media data (314) and may output at least one audio frame starting at an incorrect position (e.g., at the position 4,986 ms) to the decoder (230). As a result, the first audio data block (522) of the second media data (314) may include audio data (e.g., PCM data) of TS=4,986 ms.

[0081] The electronic device (101) (e.g., processor (200)) may consider that the first audio data block (522) includes audio data of TS=4,991 ms, and output audio data from the actual playback position P4 (526) based on the last playback position (504) of the first media data (312) to the speaker (250). However, the desired playback position P5 (524), which actually includes audio data corresponding to the last playback position (504) of the first media data (312), is later than the actual playback position P4 (526), ​​and thus, audio data in the section P4 to P5 may be output in a duplicated manner.

[0082] As described above, due to the absence of a reference frame during decoding or inaccuracy in determining the location to be demuxed, audio data of the second media data (314) may overlap with some audio data of the first media data (312), or some audio data of the second media data (314) may be omitted from audio playback, resulting in audio interruption.

[0083] FIG. 6 is a diagram illustrating playback of media files including overlapping shooting sections according to one embodiment of the present disclosure.

[0084] Referring to FIG. 6, the first media file (602) and the second media file (604) may each be captured individually and may include at least partially duplicated data (610), for example, through a motion photo function. When the electronic device (101) (e.g., the processor (200) of FIG. 2) successively plays the first media file (602) and the second media file (604) including the duplicated data (610), the audio and video of the duplicated data (610) may be output in duplicate, so a user watching the video may determine that the audio playback quality is poor.

[0085] As described above, when first media data (e.g., first media data (312) of FIG. 3 or first media file (602) of FIG. 6) that may include duplicate data and second media data (e.g., second media data (314) of FIG. 3 or second media file (604) of FIG. 6) are played continuously, the user may experience an unpleasant experience, such as a tick sound, due to audio interruption or audio duplication that may occur, which may cause discomfort.

[0086] In embodiments of the present disclosure, when first media data (e.g., first media data (312) or first media file (602)) and second media data (e.g., second media data (314) or second media file (604)) are played consecutively, the electronic device (101) (e.g., processor (200)) can obtain the last audio data block corresponding to the end point of the first media data and mark the last playback position within the last audio data block.

[0087] In embodiments of the present disclosure, media data may include audio data (e.g., compressed audio frames), video data (e.g., compressed video frames), and metadata, and media data collected over a specified period of time or a specified amount of time and stored in memory with a single extension may be defined as a media file (e.g., media file (310)). In embodiments of the present disclosure, the first media data or the second media data may be defined as a unit of media data that is input to a demuxing and decoding process for audio playback. In embodiments of the present disclosure, the first media data or the second media data may be defined as one media file, or at least a portion of one media file.

[0088] In embodiments of the present disclosure, an audio data block (e.g., a first audio data block or a second audio data block) may be defined as a single data unit including PCM data of a specified size generated through decoding. In one embodiment, an audio data block may be defined as a set of PCM data generated by decoding a specified number (e.g., one or more) of audio frames output from a demuxer (e.g., a demuxer (220) of FIG. 2) by a decoder (e.g., a decoder (230) of FIG. 2). In embodiments of the present disclosure, the first audio data block or the second audio data block may be the minimum unit of segmentation processing for audio reproduction.

[0089] In embodiments of the present disclosure, the electronic device (101) (e.g., the processor (200)) can accurately determine the playback start position of the second media data by obtaining a first audio data block to be searched for, corresponding to the start time of the second media data, and searching for audio data identical to the audio data of the last playback position in the second audio data block.

[0090] Embodiments of the present disclosure can prevent audio interruption when a single media file (e.g., media file (310)) is divided and played continuously, and when media files containing overlapping shooting sections (e.g., media files (602, 604) are played continuously, overlapping data can be removed and audio and video can be played continuously without interruption.

[0091] FIG. 7 is a block diagram illustrating a configuration for continuous audio playback according to one embodiment of the present disclosure.

[0092] Referring to FIG. 7, the electronic device (101) (e.g., the processor (200) of FIG. 2) may include a last audio data determination unit (712), a last playback position determination unit (714), a search target determination unit (716), an audio data search unit (718), and a playback start position determination unit (720). In one embodiment, at least one of the audio data determination unit (712), the last playback position determination unit (714), the search target determination unit (716), the audio data search unit (718), or the playback start position determination unit (720) may be implemented as a software module executed by the processor (200). According to one embodiment, at least one of the audio data determination unit (712), the last playback position determination unit (714), the search target determination unit (716), the audio data search unit (718), or the playback start position determination unit (720) may be implemented as a separate processor.

[0093] According to one embodiment, the last audio data determination unit (712) may receive a first audio data block (702) including PCM data generated by decoding at least one audio frame of the first media data (e.g., the first media data (312) or the first media file (602)) from the audio buffer (240), and may determine that the first audio data block (702) includes PCM data corresponding to the end time (end_time) of the first media data (702). The last audio data determination unit (712) may read the first audio data block (702) from the audio buffer (240) and back it up (e.g., copy and store) in another memory space. In one embodiment, the first audio data block (702) may be stored in the memory (215).

[0094] According to one embodiment, the last playback position determining unit (714) can determine the last playback position (722) of the first audio data block (702). In one embodiment, the last playback position (722) can include a byte offset based on a start point of the first audio data block (702). For example, the last playback position (722) can indicate the last byte of audio data (e.g., audio data (404) or audio data (506)) used for audio playback among the first audio data block (702).

[0095] According to one embodiment, the search target determination unit (716) controls the demuxer (220) to output at least one audio frame to be the search target among the audio frames constituting the second media data (e.g., the second media data (314) or the second media file (604)), and receives a second audio data block (704) including PCM data generated by decoding the at least one audio frame by the decoder (230) from the audio buffer (240).

[0096] According to one embodiment, the audio data search unit (718) can determine search reference audio data among the first audio data block (702) based on the last playback position (722) determined by the last playback position determination unit (714). The audio data search unit (718) can search the second audio data block using the search reference audio data, determine whether the second audio data block includes audio data matching the search reference audio data, and determine the position of the audio data matching the search reference audio data.

[0097] According to one embodiment, the playback start position determining unit (720) may determine the position found by the audio data search unit (718) as the playback start position (724) for the second media data.

[0098] FIG. 8 is a flowchart illustrating a method for continuous audio playback according to one embodiment of the present disclosure. In one embodiment, at least one of the operations described below may be executed by an electronic device (101) (e.g., processor (200) of FIG. 2 ). According to embodiments, at least one of the operations described below may be omitted, modified, or the order of operations may be changed. For example, at least one of the operations described below may be performed in parallel with another operation, or may be performed regardless of the order shown.

[0099] Referring to FIG. 8, in operation 810, the electronic device (101) (e.g., the processor (200) of FIG. 2) may obtain a first audio data block corresponding to a specified end time point (e.g., end_time) of the first media data from the first media data (e.g., the first media data (312) of FIG. 3 or the first media file (602) of FIG. 6). In one embodiment, the electronic device (101) (e.g., the processor (200)) may obtain at least one audio frame from among a plurality of audio frames of the first media data at the latter part by a demuxer (e.g., the demuxer (220) of FIG. 2), and read the first audio data block including PCM data obtained by decoding the at least one audio frame by a decoder (e.g., the decoder (230) of FIG. 2) from an audio buffer (e.g., the audio buffer (240)). In one embodiment, operation 810 may include at least one of operation 902, operation 904, or operation 906 of FIG. 9.

[0100] In operation 820, the electronic device (101) (e.g., the processor (200)) may identify a first data position (e.g., a last playback position (722 of FIG. 7)) related to the last audio data to be used for audio playback among the first audio data block based on the end time point. In one embodiment, the last playback position may indicate a position of the last audio data (e.g., one byte) used for audio playback among the first audio data block, or may indicate a position of the next data (e.g., one byte) of the last audio data used for audio playback among the first audio data block. For example, the last playback position may include a byte offset based on a start point of the first audio data block. For example, the last playback position may indicate at which byte the last audio data or the next data is located among the first audio data blocks.

[0101] In one embodiment, operation 820 may include operation 908 of FIG. 9. In one embodiment, the electronic device (101) (e.g., processor (200)) may determine the last playback position based on at least one of a timestamp and a size of the first audio data block, or an end time (end_time) of the first media data.

[0102] In operation 830, the electronic device (101) (e.g., processor (200)) may continue to play (e.g., output to speaker (250)) audio data up to the last playback position of the first audio data block. Although operation 830 is illustrated here as being performed after operations 810 and 820, the electronic device (101) (e.g., processor (200)) may perform at least one of operations 810, 820, 840, 850, or 860 while playing audio data of the first media data up to the last playback position. In one embodiment, operation 830 may include operation 910 of FIG. 9. In one embodiment, the electronic device (101) (e.g., processor (200)) may complete audio playback of the first media data after playing audio data up to the last playback position. According to one embodiment, the electronic device (101) (e.g., processor (200)) may complete audio playback of the first media data after playing audio data up to the last playback position and audio data at the last playback position.

[0103] In operation 840, the electronic device (101) (e.g., the processor (200)) may obtain a second audio data block corresponding to a specified start time of second media data (e.g., the second media data (314) or the second media file (604)). In one embodiment, the electronic device (101) (e.g., the processor (200)) may obtain one or more audio frames from among a plurality of audio frames of the second media data by a demuxer (220), and read the second audio data block including PCM data obtained by decoding the one or more audio frames by a decoder (230) from an audio buffer (240). In one embodiment, operation 840 may include the procedure of FIG. 13.

[0104] In operation 850, the electronic device (101) (e.g., the processor (200)) may search for audio data corresponding to the last playback position in the second audio data block. In one embodiment, the electronic device (101) (e.g., the processor (200)) may determine search reference audio data among the first audio data block (702) based on the last playback position identified in operation 820, and search for audio data matching the search reference audio data in the second audio data block. In one embodiment, operation 850 may include the procedure of FIG. 17.

[0105] In operation 860, the electronic device (101) (e.g., processor (200)) may determine a data position (e.g., playback start position (724)) related to the first audio data to be used for audio playback among the second audio data blocks based on the searched audio data. In one embodiment, the playback start position may indicate a start position (e.g., byte position) of the searched audio data, or may indicate a previous position of the searched audio data. For example, the playback start position may include a byte offset based on a start point of the second audio data block.

[0106] In operation 870, the electronic device (101) (e.g., processor (200)) may output audio data (e.g., PCM data) of the second media data from the playback start position of the second audio data block to the speaker (250) after audio playback of the first media data is completed based on the last playback position. According to one embodiment, the electronic device (101) (e.g., processor (200)) may complete audio playback of the first media data by outputting audio data up to the last playback position of the first audio data block to the speaker (250). Then, the electronic device (101) (e.g., processor (200)) may output audio data of the playback start position and audio data after the playback start position to the speaker (250).

[0107] FIG. 9 is a flowchart illustrating a procedure for determining a final playback position according to one embodiment of the present disclosure. In one embodiment, at least one of the operations described below may be executed by the electronic device (101) (e.g., the processor (200)). According to embodiments, at least one of the operations described below may be omitted, modified, or the order of operations may be changed. For example, at least one of the operations described below may be performed in parallel with another operation, or may be performed regardless of the order shown. In one embodiment, the procedure of FIG. 9 may correspond to operations 810, 820, and 830 of FIG. 8.

[0108] Referring to FIG. 9, in operation 902, the electronic device (101) (e.g., the processor (200) of FIG. 2) may demux and decode first media data (e.g., the first media data (312) of FIG. 3 or the first media file (602) of FIG. 6) to obtain a first audio data block (e.g., the first audio data block (702) of FIG. 7) including PCM data corresponding to at least one audio frame. In one embodiment, the electronic device (101) (e.g., the processor (200)) may read the first audio data block from an audio buffer (e.g., the audio buffer (240) of FIG. 2).

[0109] In operation 904, the electronic device (101) (e.g., processor (200)) may determine whether the first audio data block is the last audio data block of the first media data.

[0110] In one embodiment, when end_time indicating an end point of first media data (e.g., a playback end time set by the processor (200) to identify the first media data (312) among the media files (310)) is set, the electronic device (101) (e.g., the processor (200)) can determine whether the first audio data block includes audio data corresponding to end_time based on a timestamp and a size of the first audio data block. The electronic device (101) (e.g., the processor (200)) can identify the timestamp of the audio data block from at least one audio frame prior to demuxing and decoding corresponding to the audio data block.

[0111] For example, the electronic device (101) (e.g., the processor (200)) can calculate a playback time length (e.g., ms or μs) corresponding to a size (e.g., bytes) of the first audio data block based on at least one of a sampling rate, a channel count, or a sample byte size applied to the first media data. The electronic device (101) (e.g., the processor (200)) can determine that the first audio data block is the last audio data block of the first media data when the playback end time of the first audio data block, which is calculated as the sum of the timestamp and the playback time length of the first audio data block, is later than the end_time. When the playback end time of the first audio data block is earlier than the end_time, the electronic device (101) (e.g., the processor (200)) can determine that the first audio data block is not the last audio data block of the first media data.

[0112] In one embodiment, if end_time indicating the end point of the first media data is not set (for example, if the first media data includes the first media file (602)), the electronic device (101) (for example, the processor (200)) can determine whether the first audio data block includes the last decoding result in which audio data exists. If there is no subsequent decoding result, the electronic device (101) (for example, the processor (200)) can determine that the first audio data block is the last audio data block of the first media data. If there is a subsequent decoding result, the electronic device (101) (for example, the processor (200)) can determine that the first audio data block is not the last audio data block of the first media data.

[0113] According to one embodiment, if it is determined that the first audio data block is the last audio data block of the first media data, the electronic device (101) (e.g., processor (200)) may proceed to operation 906. On the other hand, if it is determined that the first audio data block is not the last audio data block of the first media data, the electronic device (101) (e.g., processor (200)) may proceed to operation 912.

[0114] In operation 912, the electronic device (101) (e.g., processor (200)) can output audio data of the first audio data block to the speaker (250).

[0115] At operation 906, the electronic device (101) (e.g., processor (200)) may back up (e.g., copy and store) the first audio data block to a separate memory area (e.g., memory (215)) so that it can be used for subsequent audio navigation (e.g., operation 850).

[0116] In operation 908, the electronic device (101) (e.g., the processor (200)) may identify a last playback position corresponding to the end_time of the first media data from the first audio data block. The last playback position may be calculated based on metadata of the first media data (e.g., at least one of a sampling rate, a channel count, or a sample byte size), a timestamp of the first audio data block (1100), or an end_time set for at least one of the first media data. An embodiment of operation 908 may be described below with reference to FIG. 10.

[0117] In operation 910, the electronic device (101) (e.g., the processor (200)) may output audio data (e.g., audio data to be used (1102)) up to the last playback position of the first audio data block to the speaker (250). The remaining audio data (e.g., audio data to be skipped (1104)) after the last playback position may not be output to the speaker (250) and may be skipped. For example, the remaining audio data may be deleted immediately or after a specified time.

[0118] FIG. 10 is a flowchart illustrating a procedure for determining a final playback position for an audio data block according to one embodiment of the present disclosure. In one embodiment, at least one of the operations described below may be executed by the electronic device (101) (e.g., the processor (200) of FIG. 2 ). According to embodiments, at least one of the operations described below may be omitted, modified, or the order of operations may be changed. For example, at least one of the operations described below may be performed in parallel with another operation, or may be performed regardless of the order shown. In one embodiment, the procedure of FIG. 10 may correspond to operation 908 of FIG. 9 .

[0119] Referring to FIG. 10, in operation 1002, the electronic device (101) (e.g., the processor (200) of FIG. 2) may identify a timestamp (TS) and a size of a first audio data block. The electronic device (101) (e.g., the processor (200)) may pre-store metadata (e.g., at least one of a sampling rate (sampling_rate), a channel count (channel_count), or a sample byte size (sample_size_byte)) related to the first media data (e.g., the first media data (312) of FIG. 3 or the first media file (602) of FIG. 6)) or read the metadata from a memory (e.g., the memory (215) of FIG. 2).

[0120] In operation 1004, the electronic device (101) (e.g., processor (200)) may determine whether there is an end_time indicating a playback end point set for the first media data. For example, an end_time corresponding to a division position (e.g., 5,000 ms) designated by the electronic device (101) (e.g., processor (200)) may be set for the first media data identified for segmented playback (e.g., first media data (312)). For example, individually generated first media data (e.g., first media file (602)) may not have an end_time. If there is an end_time, the electronic device (101) (e.g., processor (200)) may proceed to operation 1006. If there is no end_time, the electronic device (101) (e.g., processor (200)) may proceed to operation 1012.

[0121] In operation 1006, the electronic device (101) (e.g., processor (200)) can calculate the length of the playback time of the first audio data block based on the size of the first audio data block.

[0122] For example, the parameter values ​​used to calculate the last playback position are as follows. The examples below are described using milliseconds (ms) as the time unit, but other examples using microseconds (μs) may also be possible.

[0123] - sampling_rate: 48,000

[0124] - channel_count : 2 (e.g. left channel and right channel)

[0125] - sample_size_byte: 2

[0126] - Timestamp (TS) of the first audio data block: 4,991 ms

[0127] - Size of the first audio data block (pcm_byte): 4,096 bytes

[0128] - end_time: 5,000ms

[0129] The electronic device (101) (e.g., processor (200)) can calculate the playback time length (e.g., pcmByteToTimeMs) of the first audio data block based on the size (e.g., pcm_byte) of the first audio data block as follows.

[0130] pcmByteToTimeMs = pcm_byte × 1,000,000 / sample_rate / channel_count / sample_size_byte

[0131] 21.333ms = 4,096 × 1,000,000 / 48,000 / 2 / 2

[0132] As above, the playback time corresponding to the first audio data block is 21.333 ms.

[0133] In operation 1008, the electronic device (101) (e.g., processor (200)) can calculate the audio size of the first audio data block to be skipped and not played.

[0134] In the above example, adding the playback time length to the timestamp of the first audio data block is 4,991 + 21.333 = 5012.333 ms, and in order to play up to 5,000 ms, which is the end_time, audio data corresponding to the last 12.333 ms (= 5012.333 ms - 5000 ms) of the first audio data block (e.g., audio data to be skipped (1104)) can be skipped.

[0135] Let drop_time1 be 12.333ms (= 12,333㎲), and the electronic device (101) (e.g., processor (200)) can calculate the size of audio data to be skipped (e.g., skipAudioSize1) of the first audio data block as follows.

[0136] pre_skipAudioSize1 = sampling_rate × channel_count × sample_size_byte × drop_time1 / 1,000,000

[0137] skipAudioSize1 = FrameSize roundup (pre_skipAudioSize1)

[0138] 2,367 = 48,000 × 2 × 2 × 12,333 / 1,000,000

[0139] 2,368 = FrameSize roundup (2,368)

[0140] The electronic device (101) (e.g., processor (200)) may use the FrameSize roundup function to align the pre_skipAudioSize1 calculated from drop_time1 in units of frame size (e.g., FrameSize = channel_count × sample_size_byte). Since FrameSize = 2 × 2 = 4, the final size of audio data to be skipped may be calculated as 2,368 bytes, which is a multiple of FrameSize.

[0141] In operation 1010, the electronic device (101) (e.g., processor (200)) can determine the last playback position of the first audio data block based on the audio size to be skipped.

[0142] In the above-described example, since the size of the audio data to be skipped is 2,368 bytes, the electronic device (101) (e.g., the processor (200)) can calculate the size of the audio data to be used as 1,728 (= bcm_byte - skipAudioSize1 = 4,096 - 2,368). Then, the last playback position can be determined as 1,728, which represents a byte offset from the start point of the first audio data block to the last byte of the audio data to be used. For the first media data for which end_time is set, the last playback position can indicate the first audio data that was not output for audio playback among the last audio data block of the first media data (e.g., the first audio data block).

[0143] One embodiment of operations 1006, 1008 and 1010 will be described below with reference to FIG. 11.

[0144] In operation 1012, the electronic device (101) (e.g., processor (200)) can identify that the first audio data block is the last audio block of the first media data generated through demuxing and decoding because there is no end_time set for the first media data, and can determine the last position of the first audio data block as the last playback position.

[0145] FIG. 11 is a diagram illustrating an example of determining a final playback position for media data with an end point set according to one embodiment of the present disclosure.

[0146] Referring to FIG. 11, a first audio data block (1100) corresponding to the end_time of the first media data (e.g., the first media data (312) of FIG. 3) may have a size of, for example, 4,096 bytes and a timestamp of 4,991 ms. The electronic device (101) (e.g., the processor (200) of FIG. 2) may calculate the size of audio data (1104) to be skipped corresponding to the last 12.333 ms of the first audio data block (1100) as 2,368 bytes based on the playback time length (e.g., 21.333 ms) and the timestamp of the first audio data block (1100).

[0147] The last playback position (1110) can be calculated as 4,096 - 2,368 = 1,728. The electronic device (101) (e.g., processor (200)) can determine the front part of the last playback position (1110) as audio data (1102) to be used for audio playback, and determine the back part as audio data (1104) to be skipped for audio playback.

[0148] FIG. 12 is a diagram illustrating an example of determining a last playback position for media data for which an end point is not set according to one embodiment of the present disclosure.

[0149] Referring to FIG. 12, a first audio data block (1200) corresponding to the last audio data of the first media data (e.g., the first media file (602) of FIG. 6) may have, for example, a size of 4,096 bytes and a timestamp of 4,991 ms. The electronic device (101) (e.g., the processor (200) of FIG. 2) may determine 4,096, which indicates the last position of the first audio data block (1200), as the last playback position (1210). For the first media data for which end_time is not set, the last playback position (1210) may indicate the last audio data to be output for audio playback.

[0150] FIG. 13 is a flowchart illustrating a procedure for determining audio data to be searched according to one embodiment of the present disclosure. In one embodiment, at least one of the operations described below may be executed by the electronic device (101) (e.g., the processor (200) of FIG. 2 ). According to embodiments, at least one of the operations described below may be omitted, modified, or the order of operations may be changed. For example, at least one of the operations described below may be performed in parallel with another operation, or may be performed regardless of the order shown. In one embodiment, the procedure of FIG. 13 may correspond to operation 840 of FIG. 8 .

[0151] Referring to FIG. 13, in operation 1302, the electronic device (101) (e.g., the processor (200) of FIG. 2) may identify the size and start time (e.g., start_time) of the second media data. For example, the start_time indicating the start time of playback for the second media data identified for segmented playback (e.g., the second media data (314) of FIG. 3) may be set to a segmentation position (e.g., 5,000 ms) specified by the electronic device (101) (e.g., the processor (200)). For example, the start_time for individually generated second media data (e.g., the second media file (604)) may be 0.

[0152] In operation 1304, the electronic device (101) (e.g., processor (200)) may determine whether start_time is greater than 0. If start_time is greater than 0, the electronic device (101) (e.g., processor (200)) may proceed to operation 1306. If start_time is not greater than 0 (e.g., 0), the electronic device (101) (e.g., processor (200)) may proceed to operation 1308.

[0153] In operation 1306, the electronic device (101) (e.g., processor (200)) may set the demuxing start position to an audio frame that is a specified value (e.g., X=2) prior to start_time. In order to secure previous audio frames to be referenced for decoding the audio frame corresponding to start_time, the electronic device (101) (e.g., processor (200)) may determine an audio frame that is X (e.g., 2) prior to the audio frame having the closest timestamp to start_time as the demuxing start position of the second media data.

[0154] In operation 1308, the electronic device (101) (e.g., the processor (200)) may set the demuxing start position to 0. In operation 1310, the electronic device (101) (e.g., the processor (200)) may demux the audio frame of the demuxing start position (e.g., the demuxing start position (1406) of FIG. 14) and subsequent audio frames through a demuxer (e.g., the demuxer (220) of FIG. 2). At least one audio frame prior to start_time may be referenced in the decoding of the audio frame corresponding to start_time in the decoder (230). The decoder (230) may improve the decoding quality by decoding the audio frame corresponding to start_time with reference to the previous at least one audio frame, thereby obtaining the same decoding result as the PCM data corresponding to the first media data corresponding to end_time.

[0155] In operation 1312, the electronic device (101) (e.g., processor (200)) can decode audio frames output from the demuxer (220) by the decoder (230). The decoder (230) can store PCM data generated by decoding the audio frames in the audio buffer (250).

[0156] In operation 1314, the electronic device (101) (e.g., processor (200)) can determine whether the size (or playback time length) of audio data (e.g., PCM data) output from the decoder (230) and stored in the audio buffer (250) exceeds a specified threshold (e.g., TH1). If the size (or playback time length) of the audio data after decoding does not exceed TH1, the electronic device (101) (e.g., processor (200)) can return to operation 1312 to decode the next audio frame. If the size (or playback time length) of the audio data after decoding exceeds TH1, the electronic device (101) (e.g., processor (200)) can proceed to operation 1316.

[0157] In operation 1316, the electronic device (101) (e.g., processor (200)) may determine that audio data of a size larger than TH1 stored in the audio buffer (250) is a second audio data block to be searched. One embodiment of operations 1312, 1314, and 1316 may be described below with reference to FIG. 16.

[0158] FIG. 14 is a drawing for explaining setting of a demuxing start position according to one embodiment of the present disclosure.

[0159] Referring to FIG. 14, the demuxing audio frame table (1400) can store timestamp values ​​of audio frames input to a demuxer (e.g., the demuxer (220) of FIG. 2). For example, if the start_time set for demuxing the second media data to be segmented and played back is 5,000 ms, the timestamp closest to the start_time is 4,991 ms. The electronic device (101) (e.g., the processor (200) of FIG. 2) can determine the audio frame (1404) (e.g., an audio frame with a timestamp of 4947.33 ms) that is a specified value (e.g., 2) earlier than the audio frame (1402) having the timestamp of 4,991 ms closest to the start_time as the demuxing start position (1406).

[0160] In one embodiment, if there are no audio frames two prior to the audio frame corresponding to start_time (e.g., audio frame (1402)), the electronic device (101) (e.g., processor (200)) may determine one audio frame prior to the audio frame (e.g., audio frame with timestamp 4968.67 ms) as a demuxing start position. In one embodiment, if there are no audio frames prior to the audio frame corresponding to start_time (e.g., audio frame (1402)), the electronic device (101) (e.g., processor (200)) may determine one audio frame prior to the audio frame corresponding to start_time (e.g., audio frame (1402)) as a demuxing start position.

[0161] In one embodiment, the electronic device (101) (e.g., processor (200)) may determine an audio frame corresponding to a time position (e.g., 4,930 ms) prior to a specified time value (e.g., 70 ms) from start_time (e.g., an audio frame with a timestamp of 4,935 ms (not shown)) as the demuxing start position.

[0162] FIG. 15 is a drawing for explaining a demuxing operation from a demuxing start position according to one embodiment of the present disclosure.

[0163] Referring to FIG. 15, in operation 1502, the demuxer (e.g., the demuxer (220) of FIG. 2) may start demuxing at a demuxing start position (e.g., timestamp 4,948.33 ms) determined in, for example, operation 1306 or operation 1308. In operation 1504, the demuxer (220) may demux the second media data and output the first audio frame designated as the demuxing start position (e.g., audio frame (1514) with timestamp 4,948.33 ms). In operation 1506, the demuxer (220) may output the second audio frame (e.g., audio frame (1516) with timestamp 4,969.67 ms) from the demuxing start position. At operation 1508, the demuxer (220) can output the third audio frame from the demuxing start position (e.g., audio frame (1518) with timestamp 4,991 ms).

[0164] According to one embodiment, the audio frame (1518) with the timestamp of 4,991 ms corresponds to the start_time of the second media data, and a decoder (e.g., decoder (230) of FIG. 2) decodes the audio frame (1518) with the timestamp of 4,991 ms with reference to the audio frame (1514) with the timestamp of 4,948.33 ms and the audio frame (1516) with the timestamp of 4,969.67 ms, thereby outputting the same decoding result (e.g., PCM data) as the audio frame with the timestamp of 4,991 ms among the last audio data block (e.g., the first audio data block) of the first media data.

[0165] In one embodiment, the electronic device (101) (e.g., the processor (200) of FIG. 2) may transmit at least one of encoder padding information or encoder delay information indicating a silent audio data section acquired by the demuxer (220) to the decoder (230). The decoder (230) may discard the silent audio data within the silent audio data section without including it in the second audio data block based on at least one of the encoder padding information or the encoder delay information.

[0166] FIG. 16 is a diagram for explaining determination of a search target audio data block according to one embodiment of the present disclosure.

[0167] Referring to FIG. 16, the electronic device (101) (e.g., the processor (200) of FIG. 2) can control the demuxer (e.g., the demuxer (220) of FIG. 2) and the decoder (e.g., the decoder (230) of FIG. 2) to repeatedly perform demuxing and decoding until audio data having a size (or playback time length) greater than a specified threshold value (e.g., TH1) is secured. For example, when TH1 = 19,200 bytes (or 100 ms), an example of an operation (e.g., operation 1312, operation 1314, and operation 1316) for determining a second audio data block is described as follows.

[0168] In operation 1602, the decoder (230) decodes an audio frame with a timestamp of 4,948.33 ms set as the demuxing start position from the demuxer (220) to generate PCM data (1612) of 4,096 bytes in size and store it in an audio buffer (e.g., audio buffer (240) of FIG. 2). Since the PCM data (1612) is the first decoding result of the second media data, the size of the decoded audio data (1622) stored in the audio buffer (240) becomes 4,096 bytes.

[0169] In operation 1604, the decoder (230) decodes an audio frame with a timestamp of 4,969.67 ms from the demuxer (220) to generate PCM data (1614) of 4,096 bytes in size and store it in the audio buffer (240). The size of the decoded audio data (1624) stored in the audio buffer (240) becomes 8,192 bytes.

[0170] In operation 1606, the decoder (230) decodes an audio frame with a timestamp of 4,991 ms from the demuxer (220) to generate PCM data (1614) of 4,096 bytes in size and store it in the audio buffer (240). The size of the decoded audio data (1624) stored in the audio buffer (240) becomes 12,288 bytes.

[0171] In operation 1608, the decoder (230) decodes an audio frame with a timestamp of 5,012.33 ms from the demuxer (220) to generate PCM data (1618) of 4,096 bytes in size and store it in the audio buffer (240). The size of the decoded audio data (1628) stored in the audio buffer (240) becomes 16,384 bytes.

[0172] In operation 1610, the decoder (230) decodes an audio frame with a timestamp of 5,033.63 ms from the demuxer (220) to generate PCM data (1620) of 4,096 bytes in size and store it in the audio buffer (240). The size of the decoded audio data (1630) stored in the audio buffer (240) becomes 20,480 bytes.

[0173] The electronic device (101) (e.g., processor (200)) can determine that the decoded audio data (1630) stored in the audio buffer (240) for the second media data exceeds a size corresponding to 100 ms (e.g., 19,200 bytes), and can determine the decoded audio data (1630) stored in the audio buffer (240) as the second audio data block to be searched.

[0174] FIG. 17 is a flowchart illustrating a procedure for searching audio data according to one embodiment of the present disclosure. In one embodiment, at least one of the operations described below may be executed by the electronic device (101) (e.g., the processor (200) of FIG. 2 ). According to embodiments, at least one of the operations described below may be omitted, modified, or the order of operations may be changed. For example, at least one of the operations described below may be performed in parallel with another operation, or may be performed regardless of the order shown. In one embodiment, the procedure of FIG. 17 may correspond to operation 850 of FIG. 8 .

[0175] Referring to FIG. 17, in operation 1702, the electronic device (101) (e.g., the processor (200) of FIG. 2) may determine a search target channel. In one embodiment, the channel count of the first media data and the second media data may have a value of 2 or more, and the electronic device (101) (e.g., the processor (200)) may determine at least one channel (e.g., the L channel) to be used as a search target among a plurality of channels (e.g., the L channel and the R channel).

[0176] According to one embodiment, when each audio frame of the first media data and the second media data includes two or more audio channels (e.g., a left audio channel and a right audio channel), the electronic device (101) (e.g., the processor (200)) can more quickly and efficiently find the location of matching audio data in the first media data and the second media data by determining one of the audio channels as a search target.

[0177] According to one embodiment, if the value of the channel count is 1, all audio data can be compared. If the value of the channel count is greater than or equal to 2, the electronic device (101) (e.g., the processor (200)) can determine one audio channel (e.g., the left channel) as a search target. In one embodiment, the electronic device (101) (e.g., the processor (200)) can determine one or more audio channels (e.g., the left channel and the right channel) as search targets for more accurate audio search. One embodiment of operation 1702 will be described below with reference to FIG. 18.

[0178] In operation 1704, the electronic device (101) (e.g., processor (200)) can calculate the search reference data size (e.g., TH2 = FrameSize × NofFrames = 4 × 8 = 32 bytes) from the frame size (e.g., FrameSize = 4) and the specified number of search reference frames (e.g., NofFrames = 8). The electronic device (101) (e.g., processor (200)) can set the number of search reference frames according to any criterion.

[0179] In operation 1706, the electronic device (101) (e.g., processor (200)) may determine whether the size of audio data to be skipped (e.g., skipAudioSize1) according to the last playback position of the first audio data block is greater than or equal to the search reference data size. If audio data to be skipped that is not used for audio playback is decoded from a number of audio frames that is greater than or equal to the number of search reference data frames to be used for audio search, the size of the audio data to be skipped may be greater than or equal to the search reference data size. If the size of the audio data to be skipped is greater than or equal to the search reference data size, the electronic device (101) (e.g., processor (200)) may proceed to operation 1708. If the size of the audio data to be skipped is less than the search reference data size, the electronic device (101) (e.g., processor (200)) may proceed to operation 1710.

[0180] In operation 1708, the electronic device (101) (e.g., processor (200)) may determine audio data starting from the last playback position of the first audio data block and thereafter as search reference data. In one embodiment, the search reference data may include audio data (e.g., PCM data) of the size of the search reference data among audio data (e.g., audio data to be skipped (1104)) that is not used for audio playback of the first audio data block. In one embodiment, the search reference data may include PCM data of at least one channel (e.g., L channel) selected from among the PCM data starting from the last playback position of the first audio data block and thereafter. One embodiment of operations 1704, 1706, and 1708 will be described below with reference to FIG. 19.

[0181] In operation 1710, the electronic device (101) (e.g., the processor (200)) may determine audio data up to the last playback position of the first audio data block as search reference data. In one embodiment, the search reference data may include audio data (e.g., PCM data) of the size of the search reference data among audio data used for audio reproduction of the first audio data block (e.g., audio data to be skipped (1104)). In one embodiment, the search reference data may include PCM data of at least one channel (e.g., L channel) selected from among the last playback position of the first audio data block and the PCM data before it. When the search reference data includes audio data used for audio reproduction, the electronic device (101) (e.g., the processor (200)) may determine audio data after audio data matching the search reference data among the second media data as a playback start position.

[0182] In one embodiment, if end_time is not set for the first media data, as described in FIG. 12, the last playback position (1210) may be determined as the last position of the first audio data block (1200), and the electronic device (101) (e.g., processor (200)) may determine audio data prior to the last position of the first audio data block (1200) as the search reference data. One embodiment of operation 1710 will be described below with reference to FIG. 20.

[0183] In operation 1712, the electronic device (101) (e.g., processor (200)) may determine whether the start_time set for the second media data is greater than 0. If the start_time is greater than 0, the electronic device (101) (e.g., processor (200)) may proceed to operation 1714. If the start_time is not greater than 0 (e.g., 0), the electronic device (101) (e.g., processor (200)) may proceed to operation 1716.

[0184] In operation 1714, the electronic device (101) (e.g., processor (200)) can search for a second audio data block starting from a data location corresponding to start_time. In operation 1716, the electronic device (101) (e.g., processor (200)) can search for a second audio data block starting from a start location of the second audio data block. In operations 1714 and 1716, the electronic device (101) (e.g., processor (200)) can search for audio data matching search reference data in the second audio block.

[0185] In one embodiment, the electronic device (101) (e.g., processor (200)) can set a search start position of a second audio data block based on a timestamp of a first audio data block. For example, if the timestamp of the first audio data block is 4,991 ms, the electronic device (101) (e.g., processor (200)) can determine a position of a second audio data block corresponding to 4,991 ms, and search for audio data matching the search reference data from the determined position.

[0186] In operation 1718, the electronic device (101) (e.g., processor (200)) may determine whether audio data matching the search reference data exists within the second audio data block. If audio data matching the search reference data exists, the electronic device (101) (e.g., processor (200)) may proceed to operation 1720. On the other hand, if audio data matching the search reference data does not exist, the electronic device (101) (e.g., processor (200)) may proceed to operation 1722.

[0187] In operation 1720, the electronic device (101) (e.g., processor (200)) may determine a playback start position based on a position of the searched audio data within a second audio data block. In one embodiment, the playback start position may be determined as a position of the searched audio data, or may be determined as a next byte of the searched audio data. In operation 1722, the electronic device (101) (e.g., processor (200)) may determine a start position (e.g., 0 byte) of the second audio data block as a playback start position.

[0188] An embodiment of operations 1712, 1714, 1718, and 1720 will be described below with reference to FIGS. 21, 22, and 23. An embodiment of operation 1722 will be described below with reference to FIG. 24.

[0189] FIG. 18 is a diagram for explaining determination of a search target channel according to one embodiment of the present disclosure.

[0190] Referring to FIG. 18, audio frames corresponding to the number of search reference frames (1800) (e.g., 8) specified among the first audio data blocks are illustrated. When the channel count is 2, each audio frame (e.g., audio frame (1802)) defined by a specified frame size (e.g., FrameSize) may include audio data of an L channel and audio data of an R channel.

[0191] For each audio frame (e.g., audio frame (1802)), audio data of the L channel and audio data of the R channel may be repeatedly arranged. When the sample byte size (sample_size_byte) is 2, the electronic device (101) (e.g., the processor (200) of FIG. 2) may calculate the frame size as follows based on the channel count and the sample byte size.

[0192] FrameSize = channel_count × sample_size_byte

[0193] An electronic device (101) (e.g., a processor (200) of FIG. 2) can identify audio data (e.g., PCM data) of an L channel at the starting point of audio data, and move by FrameSize = 4 bytes to identify audio data (e.g., PCM data) of the next L channel.

[0194] FIG. 19 is a diagram illustrating determination of search reference data based on the last playback position according to one embodiment of the present disclosure.

[0195] Referring to FIG. 19, according to one embodiment, a first audio data block (1900) corresponding to the end_time of the first media data (e.g., the first media data (312)) may have a size of 4,096 bytes and a timestamp of 4,991 ms. According to one embodiment, the first audio data block (1900) may be divided into 1,728 bytes of audio data to be used (1902) and 2,368 bytes of audio data to be skipped (1904) based on the last playback position (1906).

[0196] An electronic device (101) (e.g., a processor (200) of FIG. 2) may determine, among audio data (1904) to be skipped, a search reference data (1908) of a specified size (e.g., 32 bytes) starting from a last playback position (1906) (e.g., 1,728 bytes). The search reference data (1908) may include PCM data of the specified search reference data size (e.g., 32 bytes). In one embodiment, the search reference data (1908) may include PCM data of at least one channel (e.g., L channel) selected from among the last playback position of the first audio data block and the subsequent PCM data.

[0197] The electronic device (101) (e.g., processor (200)) can determine the seek reference data (1908) starting from the last playback position (1906) (e.g., 1,728 bytes) based on the last playback position (1906) indicating the location of data not used for audio playback.

[0198] FIG. 20 is a diagram illustrating determination of search reference data based on the last position according to one embodiment of the present disclosure.

[0199] Referring to FIG. 20, the first audio data block (2000) may include the last audio data (2002) of the first media data for which end_time is not set, and all audio data (2002) of the first audio data block (2000) may be used for audio playback. The electronic device (101) (e.g., the processor (200) of FIG. 2) may determine audio data of a specified size (e.g., 32 bytes) prior to the last playback position (2004) indicating the last position of the first audio data block (2000) as search reference data (2006).

[0200] An electronic device (101) (e.g., a processor (200)) may determine audio data (e.g., PCM data) of a specified search reference data size (e.g., 32 bytes) among audio data (2002) used for audio reproduction of a first audio data block (2000) as search reference data (2006). In one embodiment, the search reference data (2000) may include PCM data of at least one channel (e.g., L channel) selected from among the last reproduction position (2004) of the first audio data block (2000) and the PCM data before that.

[0201] FIG. 21 is a diagram for explaining audio data search according to one embodiment of the present disclosure.

[0202] Referring to FIG. 21, a first audio data block (2100) corresponding to the end_time of the first media data (e.g., the first media data (312) of FIG. 3) may have, for example, a size of 4,096 bytes and a timestamp of 4,991 ms. The first audio data block (2100) may be divided into audio data to be used (2102) of, for example, 1,728 bytes and audio data to be skipped (2104) of 2,368 bytes based on the last playback position (2106). The electronic device (101) (e.g., the processor (200) of FIG. 2) may determine, among the audio data to be skipped (2104), search reference data (2108) of a specified size (e.g., 32 bytes) starting from the last playback position (2106) (e.g., 1,728 bytes).

[0203] For channel count 2, the electronic device (101) (e.g., processor (200)) can use only the L channel for audio data search. The search reference data (2108) can include L channel audio data (e.g., PCM data) corresponding to a specified number (e.g., 8) of audio frames, and the electronic device (101) (e.g., processor (200)) can search the second audio data block (2110) to determine whether the second audio data block (2110) includes audio data that matches the search reference data (2108).

[0204] An electronic device (101) (e.g., a processor (200)) may set a search start position (2116) (e.g., 9,920 bytes) of a first audio data block (2110) corresponding to a start_time (e.g., 5,000 ms) set for the second media data, and may start searching for audio data matching the search reference data (2108) from the search start position (2116). In one embodiment, the second audio data block (2110) may include audio data demuxed and decoded from audio frames starting from an audio frame of 4,948.33 ms (e.g., audio frame (1404)) rather than an audio frame of 4,991 ms corresponding to start_time=5,000 ms (e.g., audio frame (1402)), and thus the seek start position (2116) corresponding to start_time of the second audio data block (2110) may be set to a non-zero data position (e.g., 9,920 bytes).

[0205] The start_time of the second media data is 5,000 ms, and 5,000 ms - 4,948.33 ms = 51.667 ms (= 51,667 μs). Here, if drop_time2 is 57,667 μs, the size of audio data to be skipped (e.g., skipAudioSize2) of the second audio data block (2110) can be calculated as follows. In the formula below, pre_skipAudioSize2 can be an input of the roundup function in the form of an integer value.

[0206] pre_skipAudioSize2 = sampling_rate × channel_count × sample_size_byte × drop_time2 / 1,000,000

[0207] skipAudioSize2 = FrameSize roundup (pre_skipAudioSize2)

[0208] 9,920.064 = 48,000 × 2 × 2 × 51,667 / 1,000,000

[0209] 9,920 = FrameSize roundup (9,920)

[0210] According to the above calculation formula, the electronic device (101) (e.g., processor (200)) can calculate the size of audio data to be skipped in audio data search among the second audio data blocks (2110) as 9,920 bytes. The size of audio data to be skipped becomes the search start position (2116) of the second audio data block (2110).

[0211] The electronic device (101) (e.g., processor (200)) can search for audio data matching the search reference data (2108) from the search start position (2116). If the matching data is not found in the audio data (2114) from the search start position (2116) to the last position of the second audio data block (2110), the electronic device (101) (e.g., processor (200)) can search again for the matching data in the audio data (2112) from the first position of the second audio data block (2110) to before (e.g., 9,916 bytes) the search start position (2116).

[0212] The electronic device (101) (e.g., processor (200)) can search for audio data matching the L channel audio data of the search reference data (2108) from the search start position (2116) of the second audio data block (2110) in the above-described order.

[0213] FIG. 22 is a diagram for explaining determination of a playback start position based on a search result according to one embodiment of the present disclosure.

[0214] Referring to FIG. 22, a first audio data block (2200) corresponding to the end_time of the first media data (e.g., the first media data (312) of FIG. 3) may have, for example, a size of 4,096 bytes and a timestamp of 4,991 ms. The first audio data block (2200) may be divided into audio data to be used (2102) of, for example, 1,728 bytes and audio data to be skipped (2204) of 2,368 bytes based on the last playback position (2206). The electronic device (101) (e.g., the processor (200) of FIG. 2) may determine, among the audio data to be skipped (2204), search reference data (2208) of a specified size (e.g., 32 bytes) starting from the last playback position (2206) (e.g., 1,728 bytes). In one embodiment, the search criteria data (2208) may include audio data corresponding to at least one selected channel (e.g., L channel audio data).

[0215] A second audio data block (2210) to be used as a search target may have, for example, a size of 20,480 bytes and a timestamp of 4,948.33 ms. The electronic device (101) (e.g., the processor (200)) may search the second audio data block (2210) to determine whether the second audio data block (2210) includes audio data that matches the search reference data (2208). In one embodiment, the electronic device (101) (e.g., the processor (200)) may search the second audio data block (2210) using only audio data (e.g., 16 bytes) corresponding to at least one channel selected from the search reference data (2208).

[0216] In one embodiment, the electronic device (101) (e.g., processor (200)) may find audio data (2216a) that matches the search reference data (2208) at a location corresponding to 4968.33 ms of the second audio data block (2110). The found location may be determined as a playback start location (2216). The electronic device (101) (e.g., processor (200)) may decide to skip audio data (2212) of 3,840 bytes in size prior to the playback start location (2216) from audio playback, and to use audio data (2214) of 16,640 bytes in size at and after the playback start location (2216) for audio playback.

[0217] In one embodiment, using the sampling rate, channel count, and sample byte size, the playback time corresponding to 3,840 bytes can be calculated as 20ms (= 3,840 × 1,000,000 / 48,000 / 2 / 2). In one embodiment, the electronic device (101) (e.g., processor (200)) can change the timestamp of the second audio data block (2210) to 5,000ms. In another embodiment, the electronic device (101) (e.g., processor (200)) can change the timestamp of the second audio data block (2210) to 4968.33ms (= 4948.33ms + 20ms). The electronic device (101) (e.g., processor (200)) can output synchronized video frames according to the changed timestamp (e.g., 4968.33ms).

[0218] FIG. 23 is a diagram for explaining determination of a playback start position based on a search result according to one embodiment of the present disclosure.

[0219] Referring to FIG. 23, the first audio data block (2300) may have, for example, a size of 4,096 bytes and a timestamp of 4,991 ms. The first audio data block (2300) may include the last audio data (2302) of the first media data for which end_time is not set, and all audio data (2302) (for example, 4,096 bytes) of the first audio data block (2300) may be used for audio playback. The electronic device (101) (for example, the processor (200) of FIG. 2) may determine audio data of a specified size (for example, 32 bytes) prior to the last playback position (2304) indicating the last position of the first audio data block (2300) as search reference data (2306).

[0220] An electronic device (101) (e.g., a processor (200)) may determine audio data (e.g., PCM data) of a specified search reference data size among audio data (2302) used for audio reproduction of a first audio data block (2300) as search reference data (2306). In one embodiment, the search reference data (2306) may include PCM data of at least one channel (e.g., L channel) selected from among the last reproduction position (2304) of the first audio data block (2300) and the PCM data before that.

[0221] A second audio data block (2310) to be used as a search target may have, for example, a size of 20,480 bytes and a timestamp of 0 ms. The electronic device (101) (e.g., the processor (200)) may search the second audio data block (2310) to determine whether the second audio data block (2310) includes audio data that matches the search reference data (2306). In one embodiment, the electronic device (101) (e.g., the processor (200)) may search the second audio data block (2310) using only audio data (e.g., 16 bytes) corresponding to at least one channel selected from the search reference data (2306).

[0222] In one embodiment, the electronic device (101) (e.g., processor (200)) may find audio data (2316a) that matches the search reference data (2306) at a location (2316) corresponding to 16,600 bytes of the second audio data block (2310). The playback start location (2318) may be determined as a location (e.g., 16,632 bytes) after the search reference data size (e.g., 32 bytes) from the searched location (2316). The electronic device (101) (e.g., processor (200)) may decide to skip audio data (2312) of 16,600 bytes before the playback start position (2318) and 32 bytes of seek reference data (2316a) after the sought position (2316) in audio playback, and to use audio data (2314) of 3,848 bytes at and after the playback start position (2318) for audio playback.

[0223] In one embodiment, the electronic device (101) (e.g., processor (200)) can change the timestamp of the second audio data block (2310) to 86.625 ms. The electronic device (101) (e.g., processor (200)) can output synchronized video frames according to the changed timestamp (=86.625 ms).

[0224] FIG. 24 is a diagram for explaining an operation according to a failure to search audio data according to one embodiment of the present disclosure.

[0225] Referring to FIG. 24, a second audio data block (2410) to be used as a search target may have a size of 20,480 bytes and a timestamp of 4,948.33 ms. The electronic device (101) (e.g., processor (200)) may not find audio data matching search reference data (not shown) in the second audio data block (2410). Then, the playback start position (2416) for the second audio data block (2410) may be determined to correspond to the start_time (e.g., 5,000 ms) set for the second media data.

[0226] In one embodiment, since the second audio data block (2410) includes audio data corresponding to a timestamp 4948.33 ms prior to the start_time, the electronic device (101) (e.g., the processor (200)) can calculate the size of audio data (2412) to be skipped in audio playback among the second audio data block (2410). Since the start_time of the second media data is 5,000 ms, the skip time of the second audio data block (2410) is 5,000 ms - 4,948.33 ms = 51.667 ms (= 51,667 μs). Here, the audio data size corresponding to 57,667 μs can be calculated as follows.

[0227] AudioSize = sampling_rate × channel_count × sample_size_byte × drop_time3 / 1,000,000

[0228] 9,920.064 = 48,000 × 2 × 2 × 51,667 / 1,000,000

[0229] According to the above calculation formula, the electronic device (101) (e.g., processor (200)) can determine the size of audio data to be skipped in audio playback among the second audio data block (2410) as 9,920 bytes. The size of audio data to be skipped becomes the playback start position (2416) of the second audio data block (2410), which corresponds to start_time = 5,000 ms. The electronic device (101) (e.g., processor (200)) can skip audio data (2412) prior to the playback start position (2416) in audio playback, and continuously output audio data (2414) at and after the playback start position (2416) to the speaker (250) after the playback of the first media data.

[0230] In one embodiment, the electronic device (101) (e.g., processor (200)) can determine a timestamp of a video frame to be output based on a timestamp of audio data output based on the playback start position, decode the video frame of the determined timestamp, and output it to the display module (160) so as to be synchronized with the audio data.

[0231] FIG. 25 is a diagram for explaining an operation of outputting video frames according to a changed timestamp of audio data according to one embodiment of the present disclosure.

[0232] Referring to FIG. 25, a first audio data block (2500) may have a size of 4,096 bytes and a timestamp of 4,991 ms. The first audio data block (2500) may include the last audio data (2502) of the first media data for which end_time is not set, and all audio data (2502) (e.g., 4,096 bytes) of the first audio data block (2500) may be used for audio playback. The electronic device (101) (e.g., the processor (200) of FIG. 2) may determine audio data of a specified size (e.g., 32 bytes) prior to the last playback position (2504) indicating the last position of the first audio data block (2500) as search reference data (2506).

[0233] An electronic device (101) (e.g., a processor (200)) may determine audio data (e.g., PCM data) of a specified search reference data size among audio data (2502) used for audio reproduction of a first audio data block (2500) as search reference data (2506). In one embodiment, the search reference data (2506) may include PCM data of at least one channel (e.g., L channel) selected from among the last reproduction position (2504) of the first audio data block (2500) and the PCM data before that.

[0234] A second audio data block (2510) to be used as a search target may have a size of 20,480 bytes and a timestamp of 0 ms. The electronic device (101) (e.g., processor (200)) may search the second audio data block (2510) to determine whether the second audio data block (2510) includes audio data that matches the search reference data (2506). In one embodiment, the electronic device (101) (e.g., processor (200)) may search the second audio data block (2510) using only audio data (e.g., 16 bytes) corresponding to at least one channel selected from the search reference data (2506).

[0235] In one embodiment, the electronic device (101) (e.g., processor (200)) may find audio data (2516a) that matches the search reference data (2506) at a location (2516) corresponding to 16,600 bytes of the second audio data block (2510). The playback start location (2518) may be determined as a location (e.g., 16,632 bytes) after the search reference data size from the searched location (2516). The electronic device (101) (e.g., processor (200)) may decide to skip audio data (2512) of 16,600+32 bytes before the playback start location (2518) from audio playback, and to use audio data (2514) of 3,848 bytes at and after the playback start location (2518) for audio playback.

[0236] In one embodiment, the timestamp of the playback start position (2518) for the second media data may be changed to 86.625 ms. The electronic device (101) (e.g., the processor (200)) may demux the second media data (e.g., the second media file (604)) by a demuxer (e.g., the demuxer (220) of FIG. 2) to obtain video frames (2520, 2522, 2524, 2526), ​​and decode the video frames (2520, 2522, 2524, 2526) by a video decoder (not shown). When the electronic device (101) (e.g., processor (200)) outputs (e.g., renders video) decoded video frames, the video frames (2520, 2522, 2524) of 0, 33.33 ms, and 66.67 ms prior to the timestamp (e.g., 86.625 ms) of the playback start position (2518) may be skipped (e.g., not rendered). The electronic device (101) (e.g., processor (200)) may output the video frame (2526) of 100 ms and subsequent video frames (not shown) to the display module (160) through video rendering.

[0237] Through the above operations, the electronic device (101) (e.g., processor (200)) can continuously play first media data (e.g., first media file (602)) and second media data (604)) including duplicated shooting sections (e.g., duplicate data (610)), and can output the duplicated audio sections and video sections without interruption.

[0238] An electronic device (101) and its operating method according to embodiments of the present disclosure can prevent audio interruption by dividing a media file (310) into first media data (312) and second media data (314) that may include duplicate data and playing them continuously, finding the exact last playback position of the first media data (312), and determining a playback start position of the second media data (314) based on the last playback position.

[0239] An electronic device (101) and its operating method according to embodiments of the present disclosure can prevent audio interruption by determining a last playback position of the first media file (602) and a second media file (604) including overlapping shooting sections when playing them consecutively, finding duplicate data in the second media file (604) using the last playback position, and outputting audio after excluding the duplicate data.

[0240] The electronic device (101) and its operating method according to embodiments of the present disclosure can provide a user with an experience of continuous playback of both video and audio without interruption by skipping over redundant data sections based on a changed audio timestamp and outputting video frames synchronized with audio frames.

[0241] An electronic device (101) according to one embodiment may include a memory (215) for storing instructions, a speaker (250), and at least one processor (200) operatively connected to the memory and the speaker. The instructions, when executed by the at least one processor, may cause the electronic device to obtain a first audio data block corresponding to an end time of the first media data from first media data. The instructions, when executed by the at least one processor, may cause the electronic device to identify a last playback position of the first audio data block based on the end time. The instructions, when executed by the at least one processor, may cause the electronic device to obtain a second audio data block to be used as a search target from the second media data based on a start time of second media data corresponding to the end time. The instructions, when executed by the at least one processor, may cause the electronic device to search for audio data corresponding to the last playback position in the second audio data block. The above instructions, when executed by the at least one processor, may cause the electronic device to determine a playback start position of the second audio data block based on the searched audio data. The above instructions, when executed by the at least one processor, may cause the electronic device to output audio data from the playback start position of the second media data to the speaker after audio playback of the first media data up to the last playback position of the first audio data block is completed.

[0242] In one embodiment, the first audio data block may include PCM (pulse code modulation) data obtained by decoding one or more audio frames of the first media data. In one embodiment, the second audio data block may include PCM data obtained by decoding one or more audio frames of the second media data.

[0243] In one embodiment, the instructions may cause the electronic device to generate decoded audio data by decoding a specified number of audio frames prior to and one or more audio frames subsequent to an audio frame corresponding to the start point in the second media data, and to obtain the second audio data block of a specified size including the decoded audio data.

[0244] In one embodiment, the instructions may cause the electronic device to determine search reference data including audio data of a specified size based on the last playback position in the first audio data block, search for audio data matching the search reference data in the second audio data block, and determine the playback start position based on the position of the audio data matching the search reference data.

[0245] In one embodiment, the instructions may cause the electronic device to select at least one search target channel based on a channel count of the first media data, and to compare audio data of the selected at least one search target channel of the search criteria data with the second audio data block.

[0246] In one embodiment, the instructions may cause the electronic device to calculate a size of audio data to be skipped and not played back among the first audio data blocks based on an end point of the first media data, determine whether the size of the audio data to be skipped is greater than a size of the search reference data, determine the search reference data to include audio data after the last playback position if the size of the audio data to be skipped is greater than a size of the search reference data, and determine the search reference data to include audio data before the last playback position if the size of the audio data to be skipped is not greater than a size specified for the search reference data.

[0247] In one embodiment, the commands may cause the electronic device to change a timestamp of the second audio data block based on the playback start position.

[0248] In one embodiment, the last playback position may be determined based on the size of the first audio data block, the sampling rate, the channel count, the sample byte size, and a timestamp indicating the start time of playback of the first audio data block.

[0249] In one embodiment, the commands may cause the electronic device to output audio data before and at the last playback position among the first audio data blocks to the speaker, and skip audio data after the last playback position among the first audio data blocks.

[0250] In one embodiment, the commands may cause the electronic device to skip audio data before the playback start position among the second audio data blocks, and output audio data at and after the playback start position among the second audio data blocks to the speaker.

[0251] An operating method of an electronic device (101) according to an embodiment may include an operation (810) of obtaining a first audio data block corresponding to an end time point of the first media data from first media data. The method may include an operation (820) of identifying a last playback position of the first audio data block based on the end time point. The method may include an operation (840) of obtaining a second audio data block to be used as a search target from the second media data based on a start time point of the second media data corresponding to the end time point. The method may include an operation (850) of searching for audio data corresponding to the last playback position in the second audio data block. The method may include an operation (860) of determining a playback start position of the second audio data block based on the searched audio data. The method may include an operation (870) of outputting audio data from the playback start position of the second media data to a speaker after audio playback of the first media data up to the last playback position of the first audio data block is completed.

[0252] In one embodiment, the first audio data block may include PCM (pulse code modulation) data obtained by decoding one or more audio frames of the first media data. In one embodiment, the second audio data block may include PCM data obtained by decoding one or more audio frames of the second media data.

[0253] In one embodiment, the operation of obtaining the second audio data block may include an operation of generating decoded audio data by decoding audio frames that are a specified number of times prior to an audio frame corresponding to the start point in time among the second media data and one or more audio frames thereafter, and an operation of obtaining the second audio data block of a specified size including the decoded audio data.

[0254] In one embodiment, the operation of searching for audio data corresponding to the last playback position may include an operation of determining search reference data including audio data of a specified size based on the last playback position in the first audio data block, and an operation of searching for audio data matching the search reference data in the second audio data block.

[0255] In one embodiment, the operation of searching for audio data corresponding to the last playback position may include the operation of selecting at least one search target channel based on a channel count of the first media data, and the operation of comparing audio data of the selected at least one search target channel of the search reference data with the second audio data block.

[0256] In one embodiment, the operation of determining the search reference data may include an operation of calculating a size of audio data to be skipped without audio playback among the first audio data blocks based on an end point of the first media data, an operation of determining whether the size of the audio data to be skipped is greater than the size of the search reference data, an operation of determining the search reference data to include audio data after the last playback position if the size of the audio data to be skipped is greater than the size of the search reference data, and an operation of determining the search reference data to include audio data before the last playback position if the size of the audio data to be skipped is not greater than a size specified for the search reference data.

[0257] In one embodiment, the method may further include changing a timestamp of the second audio data block based on the playback start position.

[0258] In one embodiment, the last playback position may be determined based on at least one of a size of the first audio data block, a sampling rate, a channel count, a sample byte size, or a timestamp indicating a playback start time of the first audio data block.

[0259] In one embodiment, the method may further include an operation of outputting audio data before and at the last playback position among the first audio data blocks to the speaker, and an operation of skipping audio data after the last playback position among the first audio data blocks.

[0260] In one embodiment, the operation of outputting audio data from the playback start position among the second media data to the speaker may include an operation of skipping audio data before the playback start position among the second audio data blocks, and an operation of outputting audio data from the playback start position and after the playback start position among the second audio data blocks to the speaker.

[0261] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, computer devices, portable communication devices (e.g., smartphones), portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments disclosed in this document are not limited to the aforementioned devices.

[0262] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C” can each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish the corresponding component from other corresponding components and do not limit the corresponding components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0263] The term "module" as used in this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0264] One embodiment of the present document may be implemented as software (e.g., program (140)) including one or more instructions stored in a storage medium (e.g., built-in memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (200)) of the machine (e.g., electronic device (101)), for example, a processor of the machine (e.g., master device or task performing device), may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0265] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smartphones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0266] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include a single or multiple entities. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101), Memory (215) for storing commands; Speaker (250); and At least one processor (200) operatively connected to the memory and the speaker, wherein the instructions, when executed by the at least one processor, cause the electronic device to: Obtaining a first audio data block corresponding to the end point of the first media data from the first media data, Identifying the last playback position of the first audio data block based on the above end point, Based on the start time of the second media data corresponding to the above end time, a second audio data block to be used as a search target is obtained from the second media data, Search for audio data corresponding to the last playback position in the second audio data block, Determine the start position of the playback of the second audio data block based on the above-mentioned searched audio data, An electronic device characterized in that after audio playback of the first media data up to the last playback position of the first audio data block is completed, audio data from the playback start position of the second media data is output to the speaker.

2. In the first paragraph, the first audio data block includes first PCM (pulse code modulation) data obtained by decoding one or more audio frames of the first media data, An electronic device, characterized in that the second audio data block includes second PCM data obtained by decoding one or more audio frames of the second media data.

3. In the first or second paragraph, the commands cause the electronic device to: An electronic device characterized in that it generates decoded audio data by decoding a specified number of audio frames prior to an audio frame corresponding to the start point of the second media data and one or more audio frames thereafter, and obtains a second audio data block of a specified size including the decoded audio data.

4. In any one of paragraphs 1 to 3, the commands cause the electronic device to: Determine search reference data including audio data of a specified size based on the last playback position in the first audio data block, Search for audio data matching the search reference data in the second audio data block, An electronic device characterized in that it determines the playback start position based on the position of audio data matching the above search reference data.

5. In paragraph 4, the commands cause the electronic device to: Selecting at least one search target channel based on the channel count of the above first media data, An electronic device characterized in that it compares audio data of at least one selected search target channel of the search reference data with the second audio data block.

6. In paragraph 4 or 5, the commands cause the electronic device to: Calculate the size of audio data to be skipped and not played among the first audio data blocks based on the end point of the first media data, Determine whether the size of the audio data to be skipped is larger than the size of the search reference data, If the size of the audio data to be skipped is greater than the size of the search reference data, the search reference data is determined to include audio data after the last playback position, An electronic device characterized in that the search reference data is determined to include audio data prior to the last playback position if the size of the audio data to be skipped is not greater than a specified size for the search reference data.

7. In any one of paragraphs 1 to 6, the commands cause the electronic device to: An electronic device characterized in that it changes the timestamp of the second audio data block based on the playback start position.

8. In any one of paragraphs 1 to 7, the last playback position is, An electronic device characterized in that the size of the first audio data block, the sampling rate, the channel count, the sample byte size, or the timestamp indicating the start time of playback of the first audio data block is determined based on at least one of the following.

9. In any one of paragraphs 1 to 8, the commands cause the electronic device to: Output audio data before and at the last playback position among the first audio data blocks to the speaker, An electronic device characterized in that audio data after the last playback position among the first audio data blocks is skipped.

10. In any one of paragraphs 1 to 9, the commands cause the electronic device to: Skip audio data prior to the playback start position among the above second audio data blocks, An electronic device characterized in that it outputs audio data from the playback start position and after the playback start position among the second audio data blocks to the speaker.

11. In the operating method of an electronic device (101), An operation (810) of obtaining a first audio data block corresponding to the end point of the first media data from the first media data, An operation (820) for identifying the last playback position of the first audio data block based on the above-mentioned end point, and An operation (840) of obtaining a second audio data block to be used as a search target from the second media data based on the start time of the second media data corresponding to the above end time, An operation (850) for searching audio data corresponding to the last playback position in the second audio data block; An operation (860) for determining a playback start position of the second audio data block based on the searched audio data; A method characterized by including an operation (870) of outputting audio data from after the playback start position among the second media data to a speaker after audio playback of the first media data based on the last playback position is completed.

12. In the 11th paragraph, the first audio data block includes PCM (pulse code modulation) data obtained by decoding one or more audio frames of the first media data, A method, characterized in that the second audio data block includes PCM data obtained by decoding one or more audio frames of the second media data.

13. In the 11th or 12th paragraph, the operation of obtaining the second audio data block comprises: An operation of generating decoded audio data by decoding audio frames that are a specified number of times before an audio frame corresponding to the start point of the second media data and one or more audio frames thereafter; A method characterized by comprising an operation of obtaining a second audio data block of a specified size including the decoded audio data.

14. In any one of paragraphs 11 to 13, the operation of searching for audio data corresponding to the last playback position comprises: An operation of determining search reference data including audio data of a specified size based on the last playback position in the first audio data block; A method characterized by comprising an operation of searching for audio data matching the search reference data in the second audio data block.

15. In the 14th paragraph, the operation of searching for audio data corresponding to the last playback position is, An operation of selecting at least one search target channel based on the channel count of the first media data, A method characterized by comprising an operation of comparing audio data of at least one selected search target channel of the search criteria data with the second audio data block.

Images