Electronic device, method, and non-transitory computer-readable recording medium for generating or reproducing video data including multi-source images in one track

Abstract

An electronic device is disclosed. The electronic device may comprise a processor. The electronic device may comprise a memory for storing instructions. The instructions, when executed by the processor, may cause the electronic device to acquire a first image comprising first frames through a first camera. The instructions, when executed by the processor, may cause the electronic device to acquire a second image comprising second frames through a second camera. The instructions, when executed by the processor, may cause the electronic device to generate video data including a first image and a second image in a single video track.

Description

Electronic device, method, and non-transitory computer-readable recording medium for generating or reproducing video data including multiple source images in one track

[0001] The following descriptions relate to electronic devices, methods, and non-transitory computer-readable recording media for generating or reproducing video data containing multiple source images in one track.

[0002] An electronic device may include two or more cameras. The electronic device may acquire images through the two or more cameras. The electronic device may generate video data that includes the images in different tracks.

[0003] The above information may be provided as background information (related art) to aid in understanding this document. None of the above-described content is claimed to be prior art related to this document or can be used to determine prior art.

[0004] An electronic device is disclosed. The electronic device may include a processor. The electronic device may include a memory storing instructions. The instructions, when executed by the processor, may cause the electronic device to acquire a first image including first frames through a first camera. The instructions, when executed by the processor, may cause the electronic device to acquire a second image including second frames through a second camera. The instructions, when executed by the processor, may cause the electronic device to generate video data including the first image and the second image in a single video track.

[0005] A non-transitory computer-readable storage medium is disclosed. The non-transitory computer-readable storage medium can store a program including instructions. The instructions, when executed by a processor of an electronic device, can cause the electronic device to obtain a first image including first frames through a first camera. The instructions, when executed by the processor of the electronic device, can cause the electronic device to obtain a second image including second frames through a second camera. The instructions, when executed by the processor of the electronic device, can cause the electronic device to generate video data including the first image and the second image in a single video track.

[0006] A method is disclosed. The method can be performed by an electronic device. The method can include an operation of acquiring a first image including first frames through a first camera. The method can include an operation of acquiring a second image including second frames through a second camera. The method can include an operation of generating video data including the first image and the second image in a single video track.

[0007] An electronic device is disclosed. The electronic device may include a display. The electronic device may include a processor. The electronic device may include a memory storing instructions. The instructions, when executed by the processor, may cause the electronic device to obtain video data as a single video track including a first image and a second image in the single video track. The instructions, when executed by the processor, may cause the electronic device to reproduce first frames of the first image according to timestamp numbers. The instructions, when executed by the processor, may cause the electronic device to reproduce second frames according to the timestamp numbers when an input event is received to change from the first frames to second frames of the second image while reproducing the first frames.

[0008] A non-transitory computer-readable recording medium is disclosed. The non-transitory computer-readable recording medium can store a program including instructions. The instructions, when executed by a processor of an electronic device including a display, can cause the electronic device to obtain video data as a single video track including a first image and a second image in the single video track. The instructions, when executed by the processor, can cause the electronic device to reproduce first frames of the first image according to timestamp numbers. The instructions, when executed by the processor, can cause the electronic device to reproduce second frames according to the timestamp numbers when an input event is received to change from the first frames to second frames of the second image while reproducing the first frames.

[0009] A method is disclosed. The method can be performed by an electronic device including a display. The method can include obtaining video data as a single video track including a first image and a second image in the single video track. The method can include playing back first frames of the first image according to timestamp numbers. The method can include playing back the second frames according to the timestamp numbers when, while playing back the first frames, an input event is received to change from the first frames to second frames of the second image.

[0010] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0011] FIG. 2A is a block diagram of electronic devices according to one embodiment.

[0012] FIG. 2b is a block diagram of programs for encoding an image, according to one embodiment.

[0013] FIG. 2c is a block diagram of programs for decoding an image, according to one embodiment.

[0014] FIG. 3A illustrates an example of frames included in images acquired by an electronic device through cameras.

[0015] FIG. 3b illustrates an example of frames included in images acquired by an electronic device through cameras.

[0016] FIG. 3c illustrates an example of an operation in which an electronic device arranges frames included in images acquired through cameras into one track.

[0017] FIG. 4a illustrates an example of frames acquired through multiple cameras according to one embodiment.

[0018] FIG. 4b illustrates an example of a structure in which frames of multiple images are aligned on one track, according to one embodiment.

[0019] Figure 4c illustrates an example of a structure in which frames of multiple images are aligned to different tracks.

[0020] FIG. 5a illustrates an example of a structure of video data according to one embodiment.

[0021] FIG. 5b illustrates an example of an operation for specifying the position of a frame according to the structure of video data, according to one embodiment.

[0022] FIG. 6a illustrates an example of a decoding order of frames according to an event, according to one embodiment.

[0023] FIG. 6b illustrates an example of a decoding order of frames according to an event, according to one embodiment.

[0024] FIG. 6c illustrates an example of a decoding order of frames according to an event, according to one embodiment.

[0025] FIG. 6d illustrates an example of an operation of decoding with reference to a frame of another image, according to one embodiment.

[0026] FIG. 7A illustrates an example of a frame of video data played by an electronic device according to one embodiment.

[0027] FIG. 7b illustrates an example of frames of video data played by an electronic device according to one embodiment.

[0028] FIG. 7c illustrates an example of a frame of video data played by an electronic device according to one embodiment.

[0029] FIG. 7d illustrates an example of frames of video data played by an electronic device according to one embodiment.

[0030] FIG. 8a illustrates an example of frames acquired through multiple cameras according to one embodiment.

[0031] FIG. 8b illustrates an example of the order of frames according to one embodiment.

[0032] FIG. 9 illustrates an example of multiple cameras having different fields of view, according to one embodiment.

[0033] FIG. 10 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0034] FIG. 11 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0035] FIG. 12 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0036] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0037] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0038] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0039] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0040] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0041] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0042] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0043] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0044] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0045] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0046] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0047] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0048] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0049] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0050] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0051] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0052] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0053] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0054] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for realizing eMBB, a loss coverage (e.g., 664 dB or less) for realizing mMTC, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 6 ms or less for round trip) for realizing URLLC.

[0055] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0056] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0057] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0058] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0059] FIG. 2A is a block diagram of electronic devices (201, 205) according to one embodiment.

[0060] Referring to FIG. 2A, the electronic device (201) may include a processor (221), a memory (231), a display (261), cameras (281, 283, 285), and / or a communication circuit (291). The processor (221) may correspond to the processor (120) of the electronic device (101) of FIG. 1. The memory (231) may correspond to the memory (130) of the electronic device (101) of FIG. 1. The display (261) may correspond to the display module (160) of the electronic device (101) of FIG. 1. The first camera (281) may correspond to the camera module (180) of the electronic device (101) of FIG. 1. The second camera (283) may correspond to the camera module (180) of the electronic device (101) of FIG. 1. The third camera (285) may correspond to the camera module (180) of the electronic device (101) of FIG. 1. The communication circuit (291) may correspond to the communication module (190) of the electronic device (101) of FIG. 1. The electronic device (201) of FIG. 2A may further include at least some of the components of the electronic device (101) of FIG. 1. For example, the electronic device (201) may include the input module (150) and / or the sensor module (176) of the electronic device (101). However, the present invention is not limited thereto.

[0061] In one embodiment, the cameras (281, 283, 285) of the electronic device (201) may have different camera performances. For example, the cameras (281, 283, 285) may have different lens properties. For example, the cameras (281, 283, 285) may have different at least one of an angle of view (AOV) (or focal length), a field of view (FOV), or an f-number. For example, the first camera (281) may be a wide-angle camera among a fisheye (e.g., 180 degrees), an ultra-wide-angle (e.g., 120 degrees), a wide-angle (e.g., 85 degrees), a telephoto (e.g., 36 degrees), or a super-telephoto (e.g., 11 degrees). The second camera (283) may be a telephoto camera having a different angle of view from the first camera (281). The third camera (285) may be an ultra-wide-angle camera having a different angle of view from the first camera (281) and the second camera (283). For example, the cameras (281, 283, 285) may be arranged at different locations on the electronic device (201). For example, one of the cameras (281, 283, 285) may be arranged on a first surface of the electronic device (201) and the other camera may be arranged on a second surface opposite to the first surface of the electronic device (201) so that the cameras (281, 283, 285) have different fields of view. For example, the cameras (281, 283, 285) may be arranged at different locations on the same surface of the electronic device (201) so that the cameras (281, 283, 285) have different fields of view. Hereinafter, it can be exemplified that the first camera (281) is a wide-angle camera, the second camera (283) is a telephoto camera, and the third camera (285) is an ultra-wide-angle camera.

[0062] In FIG. 2A, the electronic device (201) is illustrated as having three cameras, but this is merely an example. Depending on the embodiment, the electronic device (201) may include two or four or more cameras.

[0063] In one embodiment, the electronic device (205) may include a processor (225), a memory (235), a display (265), and / or a communication circuit (295). The processor (225) may correspond to the processor (120) of the electronic device (101) of FIG. 1. The memory (235) may correspond to the memory (130) of the electronic device (101) of FIG. 1. The display (265) may correspond to the display module (160) of the electronic device (101) of FIG. 1. The communication circuit (295) may correspond to the communication module (190) of the electronic device (101) of FIG. 1. The electronic device (205) of FIG. 2A may further include at least some of the components of the electronic device (101) of FIG. 1. For example, the electronic device (205) may include an input module (150) and / or a sensor module (176) of the electronic device (101), but is not limited thereto.

[0064] FIG. 2b is a block diagram of programs for encoding an image, according to one embodiment.

[0065] Figure 2b can be described with reference to the components of Figures 1 and 2a.

[0066] The programs of FIG. 2b can be executed by at least one of the processors (221, 225) of the electronic devices (201, 205).

[0067] Referring to FIG. 2B, the application (202) may be a program capable of providing video-related functions. For example, the application (202) may provide a video recording function. For example, the application (202) may record a video through the camera control unit (210) and / or the encoding processing unit (240). In one embodiment, the application (202) may acquire an event. For example, the application (202) may acquire a user input event. For example, the application (202) may acquire a touch event, a keyboard event, and / or a mouse event through the input module (150).

[0068] In one embodiment, the camera control unit (210) may include a source input and judgment unit (211), a time stamp generation unit (212), a coordinate calculation unit (213), and a timed meta generation unit (214).

[0069] In one embodiment, the source input and determination unit (211) can control the shooting of images through at least two cameras among a plurality of cameras (281, 283, 285). In one embodiment, the time stamp generation unit (212) can generate time stamps for frames included in images shot through at least two cameras among a plurality of cameras (281, 283, 285). In one embodiment, the coordinate calculation unit (213) can obtain information about coordinates of a user's input. In one embodiment, the timed metadata generation unit (214) can generate timed metadata for a user's input obtained through an application (146). In one embodiment, the timed metadata generation unit (214) can generate timed metadata for frames of images obtained through at least two cameras among a plurality of different cameras (281, 283, 285) having the same or corresponding time stamps.

[0070] In one embodiment, the driver (215) can control the on / off of at least one camera among the plurality of cameras (281, 283, 285) under the control of the camera control unit (210) (or the source input and determination unit (211)). In one embodiment, the driver (215) can control the image capturing of at least one camera among the plurality of cameras (281, 283, 285) under the control of the camera control unit (210) (or the source input and determination unit (211)). In one embodiment, the driver (215) can transmit images captured by at least one camera among the plurality of cameras (281, 283, 285) to the encoding processing unit (240). Here, the images transmitted from the driver (215) to the encoding processing unit (240) may be raw data.

[0071] In one embodiment, the encoding processing unit (240) may include an upscaling encoding processing unit (241), a video sampling processing unit (242), a meta track writing unit (244), a video track writing unit (245), and a format header writing unit (246). In one embodiment, the video sampling processing unit (242) may include an encoded stream tiling processing unit (243).

[0072] In one embodiment, the upscaling encoding processing unit (241) may encode source images acquired through a plurality of cameras (281, 283, 285). In one embodiment, the upscaling encoding processing unit (241) may encode some of the frames included in the source images. For example, the upscaling encoding processing unit (241) may encode frames of images that reference frames of images acquired from other cameras. In one embodiment, in the case of images streamed in real time, the upscaling encoding processing unit (241) may encode frames according to negotiation based on the capacity of a receiving device (e.g., an electronic device (205)) and / or according to a specified rule (e.g., a specified rule between a transmitting device (e.g., an electronic device (201)) and a receiving device), and store the encoded frames.

[0073] In one embodiment, the video sampling processing unit (242) may encode source images acquired through a plurality of cameras (281, 283, 285). In one embodiment, the video sampling processing unit (242) may encode frames included in each of the source images acquired through a plurality of cameras (281, 283, 285).

[0074] In one embodiment, the encoded stream tiling processing unit (243) can align the frames included in each of the encoded source images into a designated tile format. Hereinafter, with reference to FIGS. 4A and 4B, an example of an operation for the electronic device (201) to align the frames included in each of the encoded source images into a designated tile format will be described.

[0075] In one embodiment, the meta track writing unit (244) may store information about a frame to be played back among the frames included in one track of the video data within the video data. For example, the information about the frame to be played back may include a camera ID (identification) and / or a sample ID for the frame to be played back among the frames of each time stamp number. The information about the frame to be played back may also be referred to as a timed meta track.

[0076] In one embodiment, the video track writing unit (245) can generate video data that includes frames aligned in a specified tile format in one track.

[0077] In one embodiment, the format header writer (246) may include information for playing back a video contained in a track including frames aligned in a specified tile format of the video data. For example, the information for playing back the video may include a camera ID, a timestamp diff correction, an offset, a size, and / or camera info. In one embodiment, the format header writer (246) may write an additional field related to a recording mode to the header of the video data (e.g., moov). In one embodiment, the format header writer (246) may store in a timed meta track stored at a specified location of the video data (e.g., at the end of the file) for further editing of the video data.

[0078] FIG. 2c is a block diagram of programs for decoding an image, according to one embodiment.

[0079] In one embodiment, the application (206) may be a program for playing and / or editing video data. In one embodiment, the application (206) may be a program for playing and / or editing video data generated by programs for decoding images.

[0080] In one embodiment, the meta judgment unit (250) can determine whether the video data has a specified format. For example, the meta judgment unit (250) can determine whether a timed meta track exists. For example, the meta judgment unit (250) can determine whether a timed meta track exists. It can determine whether specified meta information exists in a user data area (e.g., udta). For example, the specified meta information can include information for playing a video (e.g., camera ID, time stamp difference correction, offset, size, and / or camera info).

[0081] In one embodiment, the meta judgment unit (250) may read the positions where the specified frames among a plurality of frames are recorded in the video data, depending on the playback scenario by the timed meta track and / or the specified meta information, if the timed meta track and / or the specified meta information exist. In one embodiment, the meta judgment unit (250) may determine the decoding method of the specified frames, depending on the playback scenario by the timed meta track and / or the specified meta information, if the timed meta track and / or the specified meta information exist.

[0082] In one embodiment, the meta judgment unit (250) may directly request the necessary data (or frames) from the transmitting device (e.g., electronic device (201)) according to the decoding capability and / or the playback scenario of the receiving device (e.g., electronic device (205)) when a timed meta track and / or designated meta information exist during real-time streaming, depending on the playback scenario based on the timed meta track and / or designated meta information.

[0083] In one embodiment, the I / O (input / output) processing unit (251) may include a meta extraction unit (252), a source determination unit (253), and a frame selection and reading unit (254). In one embodiment, the meta extraction unit (252) may extract a timed meta track included in video data. If the timed meta track and / or designated meta information exist, the meta extraction unit (252) may read the timed meta track and / or designated meta information, and provide the read timed meta track and / or designated meta information to the source determination unit (253). In one embodiment, the source determination unit (253) may identify a location in the video data where designated frames are recorded according to a scenario based on the timed meta track and / or designated meta information. In one embodiment, the frame selection and reading unit (254) may analyze a header of a designated format to identify a location and / or a length of a frame. In one embodiment, the frame selection and reading unit (254) may access an area to read from in a track based on the location and / or length of an identified frame.

[0084] In one embodiment, the decoding processing unit (270) may include a decoder (271) and a playback processing unit (275). In one embodiment, the playback processing unit (275) may include a coordinate calculation unit (272) and a cropping and scaling processing unit (273).

[0085] In one embodiment, the decoder (271) can decode a frame read from the frame selection and reading unit (254). In one embodiment, the playback processing unit (275) can play back the decoded frame. In one embodiment, the playback processing unit (275) can obtain location information where a user input occurred through the coordinate calculation unit (272). In one embodiment, the playback processing unit (275) can perform buffer fusion (e.g., cropping and / or scaling) on ​​the decoded frame based on the user input through the cropping and scaling processing unit (273).

[0086] FIG. 3A illustrates an example of frames included in images acquired by an electronic device through cameras.

[0087] In one embodiment, the electronic device (201) can acquire images. For example, the electronic device (201) can acquire images through cameras (281, 283, 285). For example, the electronic device (201) can acquire images through cameras (281, 283, 285) in response to identifying an image capture event. Here, the image capture event may include, but is not limited to, a situation in which a user input to a capture button is identified through a camera application.

[0088] In one embodiment, the electronic device (201) can obtain a frame (310) through the first camera (281). In one embodiment, the electronic device (201) can obtain a frame (320) through the second camera (283). In one embodiment, the electronic device (201) can obtain a frame (330) through the third camera (285). Referring to FIG. 3A, the cameras (281, 283, 285) may have different positions toward which their center points are directed. In one embodiment, when the cameras (281, 283, 285) are arranged at different positions on the same surface of the electronic device (201), the cameras (281, 283, 285) may have different positions toward which their center points are directed. Referring to FIG. 3A, the center point (311) of the frame (310) acquired through the camera (281), the center point (321) of the frame (320) acquired through the camera (283), and the center point (331) of the frame (330) acquired through the camera (285) may be different from each other. In one embodiment, the electronic device (201) may preprocess the frames (310, 320, 330) acquired by the cameras (281, 283, 285) so that the center points (311, 321, 331) of the frames (310, 320, 330) acquired by the cameras (281, 283, 285) are located at the same position. Hereinafter, the frames acquired by the cameras (281, 283, 285) may be exemplified as being preprocessed so that their respective center points are located at the same position.

[0089] In one embodiment, the electronic device (201) may display frames included in one of the images in the preview area (340) of the display (261). In one embodiment, the electronic device (201) may display frames included in one image in the preview area (340) of the display (261) while acquiring the images. For example, the electronic device (201) may display frames included in an image acquired through a main camera (e.g., a first camera (281)) in the preview area (340) of the display (261) in the order in which they were acquired.

[0090] In one embodiment, the electronic device (201) may identify at least one input (or event) while acquiring images. Here, at least one input (or event) may be related to a preview image. If the input is for changing the field of view (FOV) of the preview image, for changing the angle of view (AOV) of the preview image, or for changing (or switching) the source camera (or source image) of the preview image, the input may be recognized (or evaluated) as being related to the preview image. However, the present invention is not limited thereto.

[0091] In one embodiment, the electronic device (201) can change (or switch) the source camera (or source image) of the preview image while acquiring images. In one embodiment, the electronic device (201) can change (or switch) the source camera (or source image) of the preview image in response to at least one input (or event).

[0092] For example, in response to an input to enlarge (or zoom in) a preview image displayed through the preview area (340), the electronic device (201) may change the source camera (or source image) from the first camera (281) (or wide-angle camera) to the second camera (283) (or telephoto camera). For example, the electronic device (201) may change the source camera (or source image) of the preview image displayed in the preview area (340) or enlarge the preview image displayed in the preview area (340) based on the degree of enlargement (or degree of zoom-in) of the preview image. For example, when the magnification according to the zoom-in by the input is less than 3 times, the electronic device (201) may enlarge the preview image and display it on the display (261) without changing the source camera (or source image). For example, if the magnification according to the zoom-in by input is 3 times or more, the electronic device (201) can change the source camera (or source image).

[0093] For example, in response to an input to reduce (or zoom out) a preview image, the electronic device (201) may change the source camera (or source image) from the first camera (281) (or wide-angle camera) to the third camera (285) (or ultra-wide-angle camera). For example, the electronic device (201) may change the source camera (or source image) of the preview image displayed in the preview area (340) or reduce the preview image displayed in the preview area (340) based on the degree of reduction (or zoom-out) of the preview image. For example, when the magnification according to the zoom-out by the input exceeds 0.8 times, the electronic device (201) may reduce the preview image and display it on the display (261) without changing the source camera (or source image). For example, if the magnification according to the zoom-out by input is less than 0.8 times, the electronic device (201) can change the source camera (or source image).

[0094] For example, in response to an input for switching the source camera (or source image) of a preview image displayed in the preview area (340), the electronic device (201) can change the source camera (or source image) to a source camera (or source image) corresponding to the input.

[0095] In one embodiment, the electronic device (201) may obtain information about at least one input (or event) that occurs while acquiring images. For example, the electronic device (201) may obtain at least one of information about the time of occurrence of the at least one input (or event), whether the preview image is enlarged and / or reduced, the degree of enlargement and / or reduction, or whether the source camera is switched.

[0096] In one embodiment, the electronic device (201) can store images as a single video data. In one embodiment, the electronic device (201) can store images as a single video data that includes images in a single track. Hereinafter, an operation for the electronic device (201) to store a single video data that includes images in a single track will be described with reference to FIGS. 4A and 4B .

[0097] Figure 3b illustrates an example of frames included in images acquired by an electronic device through cameras. Figure 3c illustrates an example of an operation of arranging frames included in images acquired by an electronic device through cameras into a single track.

[0098] According to one embodiment, the electronic device (201) can acquire images through cameras (281, 283, 285).

[0099] In one embodiment, the electronic device (201) can obtain an image including frames (341, 343, 345, 347) through the first camera (281). For example, the first camera (281) can obtain an image including frames (341, 343, 345, 347) based on a shooting command from the processor (221) of the electronic device (201). For example, the first camera (281) can obtain an image including frames (341, 343, 345, 347) that are captured at a cycle (e.g., 16.6 milliseconds) according to a frame rate (e.g., 60 FPS (frames per second)). For example, the first camera (281) can acquire an image including frames (341, 343, 345, 347) captured at each of the shooting points (371, 373, 375, 379).

[0100] In one embodiment, the electronic device (201) can obtain an image including frames (351, 353, 355, 357) through the second camera (283). For example, the second camera (283) can obtain an image including frames (351, 353, 355, 357) based on a shooting command from the processor (221) of the electronic device (201). For example, the second camera (283) can obtain an image including frames (351, 353, 355, 357) that are captured at a cycle (e.g., 16.6 milliseconds) according to a frame rate (e.g., 60 FPS). For example, the second camera (283) can acquire an image including frames (351, 353, 355, 357) captured at each of the shooting points (371, 373, 375, 379).

[0101] In one embodiment, the electronic device (201) can obtain an image including frames (361, 363, 365, 367) through the third camera (285). For example, the third camera (285) can obtain an image including frames (361, 363, 365, 367) based on a shooting command from the processor (221) of the electronic device (201). For example, the third camera (285) can obtain an image including frames (361, 363, 365, 367) that are captured at a cycle (e.g., 16.6 milliseconds) according to a frame rate (e.g., 60 FPS). For example, the third camera (285) can acquire an image including frames (361, 363, 365, 367) captured at each of the shooting points (371, 373, 375, 379).

[0102] Referring to FIG. 3C, in one embodiment, the electronic device (201) may generate video data including frames (341, 343, 345, 347, 351, 353, 355, 357, 361, 363, 365, 367) in one video track (401). In one embodiment, the electronic device (201) may generate video data including one video track (401) in which frames (341, 343, 345, 347, 351, 353, 355, 357, 361, 363, 365, 367) are arranged in playback order. For example, the fact that the frames are arranged in playback order may mean that the frames (341, 343, 345, 347, 351, 353, 355, 357, 361, 363, 365, 367) are arranged in the order of the timestamps (or, playback time, presentation time). As the frames are arranged in playback order, frames included in another video may be inserted between frames included in one video. For example, other frames (351, 361) obtained between shooting points (371, 373) may be included between frames (341, 343).

[0103] FIG. 4A illustrates an example of frames acquired through multiple cameras, according to one embodiment. FIG. 4B illustrates an example of a structure in which frames of multiple images are aligned on a single track, according to one embodiment. FIG. 4C illustrates an example of a structure in which frames of multiple images are aligned on different tracks.

[0104] FIGS. 4a and 4b can be described with reference to FIGS. 1, 2a, 2b, 3a, 3b, and 3c.

[0105] According to one embodiment, the electronic device (201) can acquire images (410, 420, 430) through cameras (281, 283, 285).

[0106] The electronic device (201) can obtain an image (410) including frames (411, 413, 415, 417, 419) through the first camera (281). For example, the first camera (281) can obtain an image (410) including frames (411, 413, 415, 417, 419) based on a shooting command from the processor (221) of the electronic device (201). For example, the first camera (281) can obtain an image (410) including frames (411, 413, 415, 417, 419) that are captured at a cycle (e.g., 16.6 milliseconds) according to a frame rate (e.g., 60 FPS (frames per second)). For example, the first camera (281) can acquire an image (410) including frames (411, 413, 415, 417, 419) captured at each of the capture points (t1, t4, t7, t10, t13). For example, the time length between two adjacent capture points among the capture points (t1, t4, t7, t10, t13) can be determined according to the frame rate. For example, when the frame rate is 60, the time length between two adjacent capture points can be 16.6 milliseconds.

[0107] The electronic device (201) can obtain an image (420) including frames (421, 423, 425, 427, 429) through the second camera (283). For example, the second camera (283) can obtain an image (420) including frames (421, 423, 425, 427, 429) based on a shooting command from the processor (221) of the electronic device (201). For example, the second camera (283) can obtain an image (420) including frames (421, 423, 425, 427, 429) that are captured at a cycle (e.g., 16.6 milliseconds) according to a frame rate (e.g., 60 FPS). For example, the second camera (283) can acquire an image (420) including frames (421, 423, 425, 427, 429) captured at each of the capture points (t2, t5, t8, t11, t14). For example, the time length between two adjacent capture points among the capture points (t2, t5, t8, t11, t14) can be determined according to the frame rate. For example, when the frame rate is 60, the time length between two adjacent capture points can be 16.6 milliseconds.

[0108] The electronic device (201) can obtain an image (430) including frames (431, 433, 435, 437, 439) through the third camera (285). For example, the third camera (285) can obtain an image (430) including frames (431, 433, 435, 437, 439) based on a shooting command from the processor (221) of the electronic device (201). For example, the third camera (285) can obtain an image (430) including frames (431, 433, 435, 437, 439) that are captured at a cycle (e.g., 16.6 milliseconds) according to a frame rate (e.g., 60 FPS). For example, the third camera (285) can acquire an image (430) including frames (431, 433, 435, 437, 439) captured at each of the capture points (t3, t6, t9, t12, t15). For example, the time length between two adjacent capture points among the capture points (t3, t6, t9, t12, t15) can be determined according to the frame rate. For example, when the frame rate is 60, the time length between two adjacent capture points can be 16.6 milliseconds.

[0109] In one embodiment, each of the shooting points may represent a time stamp of a corresponding frame. For example, the shooting point (t1) of frame (411) may represent a time stamp of frame (411), the shooting point (t2) of frame (421) may represent a time stamp of frame (412), and the shooting point (t3) of frame (431) may represent a time stamp of frame (413).

[0110] Referring to FIG. 4A, the shooting points of the cameras (281, 283, 285) that operate in synchronization with the clock of the electronic device (201) may have a time difference. For example, the difference in shooting points of the sub-cameras (e.g., the second camera (283) and the third camera (285)) with respect to the main camera (e.g., the first camera (281)) may also be referred to as a time stamp difference. For example, the shooting point difference (t2-t1) between the frame (421) and the frame (411) may be the time stamp difference of the frame (421). For example, the shooting point difference (t3-t1) between the frame (431) and the frame (411) may be the time stamp difference of the frame (431). For example, the time stamp difference of the frames acquired through the main camera (e.g., the first camera (281)) may be 0.

[0111] In one embodiment, the electronic device (201) may identify at least one event (E1, E2) while acquiring images (410, 420, 430) through cameras (281, 283, 285). For example, the at least one event (E1, E2) may include identification of an input for user control of an image being captured. For example, the at least one event (E1, E2) may be related to a preview image. For example, the at least one event (E1, E2) may include identification of an input requesting a change (or switching) of a source camera (or source image) for displaying a preview image among the cameras (281, 283, 285).

[0112] For example, the electronic device (201) can identify an event (E1). For example, the electronic device (201) can identify an event (E1) for zoom-in. Depending on the magnification according to the zoom-in by the event (E1), the electronic device (201) can change the source image of the preview image displayed through the display (261). For example, if the magnification according to the zoom-in by the event (E1) is more than 0.8 times and less than 3 times, the electronic device (201) may not change the source image of the preview image displayed through the display (261) from the image (410) acquired by the first camera (281). For example, when the magnification due to zoom-in by an event (E1) is 3 times or more, the electronic device (201) can change the source image of the preview image displayed through the display (261) from the image (410) acquired by the first camera (281) to the image (420) acquired by the second camera (283).

[0113] According to one embodiment, the electronic device (201) can identify an event (E2). For example, the electronic device (201) can identify an event (E2) for zooming out. Depending on the magnification according to the zooming out by the event (E2), the electronic device (201) can change the source image of the preview image displayed through the display (261). For example, if the magnification according to the zooming out by the event (E2) is greater than 0.8 times and less than 3 times, the electronic device (201) can change the source image of the preview image displayed through the display (261) from the image (420) acquired by the second camera (283) to the image (410) acquired by the first camera (281). For example, when the magnification due to the zoom-out by the event (E2) is 0.8 times or less, the electronic device (201) may change the source image of the preview image displayed through the display (261) from the image (420) acquired by the second camera (283) to the image (430) acquired by the third camera (285). For example, when the magnification due to the zoom-out by the event (E1) is 3 times or more, the electronic device (201) may not change the source image of the preview image displayed through the display (261) from the image (420) acquired by the camera (283).

[0114] According to one embodiment, the electronic device (201) can sequentially display frames (411), (413), (425), (427), and (419) as preview images of the images on the display (261) based on at least one event (E1, E2) while the cameras (281, 283, 285) are acquiring images.

[0115] Referring to FIG. 4B, in one embodiment, the electronic device (201) may generate video data including frames (411, 413, 415, 417, 419, 421, 423, 425, 427, 429, 431, 433, 435, 437, 439) in one video track (401). In one embodiment, the electronic device (201) may generate video data including one video track (401) in which frames (411, 413, 415, 417, 419, 421, 423, 425, 427, 429, 431, 433, 435, 437, 439) are arranged in playback order. For example, the fact that the frames are arranged in playback order may mean that the frames (411, 413, 415, 417, 419, 421, 423, 425, 427, 429, 431, 433, 435, 437, 439) are arranged in the order of the timestamps (or, playback time, presentation time). As the frames are arranged in playback order, frames included in another video may be inserted between frames included in one video. For example, other frames (421, 431) acquired between the shooting times (t1, t4) may be included between frames (411, 413).

[0116] In one embodiment, the electronic device (201) can assign time stamp numbers (471, 473, 475, 477, 479) to frames (411, 413, 415, 417, 419, 421, 423, 425, 427, 429, 431, 433, 435, 437, 439). For example, the electronic device (201) can assign different time stamp numbers (471, 473, 475, 477, 479) to frames included in the same video. For example, the electronic device (201) can assign the same stamp numbers (471, 473, 475, 477, 479) to some frames among frames included in different videos.

[0117] In one embodiment, the electronic device (201) may assign time stamp numbers (471, 473, 475, 477, 479) according to time stamps of frames (411, 413, 415, 417, 419, 421, 423, 425, 427, 429, 431, 433, 435, 437, 439). In one embodiment, the electronic device (201) may assign time stamp numbers (471, 473, 475, 477, 479) according to the order of time stamps of frames (411, 413, 415, 417, 419, 421, 423, 425, 427, 429, 431, 433, 435, 437, 439).

[0118] For example, the electronic device (201) may assign the earliest time stamp number (471) to the frame (411) having the earliest time stamp among the time stamps of the frames (411, 413, 415, 417, 419, 421, 423, 425, 427, 429, 431, 433, 435, 437, 439). For example, the electronic device (201) may assign the earliest time stamp number (471) to the frame (411) having the earliest time stamp among the time stamps of the frames (411, 413, 415, 417, 419) acquired through the main camera (e.g., the first camera (281)). For example, the electronic device (201) can assign each of the time stamp numbers (473, 475, 477, 479) acquired through the main camera (e.g., the first camera (281)) to each of the frames (413, 415, 417, 419).

[0119] In one embodiment, the electronic device (201) may assign time stamp numbers of frames (421, 423, 425, 427, 429, 431, 433, 435, 437, 439) acquired through a sub-camera (e.g., a second camera (283), a third camera (285)) based on time stamp numbers (471, 473, 475, 477, 479) of frames (411, 413, 415, 417, 419) acquired through a main camera (e.g., a first camera (281)). For example, the electronic device (201) can assign the time stamp numbers of the frames (411, 413, 415, 417, 419) that have time stamps corresponding to each of the frames (421, 423, 425, 427, 429, 431, 433, 435, 437, 439) to the time stamp numbers of each of the frames (421, 423, 425, 427, 429, 431, 433, 435, 437, 439). For example, the electronic device (201) can assign the time stamp number of a first frame acquired through a main camera that has a time stamp corresponding to a second frame acquired through a sub camera to the time stamp number of the second frame. Here, the correspondence of the time stamps may mean that the time stamp difference between the frames is less than a specified time length (e.g., playback time (or duration)). If the second frame is acquired through the sub camera within a specified time length (e.g., playback time (or duration)) from the time point of acquisition of the first frame through the main camera, the time stamp of the first frame and the time stamp of the second frame may be considered to correspond to each other. For example, the time stamps of the frames (421, 431) acquired within a specified time length (e.g., playback time (or duration)) from the time point (t1) may correspond to the time stamp of the frame (411). Accordingly, the time stamp numbers of the frames (421, 431) may be identical to the time stamp number (471) of the frame (411).Similarly, the time stamp number of frames (423, 433) may be the same as the time stamp number (473) of frame (413). The time stamp number of frames (425, 435) may be the same as the time stamp number (475) of frame (415). The time stamp number of frames (427, 437) may be the same as the time stamp number (477) of frame (417). The time stamp number (479) of frames (429, 439) may be the same as the time stamp number (479) of frame (419).

[0120] In one embodiment, one video track (401) of video data may include headers (441, 443, 445, 447, 449, 451, 453, 455, 457, 459, 461, 463, 465, 467, 469) for frames (411, 413, 415, 417, 419, 421, 423, 425, 427, 429, 431, 433, 435, 437, 439). Each of the headers (441, 443, 445, 447, 449, 451, 447, 455, 457, 459, 461, 463, 465, 467, 469) may precede each of the frames (411, 413, 415, 417, 419, 421, 423, 425, 427, 429, 431, 433, 435, 437, 439). Each of the headers (441, 443, 445, 447, 449, 451, 447, 455, 457, 459, 461, 463, 465, 467, 469) can indicate the size information of the frame positioned next to it.

[0121] Referring to FIG. 4c, the electronic device (201) can generate video data in which frames (411, 413, 415, 417, 419, 421, 423, 425, 427, 429, 431, 433, 435, 437, 439) are included in a plurality of video tracks (403, 404, 405).

[0122] In one embodiment, the electronic device (201) can generate video data in which frames (411, 413, 415, 417, 419, 421, 423, 425, 427, 429, 431, 433, 435, 437, 439) acquired through each of the cameras (281, 283, 285) are included in a plurality of video tracks (403, 404, 405). For example, the electronic device (201) can generate video data in which frames (411, 413, 415, 417, 419) acquired through the camera (281) are included in a video track (403), frames (421, 423, 425, 427, 429) acquired through the camera (283) are included in a video track (404), and frames (431, 433, 435, 437, 439) acquired through the camera (285) are included in a video track (405).

[0123] For example, frames (411, 413, 415, 417, 419, 421, 423, 425, 427, 429, 431, 433, 435, 437, 439) included in each of a plurality of video tracks (403, 404, 405) can be arranged in playback order.

[0124] When images acquired through multiple cameras (481, 483, 485) as shown in FIG. 4c are stored in different video tracks (403, 404, 405), when changing video tracks during image playback, playback may be interrupted due to an operation to find a frame to play back and / or a reference frame for decoding. Specifically, when images acquired through multiple cameras (481, 483, 485) as shown in FIG. 4c are stored in different video tracks (403, 404, 405), the different video tracks (403, 404, 405) may be stored scattered within the video data at positions of arbitrary file offsets without physical continuity. In addition, when images acquired through multiple cameras (481, 483, 485) as shown in FIG. 4c are stored in different video tracks (403, 404, 405), audio data may be interleaved and stored within the video data to synchronize the images and the audio data corresponding to each of them. Therefore, when images acquired through multiple cameras (481, 483, 485) as shown in FIG. 4c are stored in different video tracks (403, 404, 405), in order to access the multiple images, a file seek and / or read operation must be performed to move the position of the current file pointer to another position, and then move it back to the current position, so that time may be required for file input and / or output. In addition, when images acquired through multiple cameras (481, 483, 485) as in FIG. 4c are stored in different video tracks (403, 404, 405), when changing the video track during image playback, it may take time to search for a frame to be played back, to search for an IDR frame and / or a P frame or a B frame for decoding the frame to be played back, and to decode the frame.In addition, when images acquired through multiple cameras (481, 483, 485) as shown in FIG. 4c are stored in different video tracks (403, 404, 405), when the video track is changed during video playback, more time may be required for post-processing (e.g., cropping) of the frame to be played back. In addition, when images acquired through multiple cameras (481, 483, 485) as shown in FIG. 4c are stored in different video tracks (403, 404, 405), when the video track is changed during video playback, more time may be required for frame playback as fusion (or synthesis) of frames included in two or more tracks is required. Accordingly, when images acquired through multiple cameras (481, 483, 485) as in FIG. 4c are stored in different video tracks (403, 404, 405), when changing the video track during video playback, it may not be easy to process the corresponding operation within a time (e.g., 16.6 ms) according to a set frame rate (e.g., 60 Hz).

[0125] In addition, when images acquired through multiple cameras (481, 483, 485) having different angles of view are stored in different video tracks (403, 404, 405), the different video tracks (403, 404, 405) may be stored in non-consecutive positions in the video data. Accordingly, when images acquired through multiple cameras (481, 483, 485) having different angles of view are required simultaneously, the electronic device must access non-consecutive positions when inputting and / or outputting files, which may consume more time for data reading.

[0126] On the other hand, when images acquired through multiple cameras (481, 483, 485) are organized into one track as shown in FIG. 4b, since images acquired through multiple cameras (481, 483, 485) are stored in consecutive locations in the video data, the time required for an operation to find a frame to be played back and / or a reference frame for decoding can be reduced. In addition, when images acquired through multiple cameras (481, 483, 485) are organized into one track as shown in FIG. 4b, by processing the reads in units of samples, the time required for an operation to find a frame to be played back and / or a reference frame for decoding can be reduced. For example, as in FIG. 4b, when images acquired through multiple cameras (481, 483, 485) are organized into one track, the file input and / or output delay according to the range-request operation of the file pointer (or track switching and seek) can be reduced according to the change of the image required when leading the bitstream of the desired playback position.

[0127] In the case where images acquired through multiple cameras (481, 483, 485) as shown in Fig. 4c are stored in different video tracks (403, 404, 405), as the video data is streamed, the downloaded data (or cached data) may not download a video track of a video different from the currently playing video. In this case, a range request may be generated to play a video different from the currently playing video. Furthermore, in this case, movement of the file pointer may be required to play a video different from the currently playing video and to move from the other video back to the currently playing video. Accordingly, multiple range requests may be generated, and buffering may occur depending on the bandwidth of the network for streaming the video data.

[0128] On the other hand, as in Fig. 4b, when images acquired through multiple cameras (481, 483, 485) are organized into one track, when video data is streamed, it is downloaded (or cached) in units of samples, thereby reducing the occurrence of range requests to play a different image from the currently playing image.

[0129] When images acquired through multiple cameras (481, 483, 485) are stored in different video tracks (403, 404, 405) as in Fig. 4c, data may increase excessively as frames that are not used at the time of playback are maintained in the video data. On the other hand, when images acquired through multiple cameras (481, 483, 485) are organized into one track as in Fig. 4b, the size of the video data may be reduced by excluding frames for images that are not required on a sample basis.

[0130] Hereinafter, the structure of a video track (401) generated by an electronic device (201) will be described with reference to FIGS. 5a and 5b.

[0131] FIG. 5A illustrates an example of a structure of video data (501) according to one embodiment. FIG. 5B illustrates an example of an operation for specifying a position of a frame according to the structure of video data according to one embodiment.

[0132] FIGS. 5A and 5B can be described with reference to FIGS. 1, 2A, 2B, 3A, 3B, 3C, 4A, 4B, and 4C.

[0133] Referring to FIG. 5A, video data (501) may include a moov box (510), an mdat box (550), and a timed meta track box (590).

[0134] In one embodiment, the moov box (510) may store metadata of the videos (410, 420, 430). For example, the moov box (510) may store metadata (521, 525) of the videos (410, 420, 430) through the stbl box (511) and / or the udta box (515). In one embodiment, the stbl box (511) may be referred to as a metadata area. In one embodiment, the udta box (515) may be referred to as a user data area.

[0135] In one embodiment, the electronic device (201) can store metadata (521) for an image (410) acquired through a main camera (e.g., a first camera (281)) through an stbl box (511).

[0136] In one embodiment, the metadata (521) stored in the stbl box (511) may be data for backward compatibility. For example, the metadata (521) stored in the stbl box (511) may be used to reproduce an image (410) acquired through a main camera (e.g., the first camera (281)) among the images (410, 420, 430).

[0137] In one embodiment, the sample ID (sample identification) field of the metadata (521) may indicate the ID of the frame (or sample) indicated by the metadata (521). Depending on the embodiment, the ID of the frame (or sample) may indicate a timestamp number. The number of samples (# of samples) field of the metadata (521) may indicate the number of frames (or samples) indicated by the metadata (521).

[0138] In one embodiment, the offset field of the metadata (521) may indicate information about the location (or address within the video data (501)) of the frame (or sample) indicated by the metadata (521). The size field of the metadata (521) may indicate information about the size of the frame (or sample) indicated by the metadata (521).

[0139] For example, the electronic device (201) can generate metadata (521) for frames (411, 413, 415) stored in the stbl box (511) as shown in Table 1 below. In Table 1, the metadata (521) is illustrated as including metadata for three frames (411, 413, 415), but this is merely an example. The metadata (521) can include metadata for all of the frames (411, 413, 415, 417, 419) included in the video (410).

[0140] sample ID123timestamp diff33 milliseconds34 milliseconds32 milliseconds# of samples111offset100022803230size480250370

[0141] Referring to Table 1, an electronic device (205) that plays back video data (501) can play back data (or frame (411)) from a start position 1000 indicated by an offset field to a position 1479 according to a length 480 indicated by a size field as a first frame for a time length of 33 milliseconds indicated by a time stamp diff field, by referring to metadata (521). In addition, the electronic device (205) can play back data (or frame (413)) from a start position 2280 indicated by an offset field to a position 2529 according to a length 250 indicated by a size field as a second frame for a time length of 34 milliseconds indicated by a time stamp diff field. Finally, the electronic device (205) can play data (or frame (415)) from the start position 3230 indicated by the offset field to the position 3599 according to the length 370 indicated by the size field as the third frame for a time length of 32 milliseconds indicated by the time stamp diff field.

[0142] In one embodiment, the electronic device (201) may store metadata (525) for images (420, 430) acquired through sub-cameras (e.g., the second camera (283) and the third camera (285)) via the udta box (515). For example, the electronic device (201) may store metadata (525) for images (410) acquired through the main camera (e.g., the first camera (281)) and images (420, 430) acquired through sub-cameras (e.g., the second camera (283) and the third camera (285)) via the udta box (515).

[0143] In one embodiment, metadata (525) stored in the udta box (515) may be used to selectively play back images (410, 420, 430). Here, selectively playing back images (410, 420, 430) may mean that the images being played back may be changed by selection during the playback time of the images (410, 420, 430).

[0144] According to one embodiment, the metadata (525) stored in the udta box (515) can be used to play back the images (410, 420, 430).

[0145] In one embodiment, the version field of the metadata (525) may indicate the version of the metadata (525). The sample ID (sample identification) field of the metadata (525) may indicate the ID of the frame (or sample) indicated by the metadata (525). Depending on the embodiment, the ID of the frame (or sample) may indicate a timestamp number. The timestamp diff field of the metadata (525) may indicate the length of the playback time of the frame (or sample) indicated by the metadata (525). The number of samples (# of samples) field of the metadata (525) may indicate the number of frames (or samples) indicated by the metadata (525). The camera ID field of the metadata (525) may indicate the ID of the camera that acquired the frame (or sample) indicated by the metadata (525).

[0146] In one embodiment, the timestamp diff correction field of the metadata (525) may indicate information for correcting the playback time point of the frame (or sample) indicated by the metadata (525). For example, the timestamp diff correction field may indicate information (or CTTS (composition time to sample)) for correcting the time difference between the decoding time point and the playback time point (or composition time) of the frame (or sample) indicated by the metadata (525). For example, the timestamp diff correction field may further include information (or CTTS) for correcting the time difference between the decoding time point and the playback time point when the frame (or sample) indicated by the metadata (525) is a specified type of frame (e.g., a B (bi-directional prediction) frame).

[0147] In one embodiment, the offset field of the metadata (525) may indicate information about the location (or address within the video data (501)) of the frame (or sample) indicated by the metadata (525). The size field of the metadata (525) may indicate information about the size of the frame (or sample) indicated by the metadata (525). The camera info field of the metadata (525) may indicate information about the camera that acquired the frame (or sample) indicated by the metadata (525) (e.g., angle of view, field of view, arrangement information, brightness, color temperature, f-number).

[0148] In one embodiment, the metadata (525) stored in the udta box (515) may include multiple fields compared to the metadata (521) stored in the stbl box (511). For example, the metadata (525) may include multiple field sets (531, 535, 539) including a camera ID field, a timestamp diff correction field, an offset field, a size field, and a camera info field. For example, the metadata (525) may include field sets (531, 535, 539) including a camera ID field, a timestamp diff correction field, an offset field, a size field, and a camera info field, as many as the number of frames having the same sample ID (or timestamp number). For example, the metadata (525) may include three field sets (531, 535, 539) when images (410, 420, 430) are acquired.

[0149] For example, the electronic device (201) can generate metadata (525) for frames (411, 413, 415, 421, 423, 425, 431, 433, 435) stored in the udta box (515) as shown in Table 2 below. In Table 2, the metadata (521) is illustrated as including metadata for nine frames (411, 413, 415, 421, 423, 425, 431, 433, 435), but this is only an example. The metadata (521) can include metadata for all frames included in the images (410, 420, 430).

[0150] version222sample ID123timestamp diff33 milliseconds34 milliseconds32 milliseconds# of samples333Field set(531)camera ID000timestamp diff correction000offset100022803230size480250370camera infoFIRST INFOFIRST INFOFIRST INFOField set(535)camera ID111timestamp diff correction211offset148025303600size500450470camera infoSECOND INFOSECOND INFOSECOND INFOField set(539)camera ID222timestamp diff correction343offset198029804070size300250320camera infoTHIRD INFOTHIRD INFOTHIRD INFO

[0151] Referring to Table 2, an electronic device (205) that plays video data (501) can play frames based on data at a location indicated by metadata (525).

[0152] In one embodiment, the electronic device (205) can identify the start position of a frame through the offset field. In one embodiment, the electronic device (205) can identify data of the length indicated by the size field from the start position of the frame identified through the offset field as one frame. For example, referring to FIG. 5B, the electronic device (205) can identify the start position of a frame with a sample ID of 1 and a camera ID of 0 as 1000 through the value 1000 indicated by the offset field of the field set (531). For example, the electronic device (205) can identify a frame with a sample ID of 1 and a camera ID of 0 as having a length of 480 from the start position 1000 through the value 480 indicated by the side field of the field set (531). For example, the electronic device (205) can identify the start position of a frame with a sample ID of 1 and a camera ID of 1 as 1480 through the value 1480 indicated by the offset field of the field set (535). For example, the electronic device (205) can identify the frame with a sample ID of 1 and a camera ID of 1 as having a length of 500 from the start position 1480 through the value 500 indicated by the side field of the field set (535). For example, the electronic device (205) can identify the start position of a frame with a sample ID of 1 and a camera ID of 2 as 1980 through the value 1980 indicated by the offset field of the field set (539). For example, the electronic device (205) can identify, through the value 300 indicated by the side field of the field set (539), that a frame with a sample ID of 1 and a camera ID of 2 has a length of 300 from the start position 1980.

[0153] For example, referring to FIG. 5B, the electronic device (205) may display a first frame on the display (265) for a time length of 33 milliseconds indicated by the time stamp diff field based on at least one frame among data from a start position 1000 indicated by the offset field to a position 1479 according to a length 480 indicated by the size field (or frame (411)), data from a start position 1480 indicated by the offset field to a position 1979 according to a length 500 indicated by the size field (or frame (421)), or data from a start position 1980 indicated by the offset field to a position 2279 according to a length 300 indicated by the size field (or frame (431)).

[0154] For example, the electronic device (205) may display a second frame on the display (265) for a time length of 34 milliseconds indicated by the time stamp diff field based on at least one frame among data from the start 2280 indicated by the position offset field to the position 2529 according to the length 250 indicated by the size field (or frame (413)), data from the start 2530 indicated by the offset field to the position 2979 according to the length 450 indicated by the size field (or frame (423)), or data from the start 2980 indicated by the offset field to the position 3229 according to the length 250 indicated by the size field (or frame (433)).

[0155] For example, the electronic device (205) may display a third frame on the display (265) for a time length of 32 milliseconds indicated by the time stamp diff field based on at least one frame among data (or frame (415)) from a start position 3230 indicated by the offset field to a position 3599 according to a length 370 indicated by the size field, data (or frame (425)) from a start position 3560 indicated by the offset field to a position 4069 according to a length 470 indicated by the size field, or data (or frame (435)) from a start position 4070 indicated by the offset field to a position 4389 according to a length 320 indicated by the size field.

[0156] In one embodiment, the electronic device (205) can adjust the playback time of frames based on the value indicated by the time stamp diff correction field. For example, the electronic device (205) can advance the playback time of frames by the value indicated by the time stamp diff correction field. For example, when playing frame (421), the electronic device (205) can advance the playback time of frame (421) by 2 milliseconds and display it on the display (265) for 33 milliseconds. For example, when playing frame (431), the electronic device (205) can advance the playback time of frame (431) by 3 milliseconds and display it on the display (265) for 33 milliseconds.

[0157] In one embodiment, the mdat box (550) can store images (410, 420, 430) and audios corresponding to the images (410, 420, 430). For example, the mdat box (550) can store images (410, 420, 430) through a video track box (551). In one embodiment, the electronic device (201) can store a video track (401) having the structure of FIG. 4B in one video track box (551) of the mdat box (550). For example, the mdat box (550) can store audios corresponding to the images (410, 420, 430) through an audio track box (555).

[0158] In one embodiment, a timed meta track box (590) may indicate information about a frame to be played. In one embodiment, the timed meta track box (590) may include information about a frame to be played among the frames having each of the time stamp numbers (e.g., a camera ID and / or a sample ID). For example, the timed meta track box (590) may indicate a frame to be played at the first time stamp among the frames (411, 421, 431) having the first time stamp number.

[0159] In one embodiment, the timed meta track box (590) may include information (e.g., camera ID, and / or sample ID) about frames to be played back depending on at least one event (E1, E2). For example, the timed meta track box (590) may indicate frame (411), frame (413), frame (425), frame (427), and frame (419) as frames to be played back when event (E1) is a zoom-in event and event (E2) is a zoom-out event.

[0160] In one embodiment, the timed meta track box (590) can store information about at least one event (E1, E2). For example, the timed meta track box (590) can include information about at least one event (E1, E2) that occurred during the shooting of the images (410, 420, 430). For example, the timed meta track box (590) can include information about an input requesting a change (or switching) of a source camera (or source image) for displaying a preview image among the cameras (281, 283, 285). Here, the change (or switching) of the source camera (or source image) can include zooming in, zooming out, and / or switching the camera.

[0161] In one embodiment, the timed meta track box (590) may include information about whether a camera has changed according to at least one event (E1, E2), and when the camera has changed.

[0162] In one embodiment, the timed meta track box (590) may indicate two or more frames among the frames included in one timestamp number as frames to be played back. For example, if the event (E1) is a zoom-in event, the timed meta track box (590) may indicate frames (415, 425) as frames to be played back simultaneously for a smooth transition from the first camera (281) to the second camera (283). In this case, the electronic device (205) may play back one frame based on the frames (415, 425) at the third timestamp. Here, the fact that one frame is based on the frames (415, 425) may mean that one frame has a median value of the frames (415, 425). The fact that one frame is based on the frames (415, 425) may mean that one frame is a weighted average from the frames (415, 425). For example, if the event (E2) is a zoom-out event, the timed meta track box (590) may indicate that frames (419, 429) are to be played back simultaneously for a smooth transition from the second camera (283) to the first camera (281). In this case, the electronic device (205) may play back one frame based on the frames (419, 429) at the fifth time stamp. Here, the fact that one frame is based on the frames (419, 429) may mean a frame having a median value of the frames (419, 429) (e.g., a frame in which the frames (419, 429) are weighted averaged).

[0163] According to an embodiment, the electronic device (201) may generate metadata (525) for some frames (411, 413, 415, 421, 425, 431) stored in the udta box (515) as shown in Table 3 below. Some frames (411, 413, 415, 421, 425, 431) of Table 3 may be some of the frames (411, 413, 415, 421, 423, 425, 431, 433, 435) included in the captured images. For example, some frames (411, 413, 415, 421, 425, 431) in Table 3 may be frames that are not excluded (or removed) from a track through post-processing among the frames (411, 413, 415, 421, 423, 425, 431, 433, 435) included in the captured images. In one embodiment, the electronic device (201) may remove (or exclude) frames that are not related to at least one event from a track. In one embodiment, the electronic device (201) may not remove (or exclude) frames related to at least one event from a track. In one embodiment, since some of the frames (411, 413, 415, 421, 423, 425, 431, 433, 435) included in the captured images are included in one track, the number of frames included in each sample may be different.

[0164] version222sample ID123timestamp diff33 milliseconds34 milliseconds32 milliseconds# of samples312field set(531)camera ID000timestamp diff correction000offset100022802530size480250370camera infoFIRST INFOFIRST INFOFIRST INFOfield set(535)camera ID11timestamp diff correction21offset14802900size500470camera infoSECOND INFOSECOND INFOfield set(539)camera ID2timestamp diff correction3offset1980size300camera infoTHIRD INFO

[0165] For example, referring to Table 3, the electronic device (205) may display the first frame on the display (265) for a time length of 33 milliseconds indicated by the time stamp diff field based on at least one frame among data from a start position 1000 indicated by the offset field to a position 1479 according to a length 480 indicated by the size field (or frame (411)), data from a start position 1480 indicated by the offset field to a position 1979 according to a length 500 indicated by the size field (or frame (421)), or data from a start position 1980 indicated by the offset field to a position 2279 according to a length 300 indicated by the size field (or frame (431)).

[0166] For example, referring to Table 3, the electronic device (205) may display the second frame on the display (265) for a time length of 34 milliseconds indicated by the time stamp diff field, based on data (or frame (413)) from the start 2280 indicated by the position offset field to the position 2529 according to the length 250 indicated by the size field.

[0167] For example, referring to Table 3, the electronic device (205) may display a third frame on the display (265) for a time length of 32 milliseconds indicated by the time stamp diff field based on at least one frame of data from a start position 2530 indicated by the offset field to a position 2899 according to a length 370 indicated by the size field (or frame (415)), or data from a start position 2900 indicated by the offset field to a position 3369 according to a length 470 indicated by the size field (or frame (425)).

[0168] Figures 6a, 6b, and 6c illustrate an example of a decoding order of frames according to an event, according to one embodiment.

[0169] FIGS. 6a, 6b, and 6c can be described with reference to FIGS. 1, 2a, 2b, 3a, 3b, 3c, 4a, 4b, 4c, 5a, and 5b.

[0170] In one embodiment, the electronic device (205) can play back images (410, 420, 430) based on video data (video data (501) of FIG. 5A). In one embodiment, the electronic device (205) can play back images (410, 420, 430) based on a video track (401) included in the video data (501). For example, the electronic device (205) can determine frames indicated by a timed meta track box (timed meta track box (590) of FIG. 5A) among frames of images (410, 420, 430) included in the video track (401). For example, the electronic device (205) can display the frames indicated by the timed meta track box (590) in the order of the time stamp number through the display (265). For example, if the timed meta track box (590) indicates frames (411, 413, 425, 427, 419) as frames to be played, the electronic device (205) can display the frames (411, 413, 425, 427, 419) in the order of the time stamp number through the display (265).

[0171] In one embodiment, the electronic device (205) can decode the frames (411, 413, 425, 427, 419) to play the frames (411, 413, 425, 427, 419). In one embodiment, the electronic device (205) can further decode frames other than the frame to be played, depending on the type of the frames (411, 413, 425, 427, 419). For example, if the frames (411, 413, 425, 427, 419) refer to a frame other than the frame to be played, the electronic device (205) can decode the referenced frame.

[0172] Referring to FIGS. 6A and 6B , frames (411, 421, 431, 419, 429, 439) may be instantaneous decoder refresh (IDR) frames. In one embodiment, frames (411, 421, 431, 419, 429, 439) that are IDR frames may be decoded without reference to other frames. In one embodiment, frames between an IDR frame and a P (prediction) frame preceding the next IDR frame may constitute one group of pictures (GOP). For example, frames (411, 413, 415, 417) may constitute one GOP.

[0173] In one embodiment, frames (413, 415, 417, 423, 425, 427, 433, 435, 437) can be P (prediction) frames. In one embodiment, a P frame can be decoded with reference to a preceding P frame and / or an IDR frame. In one embodiment, a P frame can be decoded with reference to a preceding P frame and / or an IDR frame within its GOP. For example, frame (413) can be decoded with reference to frame (411). For example, frame (415) can be decoded with reference to frame (411) and frame (413). For example, frame (415) can be decoded with reference to frame (413) that is decoded with reference to frame (411).

[0174] Referring to FIG. 6A, the electronic device (205) can further decode unplayed frames (421, 423) to play back frames (411, 413, 425, 427, 419). For example, the electronic device (205) can decode frame (411) and play back the decoded frame (411). For example, the electronic device (205) can refer to frame (411), decode frame (413), and play back the decoded frame (413). For example, the electronic device (205) can decode frame (421) and decode frame (423) to play back frame (425). For example, the electronic device (205) can decode the frame (425) with reference to the decoded frame (423) and play back the decoded frame (425). For example, the electronic device (205) can decode the frame (427) with reference to the decoded frame (425) and play back the decoded frame (427). For example, the electronic device (205) can decode the frame (419) and play back the decoded frame (419).

[0175] In FIG. 6A, the electronic device (205) can decode frames (411, 413, 425, 427, 419) in the order of frame (411), frame (413), frame (421), frame (423), frame (425), frame (427), and frame (419). In one embodiment, the electronic device (201) can play the frames (411, 413, 425, 427, 419) at a playback time according to the values ​​of the time stamp diff field and the time stamp diff correction field of each of the frames (411, 413, 425, 427, 419).

[0176] In one embodiment, the electronic device (205) can decode a frame to be played back by referring to a frame other than the frames within the same GOP. For example, the electronic device (205) can refer to a second frame included in one of the images (410, 420, 430) when decoding a first frame included in another image. Here, the first frame can have a time stamp number that is next to the second frame. For example, the electronic device (205) can decode frames (421, 423, 425, 427, 429) included in the image (420) by referring to frames (411, 413, 415, 417, 419) included in the image (410). For example, the electronic device (205) can decode frames (431, 433, 435, 437, 439) included in the image (430) by referring to frames (411, 413, 415, 417, 419) included in the image (410). However, the present invention is not limited thereto. For example, the electronic device (205) can decode frame (425) by referring to frame (413).

[0177] In one embodiment, the electronic device (205) may adjust the resolution of a second frame included in one of the images (410, 420, 430) for decoding a first frame included in the other image. In one embodiment, the electronic device (205) may upscale or downscale the resolution of an area corresponding to an area of ​​the first frame among areas of the second frame for decoding the first frame. For example, in the case of frames such as FIG. 3A, if the first frame is frame (310) and the second frame is frame (330), the electronic device (205) may upscale an area corresponding to frame (330) in frame (310) and decode frame (330) with reference to the area of ​​the upscaled frame (310). For example, in the case of frames such as FIG. 3A, if the first frame is frame (310) and the second frame is frame (320), the electronic device (205) can downscale an area corresponding to frame (320) in frame (310) and decode frame (320) by referring to the area of ​​the downscaled frame (310).

[0178] Referring to FIG. 6B, the electronic device (205) may decode the frame (421) to play the frame (425) and decode the frame (423) with reference to the decoded frame (421). In one embodiment, the electronic device (205) may adjust the resolution of the frame (413) and decode the frame (425) with reference to the frame (413) having the adjusted resolution. In the case of FIG. 6B, compared to FIG. 6A, decoding of the frames (421, 423) may not be required to play the frame (425).

[0179] In FIG. 6B, the electronic device (205) can decode frames (411, 413, 425, 427, 419) in the order of frame (411), frame (413), frame (425), frame (427), and frame (419). In one embodiment, the electronic device (205) can adjust the resolution of frame (413), which is a preceding frame, for decoding frame (425). In one embodiment, the electronic device (201) can play frames (411, 413, 425, 427, 419) at a playback time according to the values ​​of the time stamp diff field and the time stamp diff correction field of each of the frames (411, 413, 425, 427, 419).

[0180] Referring to FIG. 6C, frames (411, 421, 431, 419, 429, 439) may be IDR frames. Frames (413, 415, 417, 427, 433, 435, 437) may be P frames. Frames (423, 425) may be B (bi-directional prediction) frames. In one embodiment, a B frame may be decoded with reference to a subsequent P frame and / or a preceding IDR frame. In one embodiment, a B frame may be decoded with reference to a subsequent B or P frame and / or a preceding IDR frame within its GOP. For example, frame (425), which is a B frame, may be decoded with reference to frame (421) and frame (427). For example, frame (425) can be decoded with reference to frame (427), which is decoded with reference to frame (421). For example, frame (423), which is a B frame, can be decoded with reference to frame (421) and frame (425).

[0181] Referring to FIG. 6C, the electronic device (205) may decode the frame (421) and decode the frame (427) with reference to the decoded frame (421) in order to reproduce the frame (425). In the case of FIG. 6C, compared to FIGS. 6A and 6B, decoding of the frame (427) may be required before decoding the frame (425) in order to reproduce the frame (425).

[0182] In FIG. 6C, the electronic device (205) can decode frames (411, 413, 425, 427, 419) in the order of frame (411), frame (413), frame (421), frame (427), frame (425), and frame (419). In one embodiment, the electronic device (205) can first decode frame (427) having a later time stamp than frame (425). The time stamp diff correction field for frame (425) can further include information (or CTTS) for correcting the time difference between the decoding time and the playback time (or synthesis time). In one embodiment, the electronic device (201) can play back frames (411, 413, 425, 427, 419) at a playback time according to the values ​​of the time stamp diff field and the time stamp diff correction field of each of the frames (411, 413, 425, 427, 419).

[0183] FIG. 6d illustrates an example of an operation of decoding with reference to a frame of another image, according to one embodiment.

[0184] FIG. 6d can be explained with reference to FIG. 1, FIG. 2a, FIG. 2b, FIG. 3a, FIG. 3b, FIG. 3c, FIG. 4a, FIG. 4b, FIG. 4c, FIG. 5a, FIG. 5b, and FIG. 6b.

[0185] In one embodiment, the electronic device (205) may refer to a frame (610) included in one of the images (410, 420, 430) when decoding a frame (625) included in another image. For example, the frame (610) may be frame (413) of FIG. 6B . For example, the frame (625) may be frame (425) of FIG. 6B . For example, the frame (625) may be a frame acquired by a telephoto camera, and the frame (610) may be a frame acquired by a wide-angle camera. Here, the frame (625) may have a time stamp number that is next to the frame (610).

[0186] In one embodiment, the electronic device (205) may adjust the resolution of a frame (610) included in another image in order to decode a frame (625) included in a specific image. In one embodiment, the electronic device (205) may upscale or downscale the resolution of an area (615) corresponding to an area of ​​the frame (625) among areas of the frame (610) in order to decode the frame (625). In one embodiment, the electronic device (205) may adjust the resolution of an area (615) corresponding to the frame (625) among the frames (610) in order to play the frame (625) after playing the frame (610), and may decode the frame (625) by referring to a frame (620) having the adjusted resolution. For example, the electronic device (205) can upscale an area (615) corresponding to a frame (625) in a frame (610), and decode the frame (625) by referring to the frame (620) upscaled from the area (615).

[0187] FIG. 7A illustrates an example of a frame of video data played by an electronic device according to an embodiment. FIG. 7B illustrates an example of a frame of video data played by an electronic device according to an embodiment. FIG. 7C illustrates an example of a frame of video data played by an electronic device according to an embodiment. FIG. 7D illustrates an example of a frame of video data played by an electronic device according to an embodiment.

[0188] FIGS. 7a, 7b, 7c and 7d can be described with reference to FIGS. 1, 2a, 2b, 3a, 3b, 3c, 4a, 4b, 4c, 5a and 5b.

[0189] Referring to FIGS. 7A and 7C, the electronic device (205) can play back video data (501). In one embodiment, the electronic device (205) can play back images (410, 420, 430) based on the video track (401) included in the video data (501). For example, the electronic device (205) can display, through the display (265), frames indicated by the timed meta track box (590) among the frames of the images (410, 420, 430) included in the video track (401) in the order of the time stamp number. In one embodiment, the electronic device (205) can display, in the playback area (740) of the display (265), the frame (710) indicated by the timed meta track box (590) among the frames (710, 720, 730) having the same time stamp number.

[0190] The electronic device (205) may identify at least one event while playing back video data (501). For example, the at least one event may include identification of an input for user control of the video being played back. For example, the at least one event may be related to the playing image. For example, the at least one event may include identification of an input requesting a change (or switching) of a source video (or source camera) for displaying the playing image among the videos (410, 420, 430).

[0191] Referring to FIG. 7A, the electronic device (205) can identify an event (701) for zoom-in. For example, the electronic device (205) can identify an event (701) for zoom-in based on touch inputs (741, 743) on the display (265) moving away from each other. However, the present invention is not limited thereto. The electronic device (205) can identify an event (701) for zoom-in based on other inputs requesting zoom-in (e.g., a touch input for a zoom-in button, a voice input requesting zoom-in).

[0192] In one embodiment, the electronic device (205) can change the source image of the playback image displayed through the display (265) according to the magnification according to the zoom-in by the event (701).

[0193] For example, if the magnification according to the zoom-in by the event (701) is within the reference magnification range (e.g., more than 0.8 times and less than 3 times), the electronic device (205) may not change the source image of the playback image displayed through the display (265) from the image (410) acquired by the first camera (281). Referring to FIG. 7B, if the magnification according to the zoom-in by the event (701) is within the reference magnification range (e.g., more than 0.8 times and less than 3 times), the electronic device (205) may display an image (751) in which a portion of the frame (710) is enlarged through the display (265).

[0194] For example, if the magnification due to the zoom-in by the event (701) exceeds the reference magnification range (e.g., 3 times or more), the electronic device (205) can change the source image of the playback image displayed through the display (265) from the image (410) acquired by the first camera (281) to the image (420) acquired by the second camera (283). Referring to FIG. 7B, if the magnification due to the zoom-in by the event (701) exceeds the reference magnification range (e.g., 3 times or more), the electronic device (205) can display an image (752) based on the frame (730) rather than the frame (710) through the display (265). In one embodiment, the electronic device (205) may select a frame of an image (410) acquired by the first camera (281) and a frame of an image (420) acquired by the second camera (283) as frames to be played back simultaneously to achieve a smooth transition from an image (410) acquired by the first camera (281) to an image (420) acquired by the second camera (283). In one embodiment, the electronic device (205) may play back one frame based on the frame of the image (410) acquired by the first camera (281) and the frame of the image (420) acquired by the second camera (283). Here, the fact that one frame is based on the frame of the image (410) acquired by the first camera (281) and the frame of the image (420) acquired by the second camera (283) may mean that one frame has an intermediate value between the frames. The fact that one frame is based on a frame of an image (410) acquired by a first camera (281) and a frame of an image (420) acquired by a second camera (283) may mean that one frame is a weighted average frame of the frames.

[0195] In FIGS. 7A and 7B , the event (701) for zooming in is illustrated as being generated by a user input, but this is merely an example. Depending on the embodiment, the event (701) for zooming in may be an event recorded in the timed meta track box (590).

[0196] Referring to FIG. 7C, the electronic device (205) can identify an event (707) for zooming out. For example, the electronic device (205) can identify an event (707) for zooming out based on touch inputs (745, 747) on the display (265) coming closer to each other. However, the present invention is not limited thereto. The electronic device (205) can identify an event (707) for zooming out based on other inputs requesting zooming out (e.g., a touch input for a zooming out button, a voice input requesting zooming out).

[0197] In one embodiment, the electronic device (205) can change the source image of the playback image displayed through the display (265) according to the magnification according to the zoom-out by the event (703).

[0198] For example, if the magnification due to the zoom-out by the event (703) is within the reference magnification range (e.g., more than 0.8 times and less than 3 times), the electronic device (205) may not change the source image of the playback image displayed through the display (265) from the image (410) acquired by the first camera (281). Referring to FIG. 7d, if the magnification due to the zoom-out by the event (703) is within the reference magnification range (e.g., more than 0.8 times and less than 3 times), the electronic device (205) may display the image (761) in which the frame (710) is reduced through the display (265).

[0199] For example, when the magnification due to the zoom-out by the event (703) is outside the reference magnification range (e.g., 0.8 times or less), the electronic device (205) can change the source image of the playback image displayed through the display (265) from the image (410) acquired by the first camera (281) to the image (430) acquired by the third camera (285). Referring to FIG. 7D, when the magnification due to the zoom-out by the event (703) is outside the reference magnification range (e.g., 0.8 times or less), the electronic device (205) can display an image (762) based on the frame (720) rather than the frame (710) through the display (265).

[0200] In FIGS. 7C and 7D , the event (707) for zooming out is exemplified as being generated by a user input, but this is merely an example. Depending on the embodiment, the event (707) for zooming out may be an event recorded in the timed meta track box (590).

[0201] FIG. 8A illustrates an example of frames acquired through multiple cameras according to one embodiment. FIG. 8B illustrates an example of the order of frames according to one embodiment.

[0202] FIG. 8a and FIG. 8b can be explained with reference to FIG. 1, FIG. 2a, FIG. 3a, FIG. 3b, FIG. 3c, FIG. 4a, FIG. 4b, and FIG. 5a.

[0203] Compared to FIG. 4A, FIG. 8A illustrates that among the cameras (281, 283, 285), some cameras (e.g., the first camera (281)) can acquire images including frames (811, 813, 815, 817, 819) corresponding to the length of the time stamps (i.e., total playback time), and other cameras (e.g., the second camera (283), the third camera (285)) can acquire images including frames (827, 829, 833, 835, 837) corresponding to a length shorter than the length of the time stamps (i.e., total playback time). For example, the second camera (283) can acquire images including frames (827, 829) corresponding to a length shorter than the length of the time stamps (i.e., total playback time) of the images captured by the first camera (281). For example, the third camera (285) may acquire an image including frames (833, 835, 837) corresponding to a length shorter than the length of the time stamps (i.e., total playback time) of the image captured by the first camera (281). However, this is not limited thereto. The images acquired through each of the cameras (281, 283, 285) may all be shorter than the length of the time stamps (i.e., total playback time). Here, the total playback time may be the sum total of the time stamp diffs for the frames included in the video track (541) of the video data (501). For example, the total playback time may be the sum total of the time stamp diffs for each sample ID in the video track (541) of the video data (501). For example, if the video data (501) includes a 10-second video acquired at 30 FPS, the video track (541) may include 600 sample IDs for frames, and the number of frames may be 600 or more. For example, one sample ID may include one or more frames acquired through at least one camera among a plurality of cameras (281, 283, 285).

[0204] For example, the electronic device (201) can acquire images including frames corresponding to the length of time stamps (i.e., total playback time) through each of the cameras (281, 283, 285). Thereafter, the electronic device (201) can remove (or exclude) some of the frames included in the images acquired through the cameras (281, 283, 285) from the video track (801).

[0205] In one embodiment, the electronic device (201) may remove (or exclude) from the video track (801) frames, excluding frames acquired by the first camera (281) that provides the source image of the preview image, from among the frames acquired during a period in which at least one event (E1, E2) is not identified while acquiring images through the cameras (281, 283, 285). In one embodiment, the electronic device (201) may remove (or exclude) from the video track (801) frames that are not related to at least one event (E1, E2) in order to reduce the weight of the video track (401). In one embodiment, the frames removed (or excluded) from the video track (801) may be frames that are not related to an event and / or frames that are not provided as preview images.

[0206] In one embodiment, the electronic device (201) may, while acquiring images through the cameras (281, 283, 285), remove (or exclude) some of the frames included in the images from the video track (801) based on at least one event (E1, E2) that is identified. The frames that are not removed (or excluded) from the video track (801) may be frames related to the event and / or frames provided as preview images. Here, among the frames of the source image before the event, the frames adjacent to the time when the event occurred may be recognized as related to the event. Among the frames of the source image after the event, the frames adjacent to the time when the event occurred may be recognized as related to the event. Some of the frames that are removed (or excluded) from the video track (801) may be frames that are not related to the event. Frames included in source images other than the source images before or after the event may be recognized as not related to the event. In one embodiment, the electronic device (201) may remove (or exclude) frames that are not related to at least one event (E1, E2) from the video track (801) in order to lighten the video track (401). In one embodiment, the electronic device (201) may not remove (or exclude) frames within a predetermined time from the time point at which at least one event (E1, E2) occurs from the video track (801) in order to smoothly transition between source images according to at least one event (E1, E2).

[0207] For example, if the source image before the event (E1) is an image acquired by the first camera (281) and the source image after the event (E1) is an image acquired by the third camera (285), the electronic device (201) may not remove (or exclude) frames (833, 835) adjacent to the event (E1) from the video track (801). For example, if the source image before the event (E1) is an image acquired by the first camera (281) and the source image after the event (E1) is an image acquired by the third camera (285), the electronic device (201) may remove (or exclude) frames (827, 829) adjacent to the event (E1) from the video track (801) among the frames included in the image acquired by the second camera (283).

[0208] For example, if the source image before the event (E2) is an image acquired by the third camera (285) and the source image after the event (E2) is an image acquired by the second camera (283), the electronic device (201) may not remove (or exclude) the frames (827, 829, 837) adjacent to the event (E2) from the video track (801).

[0209] Depending on the occurrence time of at least one event (E1, E2), the number of frames included in the video track (801) may vary. For example, the electronic device (201) may not remove frames within a specified time length from the occurrence time of at least one event (E1, E2) from the video track (801) for post-editing the video data (501). For example, the electronic device (201) may not remove frames related to at least one event (E1, E2) within a specified time length from the occurrence time of at least one event (E1, E2) from the video track (801). Here, frames included in the source video before or at least after at least one event (E1, E2) may be recognized as being related to at least one event (E1, E2).

[0210] In one embodiment, referring to FIG. 8B, the electronic device (201) may assign time stamp numbers of frames (833, 835, 837) acquired through a sub-camera (e.g., frames (827, 829) acquired through a second camera (283) and frames (833, 835, 837) acquired through a third camera (285)) based on time stamp numbers (871, 873, 875, 877, 879) of frames (811, 813, 815, 817, 819) acquired through a main camera (e.g., a first camera (281)). In one embodiment, one video track (801) of video data may include headers (841, 843, 845, 847, 849, 857, 859, 863, 865, 867) for frames (811, 813, 815, 817, 819, 827, 829, 833, 835, 837).

[0211] FIG. 9 illustrates an example of multiple cameras having different fields of view, according to one embodiment.

[0212] Referring to FIG. 9, cameras (911, 912, 913, 914, 915, 916) can be arranged to capture a specific space (e.g., a stadium (910)).

[0213] In one embodiment, the server (108) may be electrically connected to cameras (911, 912, 913, 914, 915, 916).

[0214] In one embodiment, the server (108) can synchronize the cameras (911, 912, 913, 914, 915, 916). In one embodiment, the server (108) can synchronize the shooting of the cameras (911, 912, 913, 914, 915, 916).

[0215] In one embodiment, the server (108) may set one camera (e.g., camera (911)) among the cameras (911, 912, 913, 914, 915, 916) as the main camera. In one embodiment, the server (108) may set other cameras among the cameras (911, 912, 913, 914, 915, 916) as sub cameras.

[0216] In one embodiment, the server (108) can generate video data in which frames acquired through cameras (911, 912, 913, 914, 915, 916) are arranged in a single video track. In one embodiment, the server (108) can generate video data in which frames acquired through cameras (911, 912, 913, 914, 915, 916) are arranged adjacent to frames acquired through the main camera. In one embodiment, the server (108) can generate video data in which frames acquired through cameras (911, 912, 913, 914, 915, 916) are arranged in a single video track in the order of their time stamps.

[0217] For example, the server (108) can transmit video data in which frames are aligned on a single video track to the electronic device (205). The electronic device (205) can play back images acquired through the cameras (911, 912, 913, 914, 915, 916) based on the video data from the server (108). For example, the electronic device (205) can play back images by switching a camera selected by an event among the cameras (911, 912, 913, 914, 915, 916).

[0218] FIG. 10 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0219] FIG. 10 can be explained with reference to FIG. 1, FIG. 2a, FIG. 3a, FIG. 3b, FIG. 3c, FIG. 4a, FIG. 4b, and FIG. 5a.

[0220] Referring to FIG. 10, in operation 1010, the electronic device (201) can acquire images (410, 420, 430) including frames through two or more cameras (281, 283, 285). The electronic device (201) can acquire an image (410) including frames (411, 413, 415, 417, 419) through the first camera (281). For example, the first camera (281) can acquire an image (410) including frames (411, 413, 415, 417, 419) based on a shooting command from the processor (221) of the electronic device (201). The electronic device (201) can obtain an image (420) including frames (421, 423, 425, 427, 429) through the second camera (283). For example, the second camera (283) can obtain an image (420) including frames (421, 423, 425, 427, 429) based on a shooting command from the processor (221) of the electronic device (201). The electronic device (201) can obtain an image (430) including frames (431, 433, 435, 437, 439) through the third camera (285). For example, the third camera (285) can acquire an image (430) including frames (431, 433, 435, 437, 439) based on a shooting command from the processor (221) of the electronic device (201).

[0221] In operation 1020, the electronic device (201) can generate video data (501) in which frames (411, 413, 415, 417, 419, 421, 423, 425, 427, 429, 431, 433, 435, 437, 439) are arranged in one video track (401) in the order of time stamp numbers (471, 473, 475, 477, 479).

[0222] In one embodiment, the electronic device (201) can generate video data including frames (411, 413, 415, 417, 419, 421, 423, 425, 427, 429, 431, 433, 435, 437, 439) in one video track (401). In one embodiment, the electronic device (201) can generate video data including one video track (401) in which frames (411, 413, 415, 417, 419, 421, 423, 425, 427, 429, 431, 433, 435, 437, 439) are arranged in playback order. For example, the fact that the frames are arranged in playback order may mean that the frames (411, 413, 415, 417, 419, 421, 423, 425, 427, 429, 431, 433, 435, 437, 439) are arranged in the order of their timestamps (or, playback time, presentation time). As the frames are arranged in playback order, frames from other videos may be inserted between frames from one video.

[0223] FIG. 11 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0224] FIG. 11 can be explained with reference to FIG. 1, FIG. 2a, FIG. 3a, FIG. 3b, FIG. 3c, FIG. 4a, FIG. 4b, and FIG. 5a.

[0225] Referring to FIG. 11, in operation 1110, the electronic device (201) may display a frame acquired through a selected camera among two or more cameras (281, 283, 285) as a preview image. For example, the electronic device (201) may display a frame acquired through a main camera (e.g., a first camera (281)) among two or more cameras (281, 283, 285) as a preview image on the display (261).

[0226] In operation 1120, the electronic device (201) may determine whether an event (E1, E2) is detected. In one embodiment, the electronic device (201) may identify at least one event (E1, E2) while acquiring images (410, 420, 430) through the cameras (281, 283, 285). For example, the at least one event (E1, E2) may include identification of an input for user control of an image being captured. For example, the at least one event (E1, E2) may be related to a preview image. For example, the at least one event (E1, E2) may include identification of an input requesting a change (or switching) of a source camera (or source image) for displaying a preview image among the cameras (281, 283, 285).

[0227] In operation 1120, if an event (E1, E2) is detected, the electronic device (201) may perform operation 1130. In operation 1120, if an event (E1, E2) is not detected, the electronic device (201) may perform operation 1120 again.

[0228] In operation 1130, the electronic device (201) may determine whether a change in the source camera is required. In one embodiment, the electronic device (201) may determine whether a change in the source camera is required by at least one event (E1, E2).

[0229] For example, in the case of a zoom-in event (E1), if the magnification according to the zoom-in is out of a specified range (e.g., more than 0.8 times to 3 times), the electronic device (201) can determine that a change of the source camera is required. For example, in the case of a zoom-in event (E1), if the magnification according to the zoom-in is within a specified range, the electronic device (201) can determine that a change of the source camera is not required. For example, in the case of a zoom-out event (E2), if the magnification according to the zoom-out is out of a specified range (e.g., more than 0.8 times to 3 times), the electronic device (201) can determine that a change of the source camera is required. For example, in the case of a zoom-out event (E2), if the magnification according to the zoom-out is within a specified range, the electronic device (201) can determine that a change of the source camera is not required.

[0230] In operation 1130, if a change in the source camera is required, the electronic device (201) may perform operation 1140. In operation 1130, if a change in the source camera is not required, the electronic device (201) may perform operation 1150 again.

[0231] In operation 1140, the electronic device (201) may record metadata for changing to a frame acquired from a different source camera than the previously displayed frame. The electronic device (201) may record information for changing to a frame acquired from a different source camera than the previously displayed frame in the timed meta track box (590). In one embodiment, the electronic device (201) may store information (e.g., camera ID and / or sample ID) about a frame to be played back according to at least one event (E1, E2) in the timed meta track box (590). In one embodiment, the electronic device (201) may store information about an input requesting a change (or switching) of a source camera (or source video) for displaying a preview image among the cameras (281, 283, 285) in the timed meta track box (590). Here, changing (or switching) the source camera (or source image) may include zooming in, zooming out, and / or switching the camera.

[0232] In operation 1150, the electronic device (201) may record metadata for maintaining a frame acquired from the same source camera as a previously displayed frame. The electronic device (201) may record information for maintaining a frame acquired from the same source camera as a previously displayed frame in a timed meta track box (590). In one embodiment, the electronic device (201) may store information (e.g., camera ID and / or sample ID) about a frame to be played back according to at least one event (E1, E2) in the timed meta track box (590). In one embodiment, the electronic device (201) may store information indicating the maintenance of a source camera (or source video) for displaying a preview image among the cameras (281, 283, 285) in the timed meta track box (590). For example, a timed meta track box (590) can indicate the retention of a source camera (or source image) by having metadata for frames with different sample IDs indicate the same camera ID.

[0233] FIG. 12 is a flowchart illustrating the operation of an electronic device according to one embodiment.

[0234] FIG. 12 can be explained with reference to FIG. 1, FIG. 2a, FIG. 3a, FIG. 3b, FIG. 3c, FIG. 6a, FIG. 6b, FIG. 6c, FIG. 7a, FIG. 7b, FIG. 7c, and FIG. 7d.

[0235] Referring to FIG. 12, in operation 1210, the electronic device (205) can play back video data (501) in which frames are arranged in the order of time stamps in one video track (401) through two or more cameras (281, 283, 285). For example, the electronic device (205) can display, through the display (265), frames indicated by the timed meta track box (590) among the frames of the images (410, 420, 430) included in the video track (401) in the order of time stamp numbers. In one embodiment, the electronic device (205) can display, in the playback area (740) of the display (265), a frame (710) indicated by the timed meta track box (590) among the frames (710, 720, 730) having the same time stamp number.

[0236] In operation 1220, the electronic device (205) may determine whether an event (701, 703) is detected. The electronic device (205) may identify at least one event while playing back video data (501). For example, the at least one event may include identification of an input for user control of the video being played back. For example, the at least one event may be related to a playing image. For example, the at least one event may include identification of an input requesting a change (or switching) of a source video (or source camera) for displaying a playing image among the videos (410, 420, 430).

[0237] In one embodiment, events (701, 703) may be events recorded in a timed meta track box (590) of video data (501), but are not limited thereto. In one embodiment, events (701, 703) may be events generated by user input during playback of videos (410, 420, 430).

[0238] In operation 1220, if an event (701, 703) is detected, the electronic device (201) may perform operation 1230. In operation 1220, if an event (701, 703) is not detected, the electronic device (201) may perform operation 1220 again.

[0239] In operation 1230, the electronic device (205) may determine whether a change in the source camera is required. In one embodiment, the electronic device (205) may determine whether a change in the source camera is required by at least one event (701, 703).

[0240] In one embodiment, the electronic device (205) can change the source image of the playback image displayed through the display (265) according to the magnification according to the zoom-in by the event (701).

[0241] For example, if the magnification according to the zoom-in by the event (701) is within the standard magnification range (e.g., more than 0.8 times and less than 3 times), the electronic device (205) may not change the source image of the playback image displayed through the display (265) from the image (410) acquired by the first camera (281). For example, if the magnification according to the zoom-in by the event (701) is outside the standard magnification range (e.g., more than 3 times), the electronic device (205) may change the source image of the playback image displayed through the display (265) from the image (410) acquired by the first camera (281) to the image (420) acquired by the second camera (283).

[0242] In operation 1230, if a change in the source camera is required, the electronic device (201) may perform operation 1240. In operation 1230, if a change in the source camera is not required, the electronic device (201) may perform operation 1250 again.

[0243] At operation 1240, the electronic device (205) may display on the display (265) a frame acquired from the same source camera as the previously displayed frame.

[0244] The electronic device (205) can display an enlarged image from a frame acquired from the same source camera as the previously displayed frame through the display (265) when the magnification according to the zoom-in by the event (701) is within the reference magnification range (e.g., more than 0.8 times and less than 3 times). The electronic device (205) can display a reduced image from a frame acquired from the same source camera as the previously displayed frame through the display (265) when the magnification according to the zoom-out by the event (703) is within the reference magnification range (e.g., more than 0.8 times and less than 3 times).

[0245] At operation 1250, the electronic device (205) may display on the display (265) a frame acquired from a source camera different from the previously displayed frame.

[0246] The electronic device (205) may display an image based on a frame acquired from a source camera different from a previously displayed frame through the display (265) when the magnification due to the zoom-in by the event (701) exceeds the reference magnification range (e.g., 3 times or more). The electronic device (205) may display an image based on a frame acquired from a source camera different from a previously displayed frame through the display (265) when the magnification due to the zoom-out by the event (703) exceeds the reference magnification range (e.g., 0.8 times or less).

[0247] As described above, the electronic device (201) may include a processor (221). The electronic device (201) may include a memory (231) that stores instructions. The instructions, when executed by the processor (221), may cause the electronic device (201) to acquire a first image (410) including first frames (411, 413, 415, 417, 419) through a first camera (281). The instructions, when executed by the processor (221), may cause the electronic device (201) to acquire a second image (420) including second frames (421, 423, 425, 427, 429) through a second camera (283). The above instructions, when executed by the processor (221), may cause the electronic device (201) to generate video data (501) including a first image (410) and a second image (420) in a single video track.

[0248] In one embodiment, one frame (411) among the first frames (411, 413, 415, 417, 419) and a frame (421) among the second frames (421, 423, 425, 427, 429) having a related time stamp may have the same time stamp number.

[0249] In one embodiment, one frame (411) among the first frames (411, 413, 415, 417, 419) and a frame (421) among the second frames (421, 423, 425, 427, 429) having a related time stamp may have the same time stamp number stored as metadata.

[0250] In one embodiment, the metadata (521) of the first frames (411, 413, 415, 417, 419) may be included in the metadata area (511) of the video data (501). In one embodiment, the metadata (525) of the second frames (421, 423, 425, 427, 429) may be included in a user data area (515) different from the metadata area.

[0251] In one embodiment, the metadata (525) of the second frames (421, 423, 425, 427, 429) may include an offset indicating a correction value for the time stamp difference between the second frame and the first frame having the same time stamp number.

[0252] In one embodiment, the metadata of the second frames (421, 423, 425, 427, 429) may indicate at least one first frame among the first frames (411, 413, 415, 417, 419) as a reference frame for decoding at least one frame among the second frames (421, 423, 425, 427, 429).

[0253] The electronic device (201) may further include a display (261). The instructions, when executed by the processor (221), may cause the electronic device (201) to detect an event for changing a source image of a preview image displayed through the display (261) from the first frames (411, 413, 415, 417, 419) to the second frames (421, 423, 425, 427, 429) while acquiring the video data including the first image (410) and the second image (420) in the single video track. The above instructions, when executed by the processor (221), may cause the electronic device (201) to generate the video data (501) including information for changing the playback video from the first frames (411, 413, 415, 417, 419) to the second frames (421, 423, 425, 427, 429) at a timestamp number corresponding to a timestamp at which the event is detected when the video is played back through the video data (501).

[0254] In one embodiment, the second frame included in the second frames (421, 423, 425, 427, 429) and indicated by the timestamp number for changing the playback video may be captured before other second frames when the event is detected.

[0255] The instructions, when executed by the processor (221), may cause the electronic device (201) to generate the video data (501) including the first image (410) that is missing at least a portion of the first frames (411, 413, 415, 417, 419) that were captured after the event was detected and the second image (420) that is missing at least a portion of the second frames (421, 423, 425, 427, 429) that were captured before the event was detected.

[0256] As described above, a non-transitory computer readable storage medium can store a program including instructions. The instructions, when executed by the processor (221) of the electronic device (201), can cause the electronic device (201) to acquire a first image (410) including first frames (411, 413, 415, 417, 419) through the first camera (281). The instructions, when executed by the processor (221) of the electronic device (201), can cause the electronic device (201) to acquire a second image (420) including second frames (421, 423, 425, 427, 429) through the second camera (283). The above instructions, when executed by the processor (221) of the electronic device (201), may cause the electronic device (201) to generate video data (501) including the first image (410) and the second image (420) in a single video track.

[0257] As described above, the method may be performed by the electronic device (201). The method may include an operation of acquiring a first image (410) including first frames (411, 413, 415, 417, 419) through a first camera (281). The method may include an operation of acquiring a second image (420) including second frames (421, 423, 425, 427, 429) through a second camera (283). The method may include an operation of generating video data (501) including the first image (410) and the second image (420) in a single video track.

[0258] As described above, the electronic device (205) may include a display (265). The electronic device (205) may include a processor (225). The electronic device (205) may include a memory (235) that stores instructions. The instructions, when executed by the processor (225), may cause the electronic device (205) to obtain video data (501) as a single video track (401) that includes a first image (410) and a second image (420) in the single video track. The instructions, when executed by the processor (225), may cause the electronic device (205) to play back first frames (411, 413, 415, 417, 419) of the first image (410) according to timestamp numbers. The above instructions, when executed by the processor (225), may cause the electronic device (205) to play the second frames (421, 423, 425, 427, 429) according to the timestamp numbers when an input event (E1, E2) is received to change from the first frames (411, 413, 415, 417, 419) to the second frames (421, 423, 425, 427, 429) of the second image (420) while playing the first frames (411, 413, 415, 417, 419).

[0259] In one embodiment, one frame (411) among the first frames (411, 413, 415, 417, 419) and a frame (421) among the second frames (421, 423, 425, 427, 429) having a related time stamp may have the same time stamp number.

[0260] In one embodiment, one frame (411) among the first frames (411, 413, 415, 417, 419) and a frame (421) among the second frames (421, 423, 425, 427, 429) having a related time stamp may have the same time stamp number stored as metadata.

[0261] In one embodiment, the metadata (521) of the first frame (411, 413, 415, 417, 419) may be included in a metadata area (511) of video data (501) including the video of the single video track (401). In one embodiment, the metadata (525) of the second frame (421, 423, 425, 427, 429) may be included in a user data area (515) different from the metadata area.

[0262] In one embodiment, the metadata (525) of the second frames (421, 423, 425, 427, 429) may include an offset indicating a correction value for a time stamp difference between the second frame and the first frame having the same time stamp number. The instructions, when executed by the processor (225), may cause the electronic device (205) to reproduce the second frames (421, 423, 425, 427, 429) at time stamps adjusted based on the offset.

[0263] In one embodiment, the metadata of the second frames (421, 423, 425, 427, 429) may indicate at least one first frame of the first frames (411, 413, 415, 417, 419) as a reference frame for decoding at least one frame of the second frames (421, 423, 425, 427, 429).

[0264] The instructions, when executed by the processor (225), may cause the electronic device (205) to identify whether an angle of view of a camera capturing the second frame (421, 423, 425, 427, 429) corresponds to an angle of view of the zoomed-out first frame in response to the input event (E1, E2) being played back through the display (265) to zoom out the first frame of the first frames (411, 413, 415, 417, 419). The instructions, when executed by the processor (225), may cause the electronic device (205) to reproduce the second frames (421, 423, 425, 427, 429) according to the time stamp numbers in response to identifying that the angle of view of the camera corresponds to the angle of view of the zoomed-out first frame. The instructions, when executed by the processor (225), may cause the electronic device (205) to reproduce the first frames (411, 413, 415, 417, 419) by zooming out in response to identifying that the angle of view of the camera does not correspond to the angle of view of the zoomed-out first frame.

[0265] The instructions, when executed by the processor (225), may cause the electronic device (205) to identify whether an angle of view of a camera capturing the second frame (421, 423, 425, 427, 429) corresponds to an angle of view of the zoomed-in first frame in response to the input event (E1, E2) being played back through the display (265) to zoom in on the first frame of the first frames (411, 413, 415, 417, 419). The instructions, when executed by the processor (225), may cause the electronic device (205) to reproduce the second frames (421, 423, 425, 427, 429) according to the time stamp numbers in response to identifying that the angle of view of the camera corresponds to the angle of view of the zoomed-in first frame. The instructions, when executed by the processor (225), may cause the electronic device (205) to reproduce the first frames (411, 413, 415, 417, 419) by zooming in according to the time stamp numbers in response to identifying that the angle of view of the camera does not correspond to the angle of view of the zoomed-in first frame.

[0266] A non-transitory computer readable storage medium, as described above, may store a program including instructions. The instructions, when executed by a processor (225) of an electronic device (205) including a display (265), may cause the electronic device (205) to obtain video data (501) as a single video track (401) including a first image (410) and a second image (420) in the single video track. The instructions, when executed by the processor (225), may cause the electronic device (205) to play back first frames (411, 413, 415, 417, 419) of the first image (410) according to timestamp numbers. The above instructions, when executed by the processor (225), may cause the electronic device (205) to play the second frames (421, 423, 425, 427, 429) according to the timestamp numbers when an input event (E1, E2) is received to change from the first frames (411, 413, 415, 417, 419) to the second frames (421, 423, 425, 427, 429) of the second image (420) while playing the first frames (411, 413, 415, 417, 419).

[0267] As described above, the method may be performed by an electronic device (205) including a display (265). The method may include an operation of obtaining video data (501) as a single video track (401) including a first image (410) and a second image (420) in the single video track. The method may include an operation of playing back first frames (411, 413, 415, 417, 419) of the first image (410) according to timestamp numbers. The method may include, when an input event (E1, E2) is received to change from the first frames (411, 413, 415, 417, 419) to the second frames (421, 423, 425, 427, 429) of the second image (420) while playing the first frames (411, 413, 415, 417, 419), playing the second frames (421, 423, 425, 427, 429) according to the timestamp numbers.

[0268] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0269] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0270] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0271] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0272] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., by download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0273] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (201), processor (221), and It includes a memory (231) that stores instructions, and when the instructions are executed by the processor (221), the electronic device (201) A first image (410) including first frames (411, 413, 415, 417, 419) is acquired through a first camera (281), A second image (420) including second frames (421, 423, 425, 427, 429) is acquired through a second camera (283), Causing to generate video data (501) including a first image (410) and a second image (420) in a single video track. Electronic devices.

2. In claim 1, A frame (411) among the first frames (411, 413, 415, 417, 419) and a frame (421) among the second frames (421, 423, 425, 427, 429) having a related time stamp are characterized in that they have the same time stamp number. Electronic devices.

3. In claims 1 to 2, A frame (411) among the first frames (411, 413, 415, 417, 419) and a frame (421) among the second frames (421, 423, 425, 427, 429) having a related time stamp are characterized in that they have the same time stamp number stored as metadata. Electronic devices.

4. In claims 1 to 3, The metadata (521) of the above first frames (411, 413, 415, 417, 419) is included in the metadata area (511) of the video data (501), The metadata (525) of the above second frames (421, 423, 425, 427, 429) is included in a user data area (515) different from the metadata area. Electronic devices.

5. In claims 1 to 4, The metadata (525) of the second frames (421, 423, 425, 427, 429) includes an offset indicating a correction value for the time stamp difference between the second frame and the first frame having the same time stamp number. Electronic devices.

6. In claims 1 to 5, The metadata of the second frames (421, 423, 425, 427, 429) designates at least one first frame among the first frames (411, 413, 415, 417, 419) as a reference frame for decoding at least one frame among the second frames (421, 423, 425, 427, 429). Electronic devices.

7. In claims 1 to 6, Further including a display (261), When the above instructions are executed by the processor (221), the electronic device (201) While acquiring the video data including the first image (410) and the second image (420) in the single video track, detecting an event for changing the source image of the preview image displayed through the display (261) from the first frames (411, 413, 415, 417, 419) to the second frames (421, 423, 425, 427, 429), When the video is played back through the above video data (501), the video data (501) is generated, which includes information for changing the played back video from the first frames (411, 413, 415, 417, 419) to the second frames (421, 423, 425, 427, 429) at the timestamp number corresponding to the timestamp at which the event is detected. Electronic devices.

8. In claims 1 to 7, The second frame, which is included in the second frames (421, 423, 425, 427, 429) and is indicated by the timestamp number for changing the playback image, is captured before other second frames when the event is detected. Electronic devices.

9. In claims 1 to 8, When the above instructions are executed by the processor (221), the electronic device (201) Causing to generate the video data (501) including the first image (410) that is free of at least a portion captured after the event is detected among the first frames (411, 413, 415, 417, 419) and the second image (420) that is free of at least a portion captured before the event is detected among the second frames (421, 423, 425, 427, 429). Electronic devices.

10. In an electronic device (205), Display (265), processor (225), and It includes a memory (235) that stores instructions, and when the instructions are executed by the processor (225), the electronic device (205) Obtaining video data (501) as a single video track (401) including a first image (410) and a second image (420) in a single video track, The first frames (411, 413, 415, 417, 419) of the first video (410) are played according to the timestamp numbers, While playing back the first frames (411, 413, 415, 417, 419), if an input event (E1, E2) is received to change from the first frames (411, 413, 415, 417, 419) to the second frames (421, 423, 425, 427, 429) of the second image (420), causing the second frames (421, 423, 425, 427, 429) to be played back according to the timestamp numbers. Electronic devices.

11. In claim 10, A frame (411) among the first frames (411, 413, 415, 417, 419) and a frame (421) among the second frames (421, 423, 425, 427, 429) having a related time stamp are characterized in that they have the same time stamp number. Electronic devices.

12. In claims 10 to 11, A frame (411) among the first frames (411, 413, 415, 417, 419) and a frame (421) among the second frames (421, 423, 425, 427, 429) having a related time stamp are characterized in that they have the same time stamp number stored as metadata. Electronic devices.

13. In claims 10 to 12, The metadata (521) of the first frame (411, 413, 415, 417, 419) is included in the metadata area (511) of the video data (501) including the video of the single video track (401), The metadata (525) of the second frame (421, 423, 425, 427, 429) is included in a user data area (515) different from the metadata area. Electronic devices.

14. In a non-transitory computer readable storage medium, Save a program containing instructions, When the above instructions are executed by the processor (221) of the electronic device (201), the electronic device (201), A first image (410) including first frames (411, 413, 415, 417, 419) is acquired through a first camera (281), A second image (420) including second frames (421, 423, 425, 427, 429) is acquired through a second camera (283), Causing to generate video data (501) including a first image (410) and a second image (420) in a single video track. A non-transitory computer-readable recording medium.

15. In a non-transitory computer readable storage medium, Save a program containing instructions, The above instructions, when executed by the processor (225) of the electronic device (205) including the display (265), cause the electronic device (205) to: Obtaining video data (501) as a single video track (401) including a first image (410) and a second image (420) in a single video track, The first frames (411, 413, 415, 417, 419) of the first video (410) are played according to the timestamp numbers, While playing back the first frames (411, 413, 415, 417, 419), if an input event (E1, E2) is received to change from the first frames (411, 413, 415, 417, 419) to the second frames (421, 423, 425, 427, 429) of the second image (420), causing the second frames (421, 423, 425, 427, 429) to be played back according to the timestamp numbers. A non-transitory computer-readable recording medium.

Images