Electronic device for generating a content and method of operation thereof

Abstract

An electronic device according to one embodiment of the present disclosure includes a memory configured to store a first content including at least one moving object, and a processor connected to the memory. The memory may store one or more instructions such that, at execution, the processor identifies the at least one moving object in the first content, identifies attribute information of the at least one moving object, determines the priority of the at least one moving object based on the attribute information, searches for a sound source for the at least one moving object based on the priority and the attribute information, determines at least one of the at least one moving objects as a motion-providing object, determines a sound playback method for the determined motion-providing object, forms a portion of the first content excluding the determined motion-providing object as a fixed image, and stores the searched sound source based on the determined sound playback method to form a second content.

Description

Technology Field

[0001] Various embodiments disclosed in this document relate to an electronic device for forming content and a method of operating the same. Background Technology

[0002] A cinemagraph is a compound word of "cinema" and "photograph," referring to content created by designating a moving area within a video so that only the parts to be emphasized move while the rest remain static.

[0003] Therefore, cinemagraphs combine the static feel of photographs with the vitality of videos, giving the impression of an exquisite blend of photography and video.

[0004] The problem may be finding a method to conveniently and easily form a cinemagraph that provides diverse experiences to the user. The problem to be solved

[0005] The present invention relates to an apparatus and a method of operation that provide a method for forming a cinemagraph capable of quickly and conveniently forming content including a cinemagraph. means of solving the problem

[0006] An electronic device according to one embodiment of the present disclosure includes a memory configured to store a first content including at least one moving object, and a processor connected to the memory. The memory may store one or more instructions such that, at execution, the processor identifies the at least one moving object in the first content, identifies attribute information of the at least one moving object, determines the priority of the at least one moving object based on the attribute information, searches for a sound source for the at least one moving object based on the priority and the attribute information, determines at least one of the at least one moving objects as a motion-providing object, determines a sound playback method for the determined motion-providing object, forms a portion of the first content excluding the determined motion-providing object as a fixed image, and stores the searched sound source based on the determined sound playback method to form a second content.

[0007] A method for forming content by an electronic device comprising a memory configured to store at least one moving object and a processor connected to the memory, according to one embodiment of the present disclosure, may identify the at least one moving object in the first content, identify attribute information of the at least one moving object, determine a priority of the at least one moving object based on the attribute information, search for a sound source for the at least one moving object based on the priority and the attribute information, determine at least one of the at least one moving objects as a motion-providing object, determine a sound playback method for the determined motion-providing object, form a portion of the first content excluding the determined motion-providing object as a fixed image, and store the searched sound source based on the determined sound playback method to form a second content. Effects of the invention

[0008] According to the various embodiments disclosed in this document, an electronic device can automatically identify the type and attributes of objects within a video and automatically map sound to ambient sound that can be heard in the surrounding environment, rather than sound based on musical instruments and melodies, to form content.

[0009] According to the various embodiments disclosed in this document, by providing an electronic device that provides a method for creating content including a cinemagraph and a method of operating the same, a more emotional user experience can be provided and the interactive content pool can be expanded.

[0010] In addition, various effects that can be identified directly or indirectly through this document may be provided. Brief explanation of the drawing

[0011] FIG. 1 is a block diagram of an electronic device in a network environment according to various embodiments. FIG. 2 is a flowchart showing the operation of an electronic device according to one embodiment. FIG. 3 is a diagram illustrating the operation of an electronic device according to one embodiment. FIG. 4 is a diagram showing an example of a sound library to explain the operation of an electronic device according to one embodiment. FIG. 5 is a drawing for explaining an example of an electronic device forming content according to one embodiment. FIG. 6 is a table mapping the atmosphere of an image to sound, used in the operation of an electronic device according to one embodiment. FIG. 7 is a drawing for explaining an example of an electronic device forming content according to one embodiment. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Specific details for implementing the invention

[0012] Hereinafter, various embodiments of the present invention are described with reference to the accompanying drawings. However, this is not intended to limit the present invention to specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present invention.

[0014] Hereinafter, with reference to FIG. 1, the configuration of an electronic device according to one embodiment will be described.

[0015] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments.

[0016] FIG. 1 is a block diagram of an electronic device (101) in a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with an electronic device (102) through a first network (198) (e.g., a short-range wireless communication network) or with an electronic device (104) or a server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through a server (108). According to one embodiment, the electronic device (101) may include a processor (120), memory (130), input device (150), sound output device (155), display device (160), audio module (170), sensor module (176), interface (177), haptic module (179), camera module (180), power management module (188), battery (189), communication module (190), subscriber identification module (196), or antenna module (197). In some embodiments, at least one of these components (e.g., display device (160) or camera module (180)) may be omitted from the electronic device (101), or one or more other components may be added. In some embodiments, some of these components may be implemented as a single integrated circuit. For example, a sensor module (176) (e.g., fingerprint sensor, iris sensor, or light sensor) can be implemented embedded in a display device (160) (e.g., display).

[0017] The processor (120) can control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing software (e.g., a program (140)), for example, and can perform various data processing or operations. According to one embodiment, as at least part of the data processing or operations, the processor (120) can load commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) into volatile memory (132), process the commands or data stored in volatile memory (132), and store the resulting data in non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) and an auxiliary processor (123) (e.g., a graphics processing unit, an image signal processor, a sensor hub processor, or a communication processor) that can be operated independently or together with it. Additionally or generally, the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a designated function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as part thereof.

[0018] The auxiliary processor (123) can control at least some of the functions or states associated with at least one component of the electronic device (101) (e.g., display device (160), sensor module (176), or communication module (190)) on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., image signal processor or communication processor) may be implemented as part of another functionally related component (e.g., camera module (180) or communication module (190)).

[0019] The memory (130) can store various data used by at least one component of the electronic device (101) (e.g., processor (120) or sensor module (176)). The data may include, for example, software (e.g., program (140)) and input data or output data for related commands. The memory (130) may include volatile memory (132) or non-volatile memory (134).

[0020] The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0021] The input device (150) can receive commands or data to be used for a component of the electronic device (101) (e.g., processor (120)) from outside the electronic device (101) (e.g., user). The input device (150) may include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus pen).

[0022] The sound output device (155) can output a sound signal to the outside of the electronic device (101). The sound output device (155) may include, for example, a speaker or a receiver. The speaker may be used for general purposes such as multimedia playback or recording playback, and the receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from the speaker or as part thereof.

[0023] The display device (160) can visually provide information to an external (e.g., user) of the electronic device (101). The display device (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling said device. According to one embodiment, the display device (160) may include a touch circuitry configured to detect a touch, or a sensor circuitry configured to measure the intensity of the force generated by said touch (e.g., a pressure sensor).

[0024] The audio module (170) can convert sound into an electrical signal or, conversely, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through an input device (150) or output sound through an audio output device (155) or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphones) that is directly or wirelessly connected to the electronic device (101).

[0025] The sensor module (176) can detect the operating state of the electronic device (101) (e.g., power or temperature) or the external environmental state (e.g., user state) and generate an electrical signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) may include, for example, a gesture sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biosensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0026] The interface (177) may support one or more specified protocols that can be used for the electronic device (101) to be connected directly or wirelessly to an external electronic device (e.g., electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0027] The connection terminal (178) may include a connector through which the electronic device (101) can be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0028] The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module (179) may include, for example, a motor, a piezoelectric element, or an electric stimulation device.

[0029] The camera module (180) can capture still images and video. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0030] The power management module (188) can manage the power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least part of a power management integrated circuit (PMIC).

[0031] The battery (189) can supply power to at least one component of the electronic device (101). According to one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0032] The communication module (190) can support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between an electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may include one or more communication processors that operate independently of the processor (120) (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., cellular communication module, short-range wireless communication module, or GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., LAN (local area network) communication module, or power line communication module). The corresponding communication module among these communication modules can communicate with an external electronic device (104) through a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi Direct, or IrDA (infrared data association)) or a second network (199) (e.g., a cellular network, the Internet, or a long-range communication network such as a computer network (e.g., LAN or WAN). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify and authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) using subscriber information (e.g., International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module (196).

[0033] An antenna module (197) can transmit a signal or power to or from an external source (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include a single antenna comprising a radiator made of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network, such as a first network (198) or a second network (199), may be selected from the plurality of antennas, for example, by a communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. According to some embodiments, in addition to the radiator, other components (e.g., an RFIC) may be additionally formed as part of the antenna module (197).

[0034] At least some of the above components can be connected to each other via a communication method between peripheral devices (e.g., bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)) and exchange signals (e.g., commands or data) with each other.

[0035] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) through a server (108) connected to a second network (199). Each of the external electronic devices (102, 104) may be the same or different type of device as the electronic device (101). According to one embodiment, all or part of the operations performed on the electronic device (101) may be performed on one or more of the external electronic devices (102, 104, or 108). For example, if the electronic device (101) needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device (101) may request one or more external electronic devices to perform at least part of the function or service instead of performing the function or service itself or additionally. One or more external electronic devices that receive the above request may execute at least part of the requested function or service, or additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may provide the result as is or additionally processed as at least part of the response to the request. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology may be used.

[0037] Hereinafter, the operation of an electronic device according to one embodiment will be described with reference to FIGS. 2, FIGS. 3, and FIGS. 4.

[0038] FIG. 2 is a flowchart (200) showing the operation of an electronic device according to one embodiment. FIG. 3 is a diagram (300) for explaining the operation of an electronic device according to one embodiment. FIG. 4 is a diagram (400) showing an example of a sound library to explain the operation of an electronic device according to one embodiment. The operation of the electronic device described below can be performed in a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1).

[0039] Referring to FIG. 2, in operation 201, an electronic device (e.g., the electronic device (101) of FIG. 1) can identify a moving area within an image stored in the memory of the electronic device (e.g., the memory (130) of FIG. 1). According to one embodiment, the image may be at least a portion of a video captured by a camera module of the electronic device. According to one embodiment, the image may be an image portion excluding sound from a video captured by a camera module of the electronic device. According to one embodiment, the format of the image may be an animated GIF, etc. According to one embodiment, a moving area may be an area defined by objects where pixel changes exist in the image. For example, in the case of an image of a car driving in a rainy environment, the moving area may be identified as a first area containing the driving car and a second area containing the falling rain in the background. According to one embodiment, the number of moving areas (or objects) to be identified within a single image may be set by the user. For example, if the user sets the number of moving regions (or objects) to be identified to 2, only 2 moving regions can be selectively identified within a single image.

[0040] In operation 202, the electronic device can identify objects included in the identified moving area. For example, the electronic device can identify an object in the ‘first area containing a moving vehicle’ as a ‘vehicle’. For example, the electronic device can identify an object in the ‘second area containing falling rain’ as ‘rain’. According to one embodiment, the electronic device can identify objects included in the moving area using computer vision technology.

[0041] In operation 203, the electronic device can identify attribute information of the identified object. According to one embodiment, the electronic device can identify information for determining priority and / or information for determining a sound playback method as attribute information of the object.

[0042] According to one embodiment, information for determining priority may include at least one of focus information, area information, location information, and color information. According to one embodiment, focus information may include a focus degree determined inversely proportional to the degree of blur of pixels included in an object image. According to one embodiment, area information may include the ratio of the area occupied by the object within the screen. According to one embodiment, location information may include information indicating the position of the object from the center of the screen by dividing the screen into 3x3 (horizontal x vertical). According to one embodiment, color information may include information regarding difference values ​​such as hue, brightness, saturation, and contrast of the object compared to surrounding images on the screen.

[0043] According to one embodiment, information for determining the sound playback method may include at least one of associated object information, motion intensity information, motion direction information, sound playback timing information, and surrounding situation information. According to one embodiment, the associated object information may include information about surrounding objects associated with an object identified in operation 202. For example, if the identified object is 'rain', the associated object may be a dirt ground where rain is falling, on a roof, on a plastic container, etc., depending on the image. Or, for example, if the identified object is 'wave', the associated object may be a gravel beach, a sandy beach, etc., depending on the image. According to one embodiment, the motion intensity information may include information classifying the degree of movement of an object into strong, medium, or weak according to a certain standard. According to one embodiment, the motion direction information may include information about the direction in which the object moves within the screen. According to one embodiment, the sound playback timing information may include information about the time when the object starts and ends movement in the image. According to one embodiment, the surrounding situation information may include information about the location, background, atmosphere, etc., within the screen where the object exists. For example, if a person drinking coffee at a cafe is identified as an object, the surrounding context information may include details such as the cafe, people talking, and pleasantness.

[0044] In operation 204, the electronic device can determine the priority of objects within an image based on identified attribute information. According to one embodiment, the electronic device can determine the priority of objects by using at least one of focus information, area information, location information, and color information included in the information for determining priority. According to one embodiment, when using focus information, the electronic device may determine that objects with a higher degree of focus, determined inversely proportional to the degree of blur of the pixels, have a higher priority. According to one embodiment, when using area information, the electronic device may determine that objects with a higher proportion of the area they occupy within the screen have a higher priority. According to one embodiment, when using location information, the electronic device may determine that objects closer to the center of the screen have a higher priority. According to one embodiment, when using color information, the electronic device may determine that objects with a greater difference in hue, brightness, saturation, or contrast compared to surrounding images on the screen have a higher priority.

[0045] Alternatively, according to one embodiment, the electronic device may sequentially determine focus information, area information, location information, and color information included in the information for determining priority. In addition, according to one embodiment, if priority is not determined even after sequentially determining the focus information, area information, location information, and color information included in the information for determining priority, the electronic device may randomly determine the priority of the objects.

[0046] Hereinafter, with reference to FIG. 3, the method by which an electronic device determines the priority of objects will be explained in detail.

[0047] When an electronic device determines priority for objects included in the first image (310) of FIG. 3, it may first use focus information. The electronic device may determine that the priority of the hand pouring water (311) that is in focus is higher than that of the water bottle (312) that is out of focus and the moving hand (313).

[0048] When an electronic device determines the priority of objects included in the second image (320) of FIG. 3, it may first use focus information. If the electronic device determines that the degree of focus of the objects included in the second image (320) is all the same and thus the priority cannot be determined, it may next use area information. The electronic device may determine that the salad bowl (321) with the largest area in the second image (320) has a higher priority than the surrounding apples (322).

[0049] When an electronic device determines priority for objects included in the third image (330) of FIG. 3, it may first use focus information. If the electronic device determines that priority cannot be determined because the degree of focus of the objects included in the third image (330) is all the same, it may next use area information. If the electronic device determines that priority cannot be determined because the area of ​​the objects included in the third image (330) is all the same, it may next use location information. The electronic device may determine that the first dish (331) located at the center of the screen has a higher priority than the surrounding second dish (332) and third dish (333).

[0050] When the electronic device determines the priority of objects included in the fourth image (340) of FIG. 3, it may first use focus information. If the electronic device determines that the degree of focus of the objects included in the fourth image (340) is all the same and thus the priority cannot be determined, it may next use area information. If the electronic device determines that the area of ​​the objects included in the fourth image (340) is all the same and thus the priority cannot be determined, it may next use location information. If the electronic device determines that the locations of the objects included in the fourth image (340) are similar and thus the priority cannot be determined, it may next use color information. The electronic device may determine that the priority of the apple (341) has a different hue from the surrounding apples (342) is the highest.

[0051] Referring again to FIG. 2, in operation 205, the electronic device can search for a sound source corresponding to an object. According to one embodiment, the electronic device can search for a sound source having a matching tag value based on the object's name and / or attribute information in a sound library. According to one embodiment, an example of a sound library may be as shown in the table (400) in FIG. 4. According to one embodiment, the sound library may be stored in the memory of the electronic device or may be stored on a cloud server.

[0052] According to one embodiment, an electronic device may search for a sound source by considering at least one of an object's category, a main object, an associated object, a description of motion, and an intensity of motion. According to one embodiment, when searching for a sound source, the electronic device may consider which category an object or screen belongs to among nature (climate, topography, animals), a city (people, objects), etc.

[0053] According to one embodiment, when an electronic device searches for a sound source, it may consider what the main object is (e.g., bird, wave, snow, rain, glass, firework, etc.). According to one embodiment, when an electronic device searches for a sound source, it may consider what the associated object is. For example, the electronic device may determine that an object emitting a sound associated with the main object is an associated object; if the main object is snow, the electronic device may consider a person's foot, hand, etc., as an associated object. Also, for example, if the main object is rain, the electronic device may consider a dirt floor, roof, drum, etc., as an associated object. Also, for example, if the main object is a glass, the electronic device may consider ice, water, etc., as an associated object. Also, for example, if the main object is a wave, the electronic device may consider a gravel beach, sandy beach, etc., as an associated object. Additionally, if the main object is an object that emits sound on its own, such as a bird or a person, the electronic device may omit the consideration of associated objects.

[0054] According to one embodiment, when an electronic device searches for a sound source, it may consider a description of an object's movement. For example, if the main object is snow, the electronic device may consider actions such as stepping on snow with a foot or rubbing snow with a hand as a description of the object's movement. Additionally, for example, if the main object is a glass, the electronic device may consider actions such as clinking ice or pouring water as a description of the object's movement. Furthermore, for example, if the main object's movement is expressed through the sound of the main object itself, such as waves or rain, the electronic device may not consider a description of the movement.

[0055] According to one embodiment, an electronic device may consider the intensity of an object's movement when searching for a sound source. For example, when searching for a sound source, the electronic device may consider information classifying the intensity of an object's movement as strong, medium, or weak.

[0056] In operation 206, the electronic device may select at least one of the identified objects. The object selected in operation 206 may be an object to which movement is to be provided in the content to be formed by the electronic device. The content may be cinemagraph content. According to one embodiment, the electronic device may select an object for which a corresponding sound source was found in operation 205 as the object to which movement is to be provided. According to one embodiment, the electronic device may select an object with a high priority as the object to which movement is to be provided. According to one embodiment, the number of objects to be selected in operation 206 may be set by the user.

[0057] In operation 207, the electronic device can determine a sound playback method for a selected object. According to one embodiment, the electronic device can determine at least one of volume, sound playback timing, and sound playback direction in relation to the sound playback method.

[0058] According to one embodiment, an electronic device may determine the volume by considering at least one of the focus, area, and whether the object is a main object. According to one embodiment, the electronic device may determine to play at a low volume the higher the pixel blur value of the object, as the object is out of focus. According to one embodiment, the electronic device may determine to play at a volume proportional to the area occupied by the object within the screen. According to one embodiment, if the object is not a main object but corresponds to the surrounding environment, the electronic device may determine to play a sound source corresponding to the surrounding environment at a lower volume than a sound source corresponding to the main object. According to one embodiment, when determining the volume, the electronic device may determine it by sequentially considering the focus, area, and whether the object is a main object.

[0059] According to one embodiment, when determining the sound playback timing, the electronic device may consider the start and end points of an object's movement. According to one embodiment, the electronic device may determine whether the movement of an object within the screen is intermittent. According to one embodiment, the electronic device may decide to play a sound source only during the time from when the object starts moving until it ends. According to one embodiment, the electronic device may decide to fade in the volume at the start of sound source playback and fade out the volume at the end of sound source playback. According to one embodiment, the electronic device may decide to play the sound source at the highest volume at the point where the object's movement reaches its climax. According to one embodiment, the electronic device may determine the point where the degree of the object's movement is greatest as the climax.

[0060] According to one embodiment, when determining the sound playback direction, if the electronic device has two or more speaker channels, it may consider the direction of movement of an object within the screen. According to one embodiment, the electronic device may determine whether the movement of an object within the screen has directionality. According to one embodiment, the electronic device may decide to play a sound source corresponding to the position and direction of movement of an object within the screen. According to one embodiment, if an object moves from the left to the right side of the screen, the electronic device may decide to start playing the relevant sound source on the left speaker channel and then move to the right speaker channel to play it.

[0061] In operation 208, the electronic device forms a portion excluding at least one selected object as a fixed image and can form content by mapping a sound source based on a determined sound playback method. The electronic device can provide at least one selected object as a moving image in the content. According to one embodiment, the content may be cinemagraph content. According to one embodiment, the content may be a file format such as MP4.

[0062] The flowchart of FIG. 2 is merely one example, and each process of the flowchart of FIG. 2 may be omitted, merged, or changed in order depending on the example.

[0063] According to one embodiment, the electronic device may form metadata mapping an object with a relatively high selection rate (e.g., rain, dog, etc.), sound source information and sound playback method that is relatively frequently applied to it, and store it in memory (e.g., memory (130) of FIG. 1).

[0064] According to one embodiment, the electronic device can induce a selection from the user by providing the user with an example of content formed as a preview based on the metadata.

[0065] According to one embodiment, at least one of a source image, a sound library, a moving area selection algorithm, a moving object identification algorithm, an object attribute degree derivation algorithm, an object priority determination algorithm, and a sound playback method determination algorithm may be stored in the memory of an electronic device or stored in a cloud server, and the electronic device may access and use the cloud server.

[0067] Hereinafter, the operation of an electronic device according to one embodiment will be described with reference to FIGS. 5 and FIGS. 6.

[0068] FIG. 5 is a drawing (500) for explaining an example of an electronic device forming content according to one embodiment. FIG. 6 is a table (600) mapping the mood of a video to sound, used in the operation of an electronic device according to one embodiment. The operation of the electronic device described below can be performed in a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1).

[0069] Referring to FIG. 5, an electronic device (e.g., the electronic device (101) of FIG. 1) can identify a moving area within an image (500) stored in the memory of the electronic device (e.g., the memory (130) of FIG. 1). According to one embodiment, the electronic device can identify a rain-falling area as a first area (501) and a swaying bell hanging from an eaves area as a second area (502).

[0070] The electronic device can identify objects included in the identified first area (501) and second area (502). The electronic device can identify the objects in the first area (501) as “non-” and the objects in the second area (502) as “species”.

[0071] The electronic device can identify attribute information for the identified objects. The electronic device can derive attribute information for the object “B” of the first area (501) as shown in [Table 1].

[0072]

[0073] The electronic device can identify attribute information for the identified objects. The electronic device can derive attribute information for the “species” that is an object of the second area (502) as shown in [Table 2].

[0074]

[0075] The electronic device can determine the priority of objects based on the derived attribute information. Since the focus levels of 'Rain' and "Beast" are the same, the electronic device can determine that the priority of "Rain" is higher than that of "Beast" based on the ratio of screen area.

[0076] The electronic device can search for sound sources corresponding to objects. The electronic device can select a sound source from the sound library that has a tag value matching the attribute information of the objects. For the object “Rain,” the electronic device can select a sound source by considering the attribute information of “Rain,” namely “Main Object: Rain,” “Related Object: Roof,” and “Movement Intensity: Weak.”

[0077] If there is no sound source corresponding to a high-priority object, the electronic device may select a sound source for the next-highest priority object. For the object “Bell,” the electronic device may select a sound source by considering the attribute information of “Bell,” such as “Main Object: Bell” and “Movement Intensity: Weak.” According to one embodiment, the electronic device may select “Bell_Soft Strike.wma” as a sound source for the object “Bell.”

[0078] The electronic device can select a sound source for the surrounding environment and atmosphere. The electronic device can select a sound source from a sound library based on attribute information such as “Ambient Environment: Forest” and “Atmosphere: Refreshing.” At this time, the electronic device can use a table (600) that maps the atmosphere of FIG. 6 to the sound. Referring to FIG. 6, the electronic device can map attribute information such as “Main Object: Rain”, “Ambient Environment: Forest”, and “Atmosphere: Refreshing” to “Gentle bird sounds in the forest.” Based on “Gentle bird sounds in the forest,” the electronic device can select “Birds_Gentle.wma” from the sound library as a sound source.

[0079] The electronic device can select "bells" for which sound sources have been found, excluding "rain" for which no sound source has been found. When generating content, the electronic device can form the selected "bells" as moving images and "rain" as static images.

[0080] The electronic device can determine the sound playback method based on the attribute information of the “bell.” Since the “bell” occupies 1 / 30 of the entire screen area, the electronic device can determine the playback volume of the sound source corresponding to the “bell” to be 1 / 30 of the maximum playback volume. Additionally, in the case of “bird sounds” regarding the surrounding environment, the electronic device can determine the volume value of the bird sound to be at least 10 dB lower than the bell sound so that the sound source corresponding to the “bell” is not affected by the surrounding environment sound. Furthermore, since the “bell” is located at the bottom left of the screen, the electronic device can determine the playback position of the sound source for the “bell” to match this. Additionally, the electronic device can determine to play the sound source for the “bell” only during the timing when the “bell” moves intermittently. Additionally, the electronic device can determine to fade in the sound source when the movement of the “bell” begins and fade out the sound source when it ends.

[0081] The electronic device can form content by mapping selected sound sources based on a determined sound playback method, while forming the selected “bell” as a moving image and the “rain” as a static image.

[0083] Hereinafter, with reference to FIG. 7, the operation of an electronic device according to one embodiment will be described.

[0084] FIG. 7 is a drawing (700) for illustrating an example of an electronic device forming content according to one embodiment. The operation of the electronic device described below may be performed in a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1).

[0085] Referring to FIG. 7, an electronic device (e.g., the electronic device (101) of FIG. 1) can identify moving areas within an image (700) stored in the memory of the electronic device (e.g., the memory (130) of FIG. 1). According to one embodiment, the electronic device can identify a part where a firework explodes as a first area (701), a part where a bonfire is burning as a second area (702), and a part where people are moving as a third area (703).

[0086] The electronic device can identify objects included in the identified first area (701), second area (702), and third area (703). The electronic device can identify the object in the first area (701) as “fireworks,” the object in the second area (702) as “bonfires,” and the object in the third area (703) as “people.”

[0087] The electronic device can identify attribute information for the identified objects. The electronic device can derive attribute information for the object “firework” in the first area (701) as shown in [Table 3].

[0089]

[0091] The electronic device can identify attribute information for the identified objects. The electronic device can derive attribute information for the object “bonfire” in the second area (702) as shown in [Table 4].

[0092]

[0093] The electronic device can identify attribute information for the identified objects. The electronic device can derive attribute information for the “people” that are objects of the third area (703) as shown in [Table 5].

[0094]

[0096] The electronic device can determine the priority of objects based on the derived attribute information. Since the focus levels of “fireworks,” “bonfire,” and “people” are the same, the electronic device can determine the priority by assigning the order of highest priority to “fireworks,” “people,” and “bonfire” based on their screen area ratios.

[0097] The electronic device can search for sound sources corresponding to objects. The electronic device can select a sound source from a sound library that has a tag value matching the attribute information of the objects. For the object “Firework,” the electronic device can select a sound source by considering the attribute information of “Firework,” such as “Object Name: Firework” and “Movement Intensity: Strong.” According to one embodiment, the electronic device can select “Firework_Hard.wma” as a sound source for the object “Firework.”

[0098] The electronic device can select a sound source for the next-highest priority object once the selection of a sound source corresponding to a high-priority object is complete. For the object “People,” the electronic device can select a sound source by considering the attribute information of “People,” namely “Object Name: People” and “Movement Intensity: Weak.” If there is no corresponding sound source, the electronic device can proceed to the next step.

[0099] If there is no sound source corresponding to a high-priority object, the electronic device may select a sound source for the next-highest priority object. For the object “Bonfire,” the electronic device may select a sound source by considering the attribute information of “Bonfire,” such as “Object Name: Bonfire” and “Movement Strength: Weak.” According to one embodiment, the electronic device may select “Campfire_Soft.wma” as a sound source for the object “Bonfire.”

[0100] The electronic device can select a sound source for the surrounding environment and atmosphere. The electronic device can select a sound source from a sound library based on attribute information such as “surrounding environment: seaside” and “atmosphere: calm.” At this time, the electronic device can use a table (600) that maps the atmosphere of FIG. 6 to a sound. If there is no corresponding sound source, the electronic device can proceed to the next step.

[0101] The electronic device can select “fireworks” and “bonfires” for which sound sources were detected, excluding “people” for which sound sources were not detected. When generating content, the electronic device can form the selected “fireworks” and “bonfires” as moving images and “people” as static images.

[0102] The electronic device can determine the sound playback method based on the attribute information of the “fireworks.” Since the “fireworks” occupy half of the entire screen area, the electronic device can determine the playback volume of the sound source corresponding to the “fireworks” to be half the maximum playback volume. Additionally, the electronic device can determine to play the sound of the moving fireworks in a directional manner from bottom to top during the time the “fireworks” move from bottom to top of the screen. Additionally, the electronic device can determine to play the sound source for the “fireworks” only during the timing when the “fireworks” move intermittently (e.g., between 00:02 and 00:05 in the video). Additionally, the electronic device can determine to fade in the sound source when the movement of the “fireworks” begins and fade out when it ends. Furthermore, the electronic device can determine that the volume of the sound source is played at its highest level at the climax point where the “fireworks” explode (e.g., at the 00:04 point in the video).

[0103] The electronic device can determine the sound playback method based on the attribute information of the “bonfire.” Since the “bonfire” occupies 1 / 30 of the entire screen area, the electronic device can determine the playback volume of the sound source corresponding to the “bonfire” to be 1 / 30 of the maximum playback volume. Additionally, since the “bonfire” is located at the bottom of the screen, the electronic device can determine the playback position of the sound source for the “bonfire” to match this.

[0104] The electronic device can form content by mapping selected sound sources based on a determined sound playback method, while forming selected “fireworks” and “bonfires” into moving images and unselected “people” and other parts within the video into fixed images.

[0106] An electronic device according to one embodiment of the present disclosure includes a memory configured to store a first content including at least one moving object, and a processor connected to the memory. The memory may store one or more instructions such that, at execution, the processor identifies the at least one moving object in the first content, identifies attribute information of the at least one moving object, determines the priority of the at least one moving object based on the attribute information, searches for a sound source for the at least one moving object based on the priority and the attribute information, determines at least one of the at least one moving objects as a motion-providing object, determines a sound playback method for the determined motion-providing object, forms a portion of the first content excluding the determined motion-providing object as a fixed image, and stores the searched sound source based on the determined sound playback method to form a second content.

[0107] According to one embodiment of the present disclosure, the instructions may enable the processor to identify the attribute information including at least one of focus information, area information, position information, and color information of the identified at least one moving object.

[0108] According to one embodiment of the present disclosure, the instructions may enable the processor to identify focus information based on a focus degree determined inversely proportional to the blur degree of pixels corresponding to the identified at least one moving object, and to identify area information based on the ratio of the area occupied by the identified at least one moving object within the screen.

[0109] According to one embodiment of the present disclosure, the instructions may enable the processor to identify the position information based on the distance of the identified at least one moving object from the center of the first content, and to identify the color information based on at least one difference value among hue, brightness, saturation, or contrast by comparing the identified at least one moving object with an associated object in the first content.

[0110] According to one embodiment of the present disclosure, the instructions may enable the processor to determine the priority by sequentially considering focus information, area information, position information, and color information of the identified at least one moving object.

[0111] According to one embodiment of the present disclosure, the instructions may enable the processor to identify attribute information including at least one of associated object information, motion intensity information, motion direction information, sound playback timing information, and surrounding context information of the identified at least one moving object.

[0112] According to one embodiment of the present disclosure, the instructions may enable the processor to identify the motion direction information, which includes information regarding the direction of movement within the screen of the identified at least one moving object, and to determine the sound playback method such that the playback direction of the searched sound source matches the identified motion direction information.

[0113] According to one embodiment of the present disclosure, the instructions may enable the processor to identify sound playback timing information including information about a moving section within a screen of at least one identified moving object, and to determine the sound playback method such that the playback time of the searched sound source matches the sound playback timing information.

[0114] According to one embodiment of the present disclosure, the instructions may cause the processor to search for a sound source for the at least one moving object based on the attribute information in a sound library.

[0115] According to one embodiment of the present disclosure, the instructions may enable the processor to identify the portion of the first content where a change in pixels exists as the at least one moving object.

[0116] A method for forming content by an electronic device comprising a memory configured to store at least one moving object and a processor connected to the memory, according to one embodiment of the present disclosure, may identify the at least one moving object in the first content, identify attribute information of the at least one moving object, determine a priority of the at least one moving object based on the attribute information, search for a sound source for the at least one moving object based on the priority and the attribute information, determine at least one of the at least one moving objects as a motion-providing object, determine a sound playback method for the determined motion-providing object, form a portion of the first content excluding the determined motion-providing object as a fixed image, and store the searched sound source based on the determined sound playback method to form a second content.

[0117] According to one embodiment of the present disclosure, attribute information including at least one of focus information, area information, position information, and color information of at least one identified moving object can be identified.

[0118] According to one embodiment of the present disclosure, focus information can be identified based on a focus degree determined inversely proportional to the blur degree of pixels corresponding to at least one identified moving object, and area information can be identified based on the ratio of the area occupied by at least one identified moving object within the screen.

[0119] According to one embodiment of the present disclosure, the position information can be identified based on the distance of at least one identified moving object from the center of the first content, and the color information can be identified based on at least one difference value among hue, brightness, saturation, or contrast by comparing the at least one identified moving object with an associated object in the first content.

[0120] According to one embodiment of the present disclosure, the priority can be determined by sequentially considering focus information, area information, location information, and color information of at least one identified moving object.

[0121] According to one embodiment of the present disclosure, attribute information including at least one of associated object information, motion intensity information, motion direction information, sound playback timing information, and surrounding situation information of at least one identified moving object can be identified.

[0122] According to one embodiment of the present disclosure, the motion direction information including information on the direction of movement within a screen of at least one identified moving object can be identified, and the sound playback method can be determined such that the playback direction of the searched sound source matches the identified motion direction information.

[0123] According to one embodiment of the present disclosure, the sound playback timing information including information on a moving section within a screen of at least one identified moving object can be identified, and the sound playback method can be determined such that the playback time of the searched sound source matches the sound playback timing information.

[0124] According to one embodiment of the present disclosure, a sound source can be searched for for at least one moving object based on the attribute information in a sound library.

[0125] According to one embodiment of the present disclosure, a portion of the first content in which a change in pixels exists can be identified as the at least one moving object.

[0127] The electronic device according to the various embodiments disclosed in this document may be of various forms. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a consumer electronics device. The electronic device according to the embodiments of this document is not limited to the devices described above.

[0128] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may each include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as “first,” “second,” or “first” or “second” may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0129] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0130] Various embodiments of the present document may be implemented as software (e.g., program (140)) comprising one or more instructions stored in a storage medium (e.g., internal memory (136) or external memory (138)) readable by a machine (e.g., electronic device (101)). For example, a processor (e.g., processor (120)) of the machine (e.g., electronic device (101)) may call at least one of the one or more instructions stored in the storage medium and execute it. This enables the machine to be operated to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by the machine may be provided in the form of a non-transitory storage medium. Here, 'non-transient storage medium' simply means a tangible device that does not contain a signal (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily. For example, a 'non-transient storage medium' may include a buffer where data is stored temporarily.

[0131] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0132] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the components of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to the integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

Claim 1 In an electronic device, memory; a processor connected to said memory; The memory includes, at execution, one or more instructions for the processor to: identify a plurality of first objects having movement in a first content stored in the electronic device, determine a priority for the plurality of first objects based on attribute information of the plurality of first objects, determine at least one second object among the plurality of first objects to which movement is provided based on the priority, and create a second content in which movement is provided only to the at least one second object among the plurality of first objects and not to the remaining objects; the instructions for the processor to create the second content by searching for a first sound source corresponding to the at least one second object among sound sources stored inside or outside the electronic device, and mapping the searched first sound source, which is not a sound source included in the first content, to the at least one second object and storing the second content; and the instructions for the processor to create the second content by mapping the searched first sound source to the at least one second object, the second object's An electronic device that adjusts the volume of the first sound source in response to the intensity of movement and plays the first sound source in a playback section corresponding to the movement section of the second object. Claim 2 An electronic device according to claim 1, wherein the instructions enable the processor to identify attribute information including at least one of focus information, area information, location information, and color information for each of the identified plurality of first objects. Claim 3 An electronic device according to claim 2, wherein the instructions enable the processor to identify focus information based on a focus degree determined inversely proportional to the blur degree of pixels corresponding to each of the identified plurality of first objects, and to identify area information based on the ratio of the area occupied by each of the identified plurality of first objects within a screen. Claim 4 An electronic device according to claim 2, wherein the instructions enable the processor to identify the location information based on the distance of each of the identified plurality of first objects from the center of the first content, and to identify the color information based on at least one difference value among hue, brightness, saturation, or contrast by comparing each of the identified at least plurality of first objects with an associated object in the first content. Claim 5 An electronic device according to claim 2, wherein the instructions enable the processor to determine the priority by sequentially considering focus information, area information, location information, and color information for each of the identified plurality of first objects. Claim 6 An electronic device according to claim 1, wherein the instructions enable the processor to identify attribute information including at least one of associated object information, motion intensity information, motion direction information, and surrounding situation information for each of the identified plurality of first objects. Claim 7 An electronic device according to claim 1, wherein the instructions enable the processor to map the searched first sound source to the at least one second object, identify information regarding the in-screen moving direction of each of the identified plurality of first objects, and ensure that the playback direction of the first sound source matches the information regarding the identified moving direction. Claim 8 delete Claim 9 An electronic device according to claim 1, further comprising a display, wherein the instructions enable the processor to identify the plurality of first objects in the first content output through the display, and the first content comprises a video file not containing the sound source. Claim 10 delete Claim 11 A method of operating an electronic device comprises: identifying a plurality of first objects having movement in a first content stored in the electronic device; determining a priority for the plurality of first objects based on attribute information of the plurality of first objects; determining at least one second object among the plurality of first objects to which movement is provided based on the priority; and creating a second content in which movement is provided only to the at least one second object among the plurality of first objects and not to the remaining objects. The operation of creating the second content comprises: searching for a first sound source corresponding to the at least one second object among sound sources stored inside or outside the electronic device; and creating the second content by mapping the searched first sound source, which is not a sound source included in the first content, to the at least one second object. The operation of mapping the searched first sound source to the at least one second object comprises: adjusting the volume of the first sound source in response to the movement intensity of the second object; and playing the first sound source in a playback section corresponding to the movement section of the second object. method. Claim 12 delete Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images