Artificial intelligence-based foley sound providing apparatus and method

Abstract

An artificial intelligence-based foley sound providing apparatus and method are disclosed. The artificial intelligence-based foley sound providing method according to an embodiment comprises the steps of: obtaining a target image including a subject performing a target action; determining context information related to the target action by recognizing a property of at least one object including the subject from the target image; determining a duration of the target action and an intensity of the target action by tracking a posture of the subject from the target image; on the basis of the determined context information, duration, and intensity, generating a foley sound corresponding to the target action of the subject; and providing download data of the generated foley sound or download data in which the generated foley sound and the target image are merged.

Description

Artificial intelligence-based poly sound providing device and method

[0001] The disclosure below relates to a device and method for generating and providing foley sound based on artificial intelligence.

[0002] Foley sound is a sound effect mixed into a video to enhance audio quality or create a more realistic soundscape during video editing. It is created using a variety of props and equipment. While mixing foley sound is essential, creating and adding it to a video is often done manually.

[0003] Meanwhile, the recent development of various neural network models has enabled more sophisticated image analysis. A neural network model is a computer system that implements human-level intelligence, a model in which machines learn and make judgments on their own. A neural network model consists of machine learning (deep learning) technology that utilizes algorithms to classify and learn the characteristics of input data on their own, and component technologies that mimic the cognitive and judgmental functions of the human brain by utilizing machine learning algorithms. The component technologies may include, for example, at least one of: linguistic understanding technology that recognizes human language / characters; visual understanding technology that recognizes objects as if they were human vision; inference / prediction technology that judges information and logically infers and predicts; knowledge representation technology that processes human experience information into knowledge data; and motion control technology that controls the movements of autonomous vehicles and robots.

[0004] The background technology described above is technology that the inventor possessed or acquired in the process of deriving the disclosure of the present application, and cannot necessarily be said to be publicly known technology disclosed to the general public prior to the present application.

[0005] The disclosure below is intended to improve the problem of having to manually repeat all steps of the poly sound mixing work that is frequently performed in video work.

[0006] However, technical challenges are not limited to the technical challenges described above, and other technical challenges may exist.

[0007] An artificial intelligence-based poly sound providing method according to one embodiment may include the steps of: obtaining a target image including a subject performing a target action; recognizing a property of at least one object including the subject from the target image to determine context information related to the target action; tracking a posture of the subject from the target image to determine a duration and an intensity of the target action; generating a poly sound corresponding to the target action of the subject based on the determined context information, the duration and the intensity; and providing download data of the generated poly sound or download data in which the generated poly sound and the target image are merged.

[0008] The step of generating the poly sound may include the step of searching for a sound corresponding to the determined context information from a sound library, and, if a sound corresponding to the context information is searched, the step of adjusting the length of the searched sound based on the duration.

[0009] The step of generating the poly sound may further include, if a sound corresponding to the context information is not searched, a step of inputting the context information, the duration, and the intensity into a learned neural network model, and a step of generating a poly sound corresponding to the input context information, duration, and intensity as an output of the learned neural network model.

[0010] The target action may be a person's footsteps, and the properties of the at least one object may include the gender of the subject, the material of the ground on which the subject is walking, and the material of the shoes worn by the subject, and the context information may be determined in the form of text based on the gender of the recognized subject, the material of the recognized ground surface, and the material of the recognized shoes.

[0011] The target action is a person's footsteps, and the step of determining the duration of the target action and the intensity of the target action may include the step of detecting the posture of the subject from the target image, the step of tracking the detected posture to obtain the movement of the subject in the target image, and the step of determining the duration of the footsteps and the intensity of the footsteps based on the obtained movement.

[0012] The step of acquiring the movement of the subject may include, in response to the detection of the posture of the upper body of the subject, a step of acquiring the movement of the upper body by tracking the detected posture of the upper body, and a step of acquiring the movement of the lower body of the subject based on the movement of the upper body.

[0013] A method according to one embodiment further includes the steps of receiving an original image from a user device, and the step of inputting the original image into a learned neural network model to classify at least one continuous frame corresponding to the target action among the frames of the original image, wherein the step of obtaining the target image may obtain the classified at least one continuous frame as the target image.

[0014] The properties of each of the at least one object may be recognized through different neural network models trained to recognize the properties of each object from an image, and the context information may be text information that connects the outputs of the different neural network models.

[0015] According to one embodiment, the method further includes the steps of transmitting the generated poly sound to a user device, receiving a sound editing command for the poly sound from a user device that has received the poly sound, and editing the poly sound according to the sound editing command, wherein the step of providing download data may include the step of providing download data of the edited poly sound or download data of a target image into which the edited poly sound is merged.

[0016] According to one embodiment, a device for providing a poly sound includes a processor and a memory storing instructions to be executed by the processor, and when the instructions are executed by the processor, the processor may obtain a target image including a subject performing a target action, recognize a property of at least one object including the subject from the target image to determine context information related to the target action, track a posture of the subject from the target image to determine a duration of the target action and an intensity of the target action, generate a poly sound corresponding to the target action of the subject based on the determined context information, the duration, and the intensity, and provide download data of the generated poly sound or download data in which the generated poly sound and the target image are merged.

[0017] According to the artificial intelligence-based poly sound providing device and method according to an embodiment of the present disclosure, there is an effect of improving the problem of having to manually repeat all steps of poly sound mixing work frequently performed during video work.

[0018] FIG. 1 is a diagram illustrating an example of a network environment according to one embodiment.

[0019] FIG. 2 is a block diagram illustrating an example of a computer device according to one embodiment.

[0020] FIG. 3 is a diagram illustrating an example of a general appearance of a poly sound providing system according to one embodiment.

[0021] FIG. 4 is a block diagram illustrating an example of a poly sound providing device according to one embodiment.

[0022] Fig. 5 is a flowchart of a method for providing poly sound according to one embodiment.

[0023] FIG. 6 is a flowchart of a method for providing poly sounds of footsteps for an image of a full body or lower body of a person in one embodiment.

[0024] FIG. 7 is a flowchart of a method for providing poly sounds of footsteps for a video of a person's upper body in one embodiment.

[0025] FIG. 8 is a diagram illustrating an example of a user screen of a poly sound providing method according to one embodiment.

[0026] Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only and may be modified and implemented in various forms. Therefore, the actual implementation is not limited to the specific embodiments disclosed, and the scope of this specification includes modifications, equivalents, or alternatives within the technical concepts described in the embodiments.

[0027] Although terms such as "first" or "second" may be used to describe various components, these terms should be interpreted solely to distinguish one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0028] When it is said that a component is "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but there may also be other components in between.

[0029] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this document, phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can each include any one of the items listed together in that phrase, or all possible combinations thereof. In this specification, it should be understood that the terms "comprises" or "has" and the like are intended to specify the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not exclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0030] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0031] Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the attached drawings, identical components are assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0032] FIG. 1 is a diagram illustrating an example of a network environment according to one embodiment. The network environment of FIG. 1 represents an example including a plurality of electronic devices (110, 120, 130), a plurality of servers (150, 160), and a network (140). FIG. 1 is an example for explaining the invention, and the number of electronic devices or servers is not limited to that of FIG. 1. In addition, the network environment of FIG. 1 merely illustrates one example of environments applicable to the present embodiments, and the environments applicable to the present embodiments are not limited to the network environment of FIG. 1.

[0033] The plurality of electronic devices (110, 120, 130) may be fixed terminals or mobile terminals implemented as computer devices. Examples of the plurality of electronic devices (110, 120, 130) include smart phones, mobile phones, navigation systems, computers, laptops, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablet PCs, etc. For example, in FIG. 1, the shape of a smart phone is shown as an example of the electronic device (110), but in embodiments of the present invention, the electronic device (110) may actually mean one of various physical computer devices that can communicate with the server (150, 160) via a network (140) using a wireless or wired communication method.

[0034] The communication method is not limited, and may include not only a communication method that utilizes a communication network (e.g., a mobile communication network, a wired Internet, a wireless Internet, a broadcasting network) that the network (140) may include, but also short-range wireless communication between devices. For example, the network (140) may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, and the like. In addition, the network (140) may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree, or a hierarchical network.

[0035] Each of the servers (150, 160) may be implemented as a computer device or multiple computer devices that communicate with multiple electronic devices (110, 120, 130) via a network (140) to provide commands, codes, files, contents, services, etc. For example, the server (150) may be a system that provides services (for example, cloud services, game services, content provision services, group call services (or voice conference services), messaging services, mail services, social network services, map services, translation services, financial services, payment services, search services, etc.) to multiple electronic devices (110, 120, 130) connected via a network (140).

[0036] FIG. 2 is a block diagram illustrating an example of a computer device according to one embodiment. Each of the plurality of electronic devices (110, 120, 130) or servers (150, 160) described above may be implemented by the computer device (200) illustrated in FIG. 2.

[0037] Such a computer device (200) may include a memory (210), at least one processor (220), a communication interface (230), and an input / output interface (240), as illustrated in FIG. 2. The memory (210) may be a computer-readable recording medium, and may include a random access memory (RAM), a read only memory (ROM), and a non-permanent mass storage device such as a disk drive. Here, the non-permanent mass storage device such as a ROM and a disk drive may be included in the computer device (200) as a separate permanent storage device distinct from the memory (210). In addition, an operating system and at least one program code may be stored in the memory (210). These software components may be loaded into the memory (210) from a computer-readable recording medium separate from the memory (210). Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory (210) via a communication interface (230) other than a computer-readable recording medium. For example, the software components may be loaded into the memory (210) of the computer device (200) based on a computer program installed by files received over a network (140).

[0038] The processor (220) may be configured to process commands (or instructions) of a computer program by performing basic arithmetic, logic, and input / output operations. The commands may be provided to the processor (220) via the memory (210) or the communication interface (230). For example, the processor (220) may be configured to execute commands received according to program code stored in a storage device such as the memory (210).

[0039] The communication interface (230) may provide a function for the computer device (200) to communicate with other devices (e.g., the storage devices described above) via the network (140). For example, requests, commands, data, files, etc. generated by the processor (220) of the computer device (200) according to program codes stored in a recording device such as the memory (210) may be transmitted to other devices via the network (140) under the control of the communication interface (230). Conversely, signals, commands, data, files, etc. from other devices may be received by the computer device (200) via the communication interface (230) of the computer device (200) via the network (140). Signals, commands, data, etc. received via the communication interface (230) may be transmitted to the processor (220) or the memory (210), and files, etc. may be stored in a storage medium (the permanent storage device described above) that the computer device (200) may further include.

[0040] The input / output interface (240) may be a means for interfacing with an input / output device (250). For example, the input device may include a device such as a microphone, keyboard, or mouse, and the output device may include a device such as a display or speaker. As another example, the input / output interface (240) may be a means for interfacing with a device that integrates input and output functions, such as a touchscreen. The input / output device (250) may also be configured as a single device with the computer device (200).

[0041] Additionally, in other embodiments, the computer device (200) may include fewer or more components than those illustrated in FIG. 2. However, it is not necessary to explicitly illustrate most conventional components. For example, the computer device (200) may be implemented to include at least some of the input / output devices (250) described above, or may further include other components such as a transceiver, a database, etc.

[0042] FIG. 3 is a diagram illustrating an example of a general appearance of a poly sound providing system according to one embodiment.

[0043] Foley sound is a sound effect mixed into a video to enhance audio quality or create a more realistic soundscape during video editing. It is created using a variety of props and equipment. While mixing foley sound is essential, creating and adding it to a video is often done manually.

[0044] Through a poly sound providing device (320) and method according to one embodiment, poly sound suitable for an image can be generated and provided, thereby reducing the time and cost required for poly sound recording and mixing.

[0045] Referring to FIG. 3, a poly sound providing system (300) may include a user device (310) (e.g., electronic devices (110, 120, 130)) and a poly sound providing device (320) (e.g., servers (150, 160)). Here, the user device (310) and the poly sound providing device (320) may each be implemented by at least one computer device (200).

[0046] A user can input an image to which he or she wishes to add poly sound through a user device (310). The image input through the user device (310) can be uploaded to a poly sound providing device (320) through a network.

[0047] The poly sound providing device (320) that receives the uploaded video can generate poly sound based on artificial intelligence. Specifically, the poly sound providing device (320) can analyze the video using at least one neural network model and generate poly sound corresponding to the video based on the analysis results.

[0048] A neural network model can refer to a neural network that includes multiple hidden layers in addition to input and output layers. Using a neural network model, one can identify latent structures in data. That is, one can identify the latent structure of an image (e.g., which objects are in the image, the pose of the objects, etc.). Neural network models can include convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders, generative adversarial networks (GANs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), Q-networks, U-networks, Siamese networks, and generative adversarial networks (GANs).

[0049] According to one embodiment, a neural network model may be trained using at least one of supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Training a neural network model may be a process of applying knowledge to the neural network to enable the neural network model to perform a specific operation. Hereinafter, the term "model" may refer to a neural network model.

[0050] For example, referring to FIG. 4, an exemplary block diagram of a poly sound providing device (320) according to one embodiment is illustrated. The poly sound providing device (320) may include a neural network processor (405), an editor (410), and a sound library (415). In one embodiment, the neural network processor (405) and the editor (410) may each be implemented by at least one computer device (200). Alternatively, the poly sound providing device (320) may be configured by one computer device (200), and the functions of the neural network processor (405) and the editor (410) may be performed by the processor (220) of the computer device (200).

[0051] The poly sound providing device (320) can generate poly sound corresponding to an input image using a neural network processor (405), an editor (410), and a sound library (415). The functions of the neural network processor (405), the editor (410), and the sound library (415) are described in detail with reference to the block diagram in FIG. 5 and below.

[0052] The poly sound providing device (320) can provide download data so that the user can download the poly sound through the user device (310), and the user can download the poly sound.

[0053] Fig. 5 is a flowchart of a method for providing poly sound according to one embodiment.

[0054] Referring to FIG. 5, the poly sound providing device (320) may, at step (505), acquire a target image including a subject performing a target action. The target action may be predetermined. For example, the target action may be a person's footsteps. The sound of footsteps may be added to the target image as a poly sound corresponding to the footsteps in the target image.

[0055] The target video may be a video uploaded by a user, or a video classified as corresponding to a target action by a poly sound providing device (320) among the videos uploaded by the user.

[0056] In one embodiment, the target image may be created by cutting out a portion of the original image other than the portion to which the user wishes to add footstep sounds, and uploading the created target image to the poly sound providing device (320) via the user device (310). The poly sound providing device (320) may obtain the target image by receiving the target image uploaded by the user.

[0057] In another example, a user may upload an original video to the poly sound providing device (320). The target video may be a video corresponding to a frame in the original video that the poly sound providing device (320) classifies as corresponding to a target action.

[0058] For example, the neural network processor (405) of the poly sound providing device (320) may include a neural network model (e.g., an action classification model (not shown)) trained to classify at least one consecutive frame corresponding to a target action from an input image.

[0059] A neural network processor (405) can input an original image into a learned behavior classification model (not shown) to classify at least one continuous frame corresponding to a target behavior. A poly sound providing device (320) can acquire at least one continuous frame classified as corresponding to a target behavior as a target image.

[0060] The polyphonic sound providing device (320) can, in step (510), determine contextual information related to a target action from the target image. The contextual information may include information about the surrounding environment of the subject related to the target action. For example, the contextual information may include the properties of at least one object including the subject.

[0061] A poly sound providing device (320) can determine contextual information related to a target action by recognizing the properties of at least one object including a subject from a target image. The contextual information may be information expressing the properties of at least one object including the subject in text form.

[0062] The properties of an object included in a target image can be acquired using a neural network model. For example, the properties of an object included in a target image can be acquired using an image recognition model (411 to 413) included in a neural network processor (405).

[0063] Referring to FIG. 4, the neural network processor (405) may include at least one image recognition model (411 to 413) trained to recognize the properties of a specific object from an input image. Each image recognition model (411 to 413) may be a model trained to recognize the properties of different objects.

[0064] The target image acquired in step (510) can be input to at least one image recognition model (411 to 413) through a neural network processor (405). The neural network processor (405) can output the properties of at least one different object included in the target image using at least one image recognition model (411 to 413).

[0065] The poly sound providing device (320) can determine context information related to a target action using the properties of the generated object, and can provide poly sound corresponding to the situation of the target image using the determined context information.

[0066] Although not shown in the drawing, in another embodiment of the present invention, the poly sound providing device (320) may receive context information about the properties of at least one object including the subject of the target image from the user device.

[0067] Contextual information received from the user device may include information about the subject or the characteristics of surrounding objects that influence the Foley sound. For example, if the Foley sound is the sound of footsteps, information about the subject may include, but is not limited to, the subject's gender, age, weight, and clothing. Information about the characteristics of surrounding objects may include, but is not limited to, the type of shoe, the material of the shoe sole, the material of the ground the shoe touches, the space the subject is located in, and ambient noise.

[0068] The poly sound providing device (320) can automatically recognize the properties of at least one object including a subject using a neural network model, but the properties of the recognized object may not include sufficient information for poly sound generation.

[0069] This could be an example of an object property required for polyphonic sound generation being located outside the target video screen, or being included within the screen due to low screen resolution, but not yet recognized by the neural network model. Alternatively, to enhance viewer immersion, this could be an example of a case where the user wishes to artificially input the object property used in polyphonic sound generation.

[0070] In one embodiment, the user device may display a user interface on the screen of the user device that allows the user to input the properties of an object. The user interface for inputting the properties of an object may be provided by displaying a blank space for the user to input a desired word or sentence.

[0071] The user device can transmit a word or sentence input by the user to the poly sound providing device (320). The poly sound providing device (320) can generate a poly sound based on context information corresponding to the word or sentence input by the user.

[0072] In another embodiment, the user device may present a plurality of object properties with high input frequency as candidates for the user to select, and display the properties of the desired object to the user.

[0073] In another embodiment, the user device may primarily provide candidates for selection of highly relevant object properties based on the properties of the object recognized through the image recognition model. That is, the poly sound providing device (320) may determine, as candidates, properties of objects that are highly relevant to the properties of the object recognized by the image recognition model.

[0074] For example, if the objects recognized by the image recognition model include shoes and the ground, the poly sound providing device (320) can determine that the type of poly sound is the sound of footsteps. The poly sound providing device (320) can provide the user device with information about the material and area of ​​the sole of the shoe that affect the sound of footsteps, the material of the shoes worn by the subject, and spatial information about the location of the ground as candidate object properties that affect the sound of footsteps.

[0075] The user device can display the properties of the candidate object to the user through a user interface. The user device can transmit the properties of the object selected by the user to the poly sound providing device (320). The poly sound providing device (320) can generate poly sounds based on context information corresponding to the properties of the object selected by the user.

[0076] In step (515), the poly sound providing device (320) can track the subject's posture from the acquired target image to determine the duration and intensity of the target behavior. The poly sound providing device (320) can determine the duration and intensity of the target behavior using the neural network processor (405).

[0077] The pose of a subject included in a target image can be detected using a neural network model. For example, the neural network processor (405) may include a pose detection model (420) trained to detect the pose of a subject (e.g., a person) from the acquired target image.

[0078] For example, the pose detection model (420) may be a skeleton detection model that recognizes joints of the human body. The poly sound providing device (320) can track the pose of the subject based on the results of pose detection for each frame of the target image, thereby obtaining the movement (or motion) of the subject in the target image.

[0079] The subject's pose can be tracked using various algorithms or neural network models. For example, the subject's pose can be tracked using a Kalman filter-based pose tracking algorithm based on the pose detected by the pose detection model (420). For example, the subject's pose can be tracked using a neural network model trained to track continuous poses based on the pose detected by the pose detection model (420).

[0080] In an embodiment where the target image acquired in step (505) includes a target action of the subject, the neural network processor (405) can determine the duration and intensity of the target action based on the acquired movement of the subject.

[0081] For example, when a target image including the full body or lower body of a walking person is input to a neural network processor (405) and the target action is the person's footsteps, the neural network processor (405) detects the posture of the full body or lower body of the walking person through a posture detection model (420), and based on the result of the posture detection, tracks the posture of the full body or lower body of the person to obtain the movement of the full body or lower body.

[0082] The neural network processor (405) can determine the duration and intensity of a step based on the acquired subject's movements. The neural network processor (405) can identify the start and end points of a target action in the target image based on the subject's movements, and determine the difference between the two points in time as the duration of the target action. The neural network processor (405) can determine the intensity of the target action based on the subject's moving speed.

[0083] In some embodiments, the neural network processor (405) may further include a neural network model (e.g., a motion prediction model (435) of FIG. 4) that predicts motion of the subject not included in the target image based on the motion of the subject acquired from the target image. For example, in an embodiment where the target image acquired in step (505) does not include the target action of the subject, the motion prediction model (435) may further be included in the neural network processor (405) to predict motion corresponding to the target action of the subject.

[0084] For example, if a target image including the upper body of a walking person is input to a neural network processor (405) and the target action is a person's footsteps, the neural network processor (405) can detect the posture of the upper body of the walking person through a posture detection model (420) and track the posture of the upper body of the person based on the result of the posture detection to obtain the movement of the upper body, but may not be able to obtain the movement of the lower body of the person.

[0085] In this case, the neural network processor (405) can predict the movement of the lower body from the acquired movement of the upper body using a motion prediction model (435) trained to predict the movement of the lower body based on the movement of the upper body. The poly sound providing device (320) can determine the duration and intensity of the step based on the acquired movement of the lower body.

[0086] In step (520), the poly sound providing device (320) can generate a poly sound corresponding to the target behavior of the subject based on the context determined in step (510) and the duration and intensity of the target behavior determined in step (515).

[0087] Referring to FIG. 4, the poly sound providing device (320) may include a sound library (415) in which various sounds related to target actions are stored by mapping them with text expressing the sounds.

[0088] In one embodiment, the poly sound providing device (320) can retrieve a poly sound corresponding to a target action of a subject from a sound library (415) based on at least one of context, duration of the target action, and intensity of the target action.

[0089] For example, the poly sound providing device (320) can retrieve a poly sound corresponding to at least one of a context, a duration of a target action, and an intensity of a target action from a sound library (415).

[0090] In another example, the poly sound providing device (320) can retrieve a poly sound using a neural network model (e.g., the retrieval model (430) of FIG. 4) trained to output one of the sounds stored in the sound library (415) based on at least one of the context, the duration of the target action, and the intensity of the target action.

[0091] Referring to FIG. 4, the neural network processor (405) may include a search model (430) trained to output any one of the sounds stored in the sound library (415) based on at least one of the context, the duration of the target action, and the intensity of the target action.

[0092] For example, at least one of context, duration of target behavior, and intensity of target behavior can be input to a search model (430) of a neural network processor (405). The neural network processor (405) can output any one of the sounds stored in the sound library (415) using the search model (430).

[0093] The poly sound providing device (320) can generate a poly sound corresponding to a target action by adjusting the length of the output sound by considering the duration of the target action using a poly sound or search model (430) searched from a sound library (415).

[0094] Referring to FIG. 4, the poly sound providing device (320) may include an editor (410) capable of editing sounds. The editor (410) may perform an editing function of an audio track.

[0095] In an embodiment of the present invention, the editor (410) can perform synchronization adjustments so that the generated poly sound corresponds to the screen within the video. Specifically, the editor (410) can adjust the position of the audio track, cut or paste a specific point, and perform sneak in and out functions.

[0096] In one embodiment, the poly sound providing device (320) can generate poly sounds corresponding to target actions using a neural network model in parallel with generating poly sounds by searching for sounds in a sound library (415).

[0097] For example, the poly sound providing device (320) can generate a poly sound corresponding to a target action using a poly sound generation model (425) trained to generate a poly sound based on at least one of a context, a duration of a target action, and an intensity of the target action.

[0098] Referring to FIG. 4, the neural network processor (405) may include a poly sound generation model (425) trained to generate a poly sound based on at least one of a context, a duration of a target action, and an intensity of the target action. The context determined in step (510) and at least one of the duration of the target action and the intensity of the target action determined in step (515) may be input to the poly sound generation model (425). The neural network processor (405) may generate a poly sound corresponding to the target action using the poly sound generation model (425).

[0099] In step (525), the poly sound providing device (320) may provide download data of the generated poly sound or download data in which the generated poly sound and the target image are merged to the user device. The merging of the generated poly sound and the target image may be performed by an editor (e.g., the editor (410) of FIG. 4). The editor (410) may perform an operation to match the generated poly sound to the screen in the image through at least one of position adjustment, cutting or pasting, and sneak in and out functions in the audio track. The user may download the generated poly sound through the user device.

[0100] Hereinafter, with reference to FIGS. 6 and 7, an exemplary method for providing poly sounds when the target action is a person's footsteps will be described.

[0101] A video of a walking person may include the full body, upper body, or lower body of the walking person. If the full body of the person or the lower body of the person is included in the target video, the poly sound providing device (320) can track the person's movements from the target video and provide poly sounds corresponding to the person's footsteps.

[0102] Referring to FIG. 6, a flowchart of a method for providing a poly sound of a footstep, which is a target action, for a target image in which a full body or lower body of a person is captured in one embodiment is illustrated.

[0103] In one embodiment, the poly sound providing device (320) can obtain a target image including the lower body or the whole body of a walking subject (i.e., a person) in step (605).

[0104] The target video may be a video uploaded by a user, or a video classified as corresponding to a target action by a poly sound providing device (320) among the videos uploaded by the user.

[0105] In one embodiment, the target image may be a target image created by the user by cutting out a portion of the original image including the lower body or the entire body of a walking person, and uploading the created target image to the poly sound providing device (320) via the user device (310). The poly sound providing device (320) may obtain the target image by receiving the target image uploaded by the user.

[0106] In another example, a user may upload an original video to the poly sound providing device (320). The target video may be an video corresponding to a frame classified by the poly sound providing device (320) as containing the lower body or the full body of a walking person from the original video. For example, the neural network processor (405) of the poly sound providing device (320) may include a neural network model (e.g., an action classification model (not shown)) trained to classify at least one consecutive frame containing the lower body or the full body of a walking person from the input video.

[0107] A neural network processor (405) can input an original image into a learned behavior classification model (not shown) to classify at least one continuous frame corresponding to a target behavior. A poly sound providing device (320) can acquire at least one continuous frame classified as corresponding to a target behavior as a target image.

[0108] In the example of FIG. 6, step (605) may be a step included in step (505) of FIG. 5.

[0109] The poly sound providing device (320) can recognize the properties of at least one object including the subject in steps (610), (615), and (620). The at least one object may be an object related to the target action, i.e., footsteps. The at least one object related to the footsteps may be predetermined. In the example of FIG. 6, the at least one object may include the subject, i.e., the person, the ground on which the person is walking, and the shoes the person is wearing. However, this is merely an example, and the at least one object related to the target action may include various objects.

[0110] At least one property of each object can be acquired using different neural network models (e.g., the first image recognition model (411) to the Nth image recognition model (413) of FIG. 4). For example, the poly sound providing device (320) can input, in step (610), the input image into a first image recognition model trained to recognize the gender of a person, in step (615), the input image into a second image recognition model trained to recognize the material of the ground, and in step (620), the input image into a third image recognition model trained to recognize the material of a shoe.

[0111] The poly sound providing device (320) can obtain the gender of a person as an output of the first image recognition model, obtain the material of the ground as an output of the second image recognition model, and obtain the material of a shoe as an output of the third image recognition model.

[0112] In step (625), the poly sound providing device (320) can determine a context related to the target action, that is, footsteps, based on the properties of at least one object obtained in steps (610), (615), and (620).

[0113] In one embodiment, context information may be determined in text format based on the properties of at least one object. For example, the context information may be information expressing the properties of at least one object in text. For example, if a person's gender is recognized as male through multiple image recognition models in step (610), concrete is recognized as the ground material in step (615), and leather is recognized as the shoe material in step (620), the context information may be text information of "male-concrete-leather" that connects the outputs of the image recognition models. In the example of FIG. 6, steps (610), (615), (620), and (625) may be steps included in step (510) of FIG. 5.

[0114] In step (630), the poly sound providing device (320) can track the posture of the subject from the acquired target image to determine the duration and intensity of the target action. In one embodiment, the poly sound providing device (320) can determine the duration and intensity of the footsteps, which are the target actions, by using a posture detection model (e.g., the posture detection model (420) of FIG. 4) trained to recognize footsteps from an image. For example, the poly sound providing device (320) can input the acquired target image into the posture detection model to detect and track the posture of a person's lower body or full body.

[0115] The poly sound providing device (320) can determine the duration and intensity of footsteps by tracking the subject's actions based on the results of posture detection in response to the detection of the posture of the lower body or the entire body of a person. In the example of FIG. 6, step (630) may be included in step (515) of FIG. 5.

[0116] In step (635), the poly sound providing device (320) can search for a poly sound corresponding to the subject's footsteps in a sound library (e.g., the sound library (415) of FIG. 4) based on at least one of the context determined in step (625) and the duration and intensity of the footsteps determined in step (630). For example, the poly sound providing device (320) can search for a poly sound corresponding to at least one of the context, the duration of the footsteps, and the intensity of the footsteps from the sound library.

[0117] In another example, the poly sound providing device (320) can search for poly sounds using a search model (e.g., search model (430) of FIG. 4) trained to output any one of the sounds stored in the sound library based on at least one of context, duration of footsteps, and intensity of footsteps.

[0118] In step (640), the poly sound providing device (320) can generate a poly sound corresponding to the footsteps of the subject by adjusting the length of the poly sound searched in step (635) in consideration of the duration determined in step (630).

[0119] In step (645), the poly sound providing device (320) can generate poly sounds corresponding to the footsteps of the subject using a poly sound generation model (e.g., poly sound generation model (425) of FIG. 4) trained to generate poly sounds based on at least one of context, duration of footsteps, and intensity of footsteps, in parallel with steps (635) to (640). In the example of FIG. 6, steps (635), (640), and (645) can be included in step (520) of FIG. 5.

[0120] Although not illustrated in FIG. 6, a method for providing poly sound according to one embodiment may further include a step of adjusting the sync between the target image and the poly sound by using an editor (e.g., the editor (410) of FIG. 4) prior to step (650) by the poly sound providing device. In the step of adjusting the sync, the poly sound providing device may generate a single poly sound by combining the poly sound generated through steps (635) to (640) and the poly sound generated through step (645).

[0121] In FIG. 6, steps (635) to (640) are illustrated as being performed in parallel with step (645), but in another embodiment, step (635) may be performed first, and a step (not shown) may further be included for determining whether to perform step (645) based on whether the result of step (635) satisfies a preset condition. For example, in the step (not shown) for determining whether to perform step (645), the poly sound providing device may determine whether the poly sound retrieved in step (635) is appropriate. For example, the poly sound providing device may determine that the poly sound is appropriate if the difference between the length of the retrieved poly sound and the duration of the footstep is less than or equal to a preset value, and may determine that the poly sound is not appropriate if it exceeds the preset value.

[0122] For example, if the duration of the searched poly sound is 1 second and the duration of the subject's footsteps is 3 seconds, increasing the poly sound from 1 second to 3 seconds in step (640) may degrade the quality of the poly sound, so the poly sound providing device may determine that the searched poly sound is not appropriate. Step (645) may be performed if the searched poly sound is determined to be not appropriate. Step (640) may be performed if the searched poly sound is determined to be appropriate.

[0123] Again, in step (650) of FIG. 6, the poly sound providing device (320) may provide download data of the poly sound generated in at least one of steps (640) and (645) to the user device (310). In one embodiment, the poly sound providing device (320) may provide download data of the generated poly sound or download data of the generated poly sound merged with the target image based on user input. The merging of the generated poly sound with the target image may be performed by an editor (e.g., the editor (410) of FIG. 4). In the example of FIG. 6, step (650) may be included in step (525) of FIG. 5.

[0124] Referring to FIG. 7, unlike the case of FIG. 6, a flowchart of a method for providing a poly sound of footsteps, which is a target action, for a target image in which the upper body of a person is captured is illustrated.

[0125] In one embodiment, the poly sound providing device (320) can obtain a target image including the upper body of a walking subject (i.e., a person) in step (705).

[0126] The target video may be a video uploaded by a user, or a video classified as corresponding to a target action by a poly sound providing device (320) among the videos uploaded by the user.

[0127] In one embodiment, the target image may be a target image created by the user by cutting out a portion of the original image including the upper body of a walking person, and uploading the created target image to the poly sound providing device (320) via the user device (310). The poly sound providing device (320) may obtain the target image by receiving the target image uploaded by the user.

[0128] In another example, a user may upload an original video to the poly sound providing device (320). The target video may be an video corresponding to a frame classified by the poly sound providing device (320) as containing the upper body of a walking person from among the original video. For example, the neural network processor (405) of the poly sound providing device (320) may include a neural network model (e.g., an action classification model (not shown)) trained to classify at least one consecutive frame containing the upper body of a walking person from the input video.

[0129] A neural network processor (405) can input an original image into a learned behavior classification model (not shown) to classify at least one continuous frame corresponding to a target behavior. A poly sound providing device (320) can acquire at least one continuous frame classified as corresponding to a target behavior as a target image.

[0130] In the example of FIG. 7, step (705) may be a step included in step (505) of FIG. 5.

[0131] The poly sound providing device (320) can, in step (710), recognize the properties of at least one object including the subject. The at least one object may be an object related to the target action, i.e., footsteps. The at least one object related to the footsteps may be predetermined. In the example of FIG. 7, the at least one object may be the subject. However, this is merely an example, and the at least one object related to the target action may include various objects.

[0132] The properties of at least one object can be obtained using different neural network models (e.g., the first image recognition model (411) to the Nth image recognition model (413) of FIG. 4). For example, the poly sound providing device (320) can, in step (710), input the input image into the first image recognition model trained to recognize the gender of a person. The poly sound providing device (320) can obtain the gender of the person as an output of the first image recognition model.

[0133] In step (715), the poly sound providing device (320) may determine a context related to the target action, i.e., footsteps, based on the properties of the object acquired in step (710) (i.e., the person's gender). In one embodiment, the context information may be determined in text form based on the properties of at least one object. For example, the context information may be information expressing the properties of at least one object in text form. For example, if the person's gender is recognized as male in step (710), the context information may be "male." However, the person's gender is only one example, and context information expressing the properties of any object that affects the poly sound may be included. In the example of FIG. 7, steps (710) and (715) may be steps included in step (510) of FIG. 5.

[0134] Although not shown in the drawing, the poly sound providing device (320) can receive context information about the properties of at least one object including the subject of the target image from the user device.

[0135] Sometimes, the context information required for generating polyphonic sounds is located outside the target image's screen, making it difficult to generate polyphonic sounds that accurately correspond to the target image. As in the example illustrated in Fig. 7, when generating footstep sounds, only the subject's upper body may be displayed on the screen. Therefore, there may be limitations in predicting information about the subject from the movement of the upper body, or in predicting the object's properties from the surrounding environment.

[0136] Accordingly, the poly sound providing device (320) can generate poly sounds based on contextual information about the properties of an object directly input by the user. To this end, the poly sound providing device (320) can provide a user interface that can receive contextual information through a user device. The user can input contextual information about the properties of the object. The user device can transmit the input contextual information to the poly sound providing device (320).

[0137] In an embodiment of the present invention, the poly sound providing device (320) can receive contextual information about the material of shoes and the material of the ground from the user device. The poly sound providing device (320) can generate footstep sounds based on the contextual information about the material of shoes and the material of the ground directly input by the user, along with the contextual information about the gender of the person determined through the first image recognition model described above.

[0138] In another embodiment, the polyphonic sound providing device (320) may predict the properties of objects that affect footstep sounds but are not displayed on the screen, based on information recognized about objects within the target image. The properties of the predicted objects may be provided to the user in the form of contextual information expressed in text.

[0139] Although not displayed on the screen, the user can select context information appropriate for the target image among context information that affects the poly sound. The poly sound providing device (320) can receive the context information selected by the user and generate poly sound based on the context information.

[0140] In step (720), the poly sound providing device (320) can track the posture of the upper body of the subject from the target image and predict the movement of the lower body of the subject. For example, the poly sound providing device (320) can detect the posture of the upper body of the subject from the target image through a posture detection model (e.g., the posture detection model (420) of FIG. 4). In response to the detection of the posture of the upper body of the subject, the poly sound providing device (320) can track the posture of the upper body based on the result of the posture detection and obtain the movement of the upper body. The poly sound providing device (320) can obtain the movement of the lower body from the obtained movement of the upper body by using a neural network model (e.g., the motion prediction model (435) of FIG. 4) trained to predict the movement of the lower body based on the movement of the upper body.

[0141] In step (725), the poly sound providing device (320) can determine the duration and intensity of the target action, i.e., footsteps, based on the movement of the lower body predicted in step (720). For example, the poly sound providing device (320) can identify the start and end points of footsteps in the target image based on the movement of the lower body, and determine the difference between the two points in time as the duration of the footsteps. The poly sound providing device (320) can determine the intensity of the footsteps based on the speed at which the subject moves. In the example of FIG. 7, steps (720) and (725) may be steps included in step (515) of FIG. 5.

[0142] In step (730), the poly sound providing device (320) can search for a poly sound corresponding to the subject's footsteps in a sound library (e.g., the sound library (415) of FIG. 4) based on at least one of the context determined in step (715) and the duration and intensity of the footsteps determined in step (725). For example, the poly sound providing device (320) can search for a poly sound corresponding to at least one of the context, the duration of the footsteps, and the intensity of the footsteps from the sound library.

[0143] In another example, the poly sound providing device (320) can search for poly sounds using a search model (e.g., search model (430) of FIG. 4) trained to output any one of the sounds stored in the sound library based on at least one of context, duration of footsteps, and intensity of footsteps.

[0144] In step (735), the poly sound providing device (320) can generate a poly sound corresponding to the footsteps of the subject by adjusting the length of the poly sound searched in step (730) in consideration of the duration determined in step (725).

[0145] In step (740), the poly sound providing device (320) can generate poly sounds corresponding to the footsteps of the subject using a poly sound generation model (e.g., poly sound generation model (425) of FIG. 4) trained to generate poly sounds based on at least one of context, duration of footsteps, and intensity of footsteps, in parallel with steps (730) to (735). In the example of FIG. 7, steps (730), (735), and (740) can be included in step (520) of FIG. 5.

[0146] Although not illustrated in FIG. 7, a method for providing poly sound according to one embodiment may further include a step of adjusting the sync between the target image and the poly sound by using an editor (e.g., the editor (410) of FIG. 4) prior to step (745) by the poly sound providing device. In the step of adjusting the sync, the poly sound providing device may generate a single poly sound by combining the poly sound generated through steps (730) to (735) and the poly sound generated through step (740).

[0147] In FIG. 7, steps (730) to (735) are illustrated as being performed in parallel with step (740), but in another embodiment, step (730) may be performed first, and a step (not shown) may further be included for determining whether to perform step (740) based on whether the result of step (730) satisfies a preset condition. For example, in the step (not shown) for determining whether to perform step (740), the poly sound providing device may determine whether the poly sound retrieved in step (730) is appropriate. For example, the poly sound providing device may determine that the poly sound is appropriate if the difference between the length of the retrieved poly sound and the duration of the footstep is less than or equal to a preset value, and may determine that the poly sound is not appropriate if it exceeds the preset value. For example, if the duration of the searched poly sound is 1 second and the duration of the subject's footsteps is 3 seconds, increasing the poly sound of 1 second to 3 seconds in step (735) may result in a deterioration in the quality of the poly sound, and thus the poly sound providing device may determine that the searched poly sound is not appropriate.

[0148] Step (740) may be performed if the searched poly sound is determined to be inappropriate. Step (735) may be performed if the searched poly sound is determined to be appropriate.

[0149] Again, in step (745) of FIG. 7, the poly sound providing device (320) may provide download data of the poly sound generated in at least one of steps (735) and (740) to the user device (310). In one embodiment, the poly sound providing device (320) may provide download data of the generated poly sound or download data of the generated poly sound merged with the target image based on a user input. The merging between the generated poly sound and the target image may be performed by an editor (e.g., the editor (410) of FIG. 4). In the example of FIG. 7, step (745) may be included in step (525) of FIG. 5.

[0150] FIG. 8 is a diagram illustrating an example of a user screen of a poly sound providing method according to one embodiment.

[0151] In one embodiment, a user may utilize a poly sound provision service provided by a poly sound provision device (320) through a user device (310). Referring to FIG. 8, an example of a user screen (800) displayed on a user device (310) when a user utilizes a poly sound provision service through the user device (310) is illustrated.

[0152] The user screen (800) may include a project upload area (805) for receiving a project file, a video viewer area (810) for displaying an uploaded video, a timeline area (815) for displaying a video track of the uploaded video and a sound track of poly sound generated by a poly sound providing device (320), a control area (820) for adjusting sound volume and EQ (equalizer), a text tag area (825) for displaying a context generated by the poly sound providing device (320), and other areas (830).

[0153] The text tag area (825) may display, for example, context information of a target image recognized through a poly sound providing device. The other area (830) may include, for example, at least one of a user interface (UI) element that commands the download of poly sound and a UI element that commands the download of an image in which poly sound is merged.

[0154] A user can upload a video for which he or she wishes to mix poly sound through the project upload area (805) or the video viewer area (810). The poly sound providing device (320) can generate poly sound by performing the methods of FIGS. 4 to 7 on the uploaded video and transmit the generated poly sound to the user device (310). The sound track of the poly sound transmitted to the user device (310) can be displayed in the timeline area (815) of the user screen (800) together with the video track of the uploaded video.

[0155] The poly sound displayed in the timeline area (815) of the user device (310) may be, for example, a poly sound displayed on the user device (310) before the download data of steps (525) of FIG. 5, (650) of FIG. 6, and (745) of FIG. 7 is provided. The user can edit the poly sound by manipulating the sound track displayed in the timeline area (815). The poly sound providing device (320) may receive a sound editing command from the user device according to the user operation when the user edits the poly sound through the user device. The poly sound editing according to the sound editing command may be performed by the poly sound providing device (320) through an editor of the poly sound providing device (320) (e.g., the editor (410) of FIG. 4).

[0156] The poly sound providing device (320) can provide download data of the poly sound edited in step (525) of FIG. 5, step (650) of FIG. 6, and step (745) of FIG. 7, or download data of an image into which the edited poly sound is merged, if the user has edited the poly sound.

[0157] The embodiments described above may be implemented using hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented using a general-purpose computer or a special-purpose computer, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing instructions and responding.

[0158] A processing device can execute an operating system (OS) and software applications running on the operating system. Furthermore, the processing device can access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing device is sometimes described as being used singly; however, those skilled in the art will appreciate that the processing device can include multiple processing elements and / or multiple types of processing elements. For example, the processing device can include multiple processors, or a processor and a controller. Other processing configurations, such as parallel processors, are also possible.

[0159] Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing device to perform a desired operation or may, independently or collectively, command the processing device. The software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave, for interpretation by the processing device or for providing instructions or data to the processing device. The software may also be distributed over networked computer systems and stored or executed in a distributed manner. The software and data may be stored on a computer-readable recording medium.

[0160] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program commands, data files, data structures, etc., alone or in combination, and the program commands recorded on the medium may be those specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of program commands include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0161] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

[0162] Although the embodiments described above have been described with limited drawings, those skilled in the art will appreciate that various technical modifications and variations can be applied based on the described embodiments. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0163] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

[0164] The mode for carrying out the invention has been described together with the best mode for carrying out the invention above.

[0165] The present disclosure relates to a device and method for generating and providing foley sound based on artificial intelligence, and has industrial applicability as it improves the possibility of repeating all steps of the foley sound mixing work, which is conventionally performed manually during video work, to be automatically processed through artificial intelligence.

Claims

1. A method for providing artificial intelligence-based poly sound performed by a computer device, A step of acquiring a target image including a subject performing a target action; A step of recognizing the properties of at least one object including the subject from the target image and determining context information related to the target action; A step of tracking the posture of the subject from the target image to determine the duration and intensity of the target behavior; A step of generating a poly sound corresponding to the target behavior of the subject based on the determined context information, the duration, and the intensity; and A step of providing download data of the generated poly sound or download data in which the generated poly sound and the target image are merged. A method comprising:

2. In paragraph 1, The steps for generating the above poly sound are: A step of retrieving a sound corresponding to the determined context information from a sound library; and When a sound corresponding to the above context information is searched, a step of adjusting the length of the searched sound based on the duration A method comprising:

3. In paragraph 2, The steps for generating the above poly sound are: If a sound corresponding to the context information is not searched, a step of inputting the context information, the duration, and the intensity into a learned neural network model; and A step of generating a poly sound corresponding to the input context information, duration, and intensity as an output of the learned neural network model. A method further comprising:

4. In paragraph 1, The above target action is a person's footsteps, The properties of at least one object are: A method comprising the gender of the subject, the material of the ground on which the subject is walking, and the material of the shoes worn by the subject.

5. In paragraph 4, The above context information is, A method characterized in that the gender of the recognized subject, the material of the recognized ground surface, and the material of the recognized shoe are determined in the form of text.

6. In paragraph 1, The above target action is a person's footsteps, The step of determining the duration of the target behavior and the intensity of the target behavior is as follows: A step of detecting the pose of the subject from the target image; A step of tracking the detected posture to obtain the movement of the subject in the target image; and A step of determining the duration and intensity of the step based on the acquired movement. A method comprising:

7. In paragraph 6, The step of acquiring the movement of the above subject is: In response to detecting the posture of the upper body of the subject, a step of tracking the detected posture of the upper body to obtain the movement of the upper body; and A step of acquiring the movement of the lower body of the subject based on the movement of the upper body A method comprising:

8. In paragraph 1, The above method, A step of receiving an original image from a user device; and Further comprising a step of inputting the original image into a learned neural network model and classifying at least one continuous frame corresponding to the target action among the frames of the original image, The step of acquiring the above target image is: A method for acquiring at least one consecutive frame classified as the target image.

9. In paragraph 1, Each of the above at least one object's properties is: Recognized through different neural network models trained to recognize the properties of each object from the image, The above context information is, A method characterized in that the text information is a combination of outputs of different neural network models.

10. In paragraph 1, The above method, A step of transmitting the generated poly sound to a user device; A step of receiving a sound editing command for the poly sound from a user device that has received the poly sound; and Step of editing the poly sound according to the above sound editing command Including more, The step of providing the above download data is: A method comprising the step of providing download data of the edited poly sound or download data of a target image into which the edited poly sound is merged.

11. A computer program stored on a computer-readable recording medium for executing the method of any one of claims 1 to 10 in combination with hardware.

12. In computer devices, processor; and Includes a memory that stores instructions to be executed by the processor, When the instructions are executed by the processor, the processor, Acquire a target image that includes a subject performing a target action, From the target image, determine context information related to the target action by recognizing the properties of at least one object including the subject, From the target image, the posture of the subject is tracked to determine the duration and intensity of the target behavior, Based on the determined context information, duration and intensity, a poly sound corresponding to the target action of the subject is generated, and A computer device providing download data of the generated poly sound or download data in which the generated poly sound and the target image are merged.

Images