Method and apparatus for producing sound

Abstract

One or more embodiments relate to a method for creating sound, in which an apparatus identifies motion features that are characteristic of the motion of an object in a video, and requests an AI to generate music with the time between similar motion features as a unit of time and a beat as the unit of time.

Description

[Technical field]

[0001] The present invention relates to a method for producing sound. [Background technology]

[0002] The statements in this section are merely intended to provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Patent Document 1 discloses a music generation system. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2006-154777 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the inventors have recognized that at least the above embodiment has a drawback in that there is no method for creating sound based on moving images. [Means for solving the problem]

[0006] At least one aspect of the present disclosure is a method for producing a method for manufacturing a semiconductor device comprising: 1. A method of producing a sound, comprising: The device, Identifying motion features that are characteristic of the motion of an object in a video; The AI ​​is required to generate music with the time between similar motion features as a unit of time, with the unit time being one beat. method to provide. Effect of the Invention

[0007] This configuration has the advantage of being able to generate industrially applicable music based on video.

[0008] These and other aspects, features, and advantages of the present disclosure will become apparent from the following detailed written description of the preferred embodiments and aspects taken in conjunction with the following drawings, variations and modifications of which may be made without departing from the spirit and scope of the novel concepts of the present disclosure. An aspect of one embodiment of the present disclosure may be combined with or substituted for one or more of the aspects of another embodiment of the present disclosure, unless inconsistent. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In the following disclosure, many different embodiments and examples are provided for implementing different features of the presented subject matter. To simplify the disclosure, specific examples of components and arrangements are disclosed below. Of course, these are merely examples and are not intended to be limiting. For example, a structure in which a first feature is covered by or in contact with a second feature subsequently disclosed may include embodiments in which the first feature and the second feature are formed in direct contact with each other, as well as embodiments in which an additional feature is formed between the first feature and the second feature such that the first feature and the second feature are not in direct contact with each other. Furthermore, the disclosure may repeat reference numbers and / or letters in various examples. Such repetition is for the sake of brevity and clarity, and does not, in itself, require a relationship between the various embodiments and / or configurations described. Furthermore, when a first element is described as "coupled" or "bonded" to a second element, such description includes embodiments in which the first and second elements are directly coupled or bonded to each other, as well as embodiments in which the first and second elements are indirectly coupled or bonded to each other with one or more other intervening elements therebetween.

[0010] As used herein, the phrase "at least one of" includes all exemplary variations. For example, the phrase "comprises at least one of A, B, or C" is equivalent to "consisting of A, B, C and combinations thereof," and includes all possible variations of A, B, C, A+B, A+C, B+C, and A+B+C.

[0011] In this disclosure, the disclosure using a machine, an electronic operator, or a computer may include embodiments of a method, a recording medium, an apparatus, or a program. As used herein, the statement "A is B" may be replaced with "A includes B" unless there is a contradiction or unless otherwise stated in the specification.

[0012] The terms in this disclosure, including the terms set forth in the claims, may be interpreted in light of the descriptions and drawings set forth in the specification and, further, in light of what has been or may be called, represented, understood or done by one or more of the public, in the past, present or future, unless otherwise indicated by a contradictory implication in this disclosure. The operating method used in at least one or more of the embodiments can take the following embodiments. The following description will be given with reference to JP6456303 (the following reference begins), which clearly explains at least one or more of the embodiments.

[0013] As used herein, the term "computer" generally includes a processor, memory, at least one information storage / retrieval device, such as a hard drive, disk drive or flash drive or memory stick, or other non-transitory computer-readable medium or non-transitory storage device, at least one input device, such as a keyboard, mouse, point and touch device, touch screen, or microphone, and a display structure, such as a well-known computer screen, as known in the art. In addition, a computer may include one or more network connections, such as wired or wireless connections. As known in the art, such computers or computer systems may include more or less of those listed above, including, for example, but not limited to, tablet computers and smart devices, as well as other electronic media and devices.

[0014] As used herein, the term "cloud" or "cloud computing" refers to a centralized and virtualized computing facility in which all computing resources are shared. Application systems and subsystems can no longer be pointed to specific machines, as they are all in the "cloud."

[0015] As used herein, the term "Distributed Internet Services System" refers to a distributed Internet services platform that transforms Internet applications to run in various computing environments. The DIS system distributes Internet applications, including content, data, and logic, to whatever extent appropriate and along the network to any number and type of devices via Component Distribution Servers / Asset Distribution Servers. Through the DIS, Internet applications can be hosted and centralized, with services based on each user's needs, cached and run locally on the user's device or nearby locations while maintaining their integrity. Web-enabled computing devices can be upgraded with DIS software to become DIS-enabled to enjoy and run distributed Internet services. The distributed internet services system is more fully described in any one of the following patent families: U.S. Patent Nos. 7,136,857, 7150015, 7181731, 7209921, 7430610, 7685183, 7685577, 7752214, 8326883, 8386525, 8443035, 8458142, 8458222, 8473468, 8527545, and 8650226, and U.S. Patent Publication Nos. 20120005205, and 20130091252, all of which, as well as the present invention, are commonly owned by OP40, Holdings, Inc., and all of which are incorporated by reference herein. (End of quote)

[0016] The operation method used in at least one or more embodiments can be implemented in the conventional Internet manner without using a distributed Internet. At least one or more embodiments are described with reference to JP7113047 (the following references are included), which clearly explains the invention.

[0017] Embodiments including those specifically disclosed in this specification can provide an automated response system that is based on artificial intelligence and is implemented in a form that resembles a real human conversation, thereby enabling more natural conversations with users while quickly and conveniently handling inquiries, reservations, delivery orders, etc.

[0018] The electronic devices 110, 120, 130, and 140 may be fixed terminals or mobile terminals realized by computer systems. Examples of the electronic devices 110, 120, 130, and 140 include an AI speaker, a smartphone, a mobile phone, a navigation system, a personal computer (PC), a notebook PC, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a tablet, a game console, a wearable device, an internet of things (IoT) device, a virtual reality (VR) device, and an augmented reality (AR) device. As an example, FIG. 1 shows an AI speaker as the electronic device 110, but in the embodiment of the present invention, the electronic device 110 may mean one of various physical computer systems that can communicate with other electronic devices 120, 130, and 140 and / or servers 150 and 160 via a network 170 using a substantially wireless or wired communication method.

[0019] The communication method is not limited, and may include not only a communication method using a communication network (such as a mobile communication network, a wired Internet, a wireless Internet, a broadcast network, a satellite network, etc.) that the network 170 can include, but also short-range wireless communication between devices. For example, the network 170 may include any one or more of a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), the Internet, etc. Furthermore, the network 170 may include any one or more of a network topology including, but not limited to, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or a hierarchical network, etc.

[0020] Each of the servers 150 and 160 may be realized by one or more computer devices that communicate with the multiple electronic devices 110, 120, 130, and 140 via the network 170 to provide instructions, codes, files, content, services, and the like. For example, the server 150 may be a system that provides a first service to the multiple electronic devices 110, 120, 130, and 140 connected via the network 170, and the server 160 may be a system that provides a second service to the multiple electronic devices 110, 120, 130, and 140 connected via the network 170. As a more specific example, the server 150 may provide a service (such as an auto-answer service, for example) targeted by an application, which is a computer program installed and executed in the multiple electronic devices 110, 120, 130, and 140, as a first service to the multiple electronic devices 110, 120, 130, and 140. As another example, the server 160 may provide, as a second service, a service of distributing files for installing and executing the above-mentioned application to the multiple electronic devices 110, 120, 130, and 140.

[0021] Fig. 2 is a block diagram for explaining the internal configuration of an electronic device and a server in an embodiment of the present invention. In Fig. 2, the internal configuration of electronic device 110 and the internal configuration of server 150 are explained as examples of electronic devices. In addition, other electronic devices 120, 130, 140 and server 160 may have the same or similar internal configuration as electronic device 110 or server 150 described above.

[0022] The electronic device 110 and the server 150 may include memories 211 and 221, processors 212 and 222, communication modules 213 and 223, and input / output interfaces 214 and 224. The memories 211 and 221 may be non-transitory computer-readable recording media and may include non-transitory mass storage devices such as random access memory (RAM), read only memory (ROM), disk drives, solid state drives (SSD), flash memories, and the like. Here, non-transitory mass storage devices such as ROM, SSD, flash memories, and disk drives may be included in the electronic device 110 and the server 150 as separate non-transitory storage devices separate from the memories 211 and 221. In addition, the memories 211 and 221 may store an operating system and at least one program code (for example, a code for a browser installed and executed in the electronic device 110, an application installed in the electronic device 110 to provide a specific service, and the like). Such software components may be loaded from a computer-readable recording medium separate from the memories 211 and 221. Such another computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD / CD-ROM drive, a memory card, etc. In another embodiment, the software components may be loaded into the memory 211, 221 through a communication module 213, 223 that is not a computer-readable recording medium. For example, at least one program may be loaded into the memory 211, 221 based on a computer program (for example, the above-mentioned application) that is installed by a file provided via the network 170 by a developer or a file distribution system that distributes an installation file of the application (for example, the above-mentioned server 160).

[0023] The processors 212, 222 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the processors 212, 222 by the memory 211, 221 or the communication modules 213, 223. For example, the processors 212, 222 may be configured to execute instructions received according to program code stored in a storage device such as the memory 211, 221.

[0024] The communication modules 213 and 223 may provide a function for the electronic device 110 and the server 150 to communicate with each other via the network 170, or may provide a function for the electronic device 110 and / or the server 150 to communicate with other electronic devices (for example, the electronic device 120) or other servers (for example, the server 160). For example, a request generated by the processor 212 of the electronic device 110 according to a program code recorded in a recording device such as the memory 211 may be transmitted to the server 150 via the network 170 under the control of the communication module 213. Conversely, a control signal, an instruction, a content, a file, etc. provided under the control of the processor 222 of the server 150 may be received by the electronic device 110 through the communication module 213 of the electronic device 110 via the communication module 223 and the network 170. For example, control signals, instructions, content, files, etc. from the server 150 received through the communication module 213 may be transmitted to the processor 212 or memory 211, and the content, files, etc. may be recorded on a recording medium (the non-transitory recording device described above) that the electronic device 110 may further include.

[0025] The input / output interface 214 may be a means for interfacing with the input / output device 215. For example, the input device may include a device such as a keyboard, a mouse, a microphone, or a camera, and the output device may include a device such as a display, a speaker, or a haptic feedback device. As another example, the input / output interface 214 may be a means for interfacing with a device in which functions for input and output are integrated into one, such as a touch screen. The input / output device 215 may be configured as one device together with the electronic device 110. Also, the input / output interface 224 of the server 150 may be a means for interfacing with a device for input or output (not shown) that may be connected to the server 150 or included in the server 150. As a more specific example, when the processor 212 of the electronic device 110 processes the instructions of the computer program loaded in the memory 211, a service screen or content configured using data provided by the server 150 or the electronic device 120 may be displayed on a display through the input / output interface 214.

[0026] Also, in other embodiments, the electronic device 110 and the server 150 may include more components than those in FIG. 2. However, it is not necessary to clearly show most of the conventional technical components in the figure. For example, the electronic device 110 may be realized to include at least some of the input / output devices 215 described above, and may further include other components such as a transceiver, a camera, various sensors, a database, etc. As a more specific example, if the electronic device 110 is an AI speaker, various components such as various sensors, a camera module, various physical buttons, a button using a touch panel, an input / output port, a vibrator for vibration, etc., which are generally included in an AI speaker, may be realized to be further included in the electronic device 110. (End of quote)

[0027] According to at least one embodiment, a user terminal includes a control unit, a RAM, a storage unit, a graphics processing unit, a communication interface, and an interface unit, each connected by an internal bus.

[0028] According to at least one embodiment, the control unit is composed of a CPU and a ROM. The control unit executes programs stored in the storage unit and controls the user terminal. The RAM is the work area of ​​the control unit. The storage unit is a memory area for saving programs and data. The control unit reads the programs and data from the RAM and processes them. The control unit processes the programs and data loaded into the RAM, thereby outputting drawing commands to the graphics processing unit.

[0029] According to at least one embodiment, the graphics processing unit is connected to the display unit. The display unit has a display screen. When the control unit outputs a drawing command to the graphics processing unit, the graphics processing unit outputs a video signal for displaying an image on the display screen. Here, the display unit may be a touch panel equipped with a touch sensor. The touch panel of the display unit functions as the input unit.

[0030] According to at least one embodiment, the communication interface can be connected to a communication network wirelessly or by wire, and can transmit and receive data to and from a server device via the communication network. Data received via the communication interface is loaded into RAM, and arithmetic processing is performed by the control unit. An external memory (e.g., an SD card, etc.) is connected to the interface unit.

[0031] According to at least one embodiment, the user terminal is a computing device having a display screen and an input section. Examples of user terminals include conventional mobile phones, tablet terminals, smartphones, and desktop and notebook personal computers. The user terminal may be comprised of a VR goggle, i.e., a screen (or two display panels, one for each eye) attached to a frame (or headset) that is strapped or attached to the head. The user terminal has an output section for audio.

[0032] According to at least one embodiment, the user terminal is communicatively connected to the server device via a communications network, and is capable of transmitting information or receiving information via the communications network.

[0033] According to at least one embodiment, the server device includes at least a control unit, a RAM, a storage unit, and a communication interface, each of which is connected by an internal bus.

[0034] According to at least one embodiment, the control unit is composed of a CPU and a ROM, executes a program stored in the storage unit, and controls the server device. The control unit also has an internal timer that measures time. The RAM is the work area of ​​the control unit. The storage unit is a memory area for saving programs and data. The control unit reads out the programs and data from the RAM, and performs program execution processing based on information received from the user terminal, etc.

[0035] Disclosed is an AI. According to at least one embodiment, artificial intelligence includes machine learning, deep learning, generative AI, large-scale language model, LLM, foundation model, and generative AI. Generative AI uses transformers and employs a number of mechanisms called attention. Self-supervised learning and Extract Prediction are used. In this case, the AI ​​can guess the next word. Given a sentence, it guesses the next word from the sentence up to the middle. A large number of supervised learning problems are created. These create an AI that can guess the next word. Generative AI can predict grammatical structure, topic connections, and that people with this style of writing are likely to write this kind of sentence. Furthermore, generative AI can learn the structure, causal relationships, and knowledge behind the next sentence just by guessing it. Generative AI has scalability, and the larger the number of parameters, the higher the accuracy. In ordinary statistics and machine learning, if the model parameters are made too large compared to the sample size of the data, overfitting occurs. In LLM, the accuracy increases as the number of parameters is increased. One generative AI has 175 billion parameters, and the generative AI overlays supervised learning to make the dialogue smooth. They are instructed not to say anything strange. They write reviews and act as call center operators.

[0036] According to at least one embodiment, large language models (LLMs) are machine learning natural language processing models built non-comprehensively using large datasets and deep learning techniques. In general, a technique called "fine tuning" is used to train on a specific task, and they are adapted to various natural language processing (NLP) tasks such as text classification / generation, sentiment analysis, text summarization, and question answering. According to at least one embodiment, self-supervised learning is close to the intrinsic intelligence of humans. When humans behave, they always predict the next event and predict the next input. In the process, they can learn the structure of the outside world. Predicting the next word is considered to be an intrinsic intelligence and is close to what the cerebral cortex does. According to at least one embodiment, large language models memorize all the information that is input, but generalize it to the extent necessary to predict the next word. They do not generalize all the information from the beginning. Large language models require capacity to memorize information. In addition, parameters are required for this purpose. According to at least one embodiment, the large-scale language model includes eight models with 175 billion parameters and 220 billion parameters.

[0037] According to at least one embodiment, we represent videos and images as a collection of visual patches, which are small units of data similar to text tokens in LLMs. Patches effectively represent models of visual data and serve as a highly scalable and effective representation for training generative models on a wide variety of videos and images. We convert videos to patches by first compressing them into a low-dimensional latent space and then decomposing the representation into spatio-temporal patches.

[0038] According to at least one embodiment, a video compression network is a network that reduces the dimensionality of visual data, taking raw video as input and outputting a temporally and spatially compressed latent representation. An AI is trained on this compressed latent space and then generates video in this compressed latent space.

[0039] According to at least one embodiment, Spacetime Latent Patches extracts a set of spacetime patches that act as Transformer tokens given a compressed input video. The patch-based representation allows Sora to be trained on videos and images of different resolutions, lengths, and aspect ratios, and controls the size of the generated videos by placing randomly initialized patches on an appropriately sized grid at inference time.

[0040] According to at least one embodiment, the AI ​​is a diffusion model, trained to predict the original "clean" patch when fed a noisy patch (and conditioning information such as a text prompt). The AI ​​is a diffusion transformer, which shows remarkable scaling properties in a variety of domains, including language modeling, computer vision, and image generation. Diffusion transformers are also effective as video generation models. The AI ​​significantly improves the quality of the samples as the training computational effort increases.

[0041] According to at least one embodiment, the AI ​​applies caption regeneration techniques to train a highly descriptive caption model, which is then used to generate text captions for all videos in the training set. Training on highly descriptive captions improves the fidelity of the text as well as the overall quality of the generated videos. GPT is leveraged to convert short user prompts into long, detailed captions that are sent to the model. This allows the AI ​​to generate high-quality videos that accurately follow the user's prompts.

[0042] According to at least one embodiment, the AI ​​can perform vectorization in natural language processing along the following flow. First, a cleaning process is performed on the given text as preprocessing. In the cleaning process, unnecessary words such as JavaScript code and HTML tags contained in the text are deleted. These codes are used to display on the Internet, and are therefore not generally used in natural language processing. Next, the text is divided into words by morphological analysis. Morphological analysis is the classification of natural language sentences written in characters into the smallest meaningful linguistic units. As morphological analysis tools, "MeCab", "JUMAN", and "JANOME" can be used. In normalization, words with the same meaning, such as spelling variations, are unified into one word. Stop words are words that are not processed because they cannot be used in natural language processing. Examples of stop words include words that have no meaning by themselves, such as particles and auxiliary verbs. When calculating vectors, these may be removed and only meaningful words may be targeted. Vectorization may also be performed without removing these stop words. Vectorization is a process that converts words, which are strings of characters, into vectors. Vectorization converts word data into numerical data. When converting words into vectors, methods called bag of words or distributed representations are used. Bag of words is a method of vectorizing a sentence using the number of occurrences of words that appear in a given sentence. Since it focuses on how many words appear in a sentence, it does not take into account the order of words or sentences. Distributed representation is a method of vectorization that focuses on the meaning of words. By vectorizing the meaning of words, it is possible to give vectors that are close to words with similar meanings and usages, and the relationship between words can also be expressed as vectors. The expression in vectors makes it possible to add and subtract the meanings of words. In applied processing, natural language converted into numerical data can be used as input for machine learning. Specifically, vectorized natural language is input into a classifier to classify sentences.Tools used here include "TensorFlow," "scikit-learn," and "PyTorch."

[0043] Sound is disclosed. In at least one embodiment, the sound includes music or sound effects. Music expresses various emotions and stories by combining the length, pitch, intensity, and tone of the sound, and includes singing, musical instrument performance, and natural sounds. Music can be composed by determining a scale, chord, or melody. In at least one embodiment, once a scale is determined, chords are also determined according to the constituent notes of the scale. There is a theory for chord progressions, and there are patterns that have been considered to be good progressions in songs to date. Chords greatly affect the mood of a song, and may greatly affect a person's emotions and feelings at that time. Similarly, melodies will not deviate significantly if based on the constituent notes of the chord, but they tend to become monotonous, so some randomness may be required. If the chords are varied within a range that does not deviate significantly, it may be possible to prevent the chords from becoming monotonous.

[0044] A method for creating sound is disclosed in which an apparatus identifies motion characteristics that are characteristic of the motion of an object in a video (in this disclosure, the term "method" may be interpreted as being replaced with the term "step" unless expressly stated to the contrary). In at least one embodiment, a video is something or data that creates motion by displaying still images one by one in succession. A video may include audio, and the video and audio may be synchronized to provide a richer expression. An object is data that is manipulated in the video or a process related thereto. Examples of objects are moving objects in a video (including living and non-living objects. Non-inclusive examples of the former include people, people dancing, animals, plants swaying in the wind, etc. Non-inclusive examples of the latter include moving objects (such as cars), the ebb and flow of waves, flashing lights, changes in the wavelength or intensity of light, and digital representations drawn by computer graphics). In this disclosure, an object that can be recognized as moving when a person watches a video is considered to be this "object" or data corresponding to the "object." In this disclosure, a "motion feature" may be a "motion feature amount" (this applies throughout this disclosure unless there is an explicit contradictory description). A feature amount is a numerical value that represents the characteristics of data contained in a certain dataset. For example, when expressing a person's face in a video, features include "eye size," "nose height," and "skin color." In addition, when a person is dancing, the way in which the organs (including hands, feet, waist, etc.) move can also be a feature amount. By providing these numerical values ​​to a computer, the computer can distinguish between people's faces, recognize specific people, and recognize dance choreography. In addition, in the ebb and flow of the waves, the way the waves rise and fall can naturally be a feature amount. In addition, in the case of computer graphics, the way a line or surface moves and the way the color changes can also naturally be a feature amount. Non-comprehensive types of features include the following. Numerical data is data that can be expressed numerically, such as height, weight, and age. Categorical data is data that can be expressed in categories, such as gender, nationality, and occupation. Text data is data that can be expressed as text, such as sentences, words, and keywords.Image data (including stills from video) are features extracted from images, such as color, shape, texture, etc. Video data are features extracted from video, such as color, shape, texture, or the presence or change of these over time. Audio data are features extracted from audio, such as pitch, frequency, timbre, etc.

[0045] In at least one embodiment, the device can be created by any of the methods, machines, or AI described above. In this disclosure, the description of "device" can be replaced with "AI" unless there is an explicit contradictory description. The device analyzes data of a target video. The device identifies an object in the video. The target object can be identified by a user. In another embodiment, the device identifies an object that appears relatively frequently among objects present in the video. The device can also request the AI ​​to do so (throughout this disclosure, unless there is an explicit contradictory description, the description "the device performs a certain process" can be replaced with "the device requests the AI ​​to perform the process." In this case, the AI ​​can use an AI that has learned the process or judgment by the learning method (including supervised learning) as described above). In such a case, the target object is automatically identified by the device, rather than specified by the user. The device identifies a motion characteristic of the object. As an example, in a video of a dancer dancing, the dancer's waving of his hands is a motion characteristic of the object. In this embodiment, the movement features can be, of course, the movement of the dancer's head, hips, feet, and fingers. As already described, the movement features can be specified by the user. Furthermore, the device can specify, as the movement features, the movement features that appear relatively frequently among the movements of the object.

[0046] A method is disclosed for requesting an AI to generate music with the time between similar motion features as a unit time and the unit time as a beat (beat). In this disclosure, unless there is an explicitly contradictory description, the description "generate music" can be replaced with "generate sound". In at least one embodiment, a video has similar motion features. As a non-comprehensive example, a dancer may repeatedly perform the same dance choreography. In this case, the device compares the motion features of each motion and recognizes them as similar motion features. There are two methods for determining the criteria for similarity: a method in which the user determines the criteria, and a method in which the machine automatically determines the criteria. In the former, the device displays videos or still images of two or more motion features on the screen of a terminal operated by the user. The user specifies to the device whether the videos or still images of the two or more motion features are similar motion features or not similar. The device determines whether the motion features are similar based on the specification. In the latter, the device determines whether the videos or still images of the two or more motion features contain clearly different motion features. If no obvious differences in features or quantities can be detected, the videos can be determined to be similar. In this determination, the AI ​​can learn about the similarity through supervised learning. The device can utilize the functions of the AI ​​or request the AI ​​to determine the similarity and obtain the result of the determination.

[0047] The device identifies the time between the movement features. As a non-exhaustive example, for a dance, if a dancer repeats the same dance choreography (similar movement features), the device identifies the difference (or distance) between the appearance time (or application time) of each movement feature in the video. As a non-exhaustive example, assume that a video of a dance performance is 3 minutes long. If the dancer repeats a movement feature at 1 minute 10 seconds and 1 minute 15 seconds, the time between the movement features is 5 seconds, and the unit time is 5 seconds. The device recognizes the time between the movement features as the unit time. The device can transmit the unit time information to another device or AI. In another embodiment, the device can use an AI that has learned to recognize the unit time by supervised learning as described above, or request the AI ​​to recognize the unit time.

[0048] In at least one embodiment, a beat in a sound includes a unit of rhythm. As a non-exhaustive example, a beat includes a fixed time interval (a basic unit that repeats at a fixed cycle in music), a contrast of strong and weak beats (a strong beat and a weak beat, and this contrast generates a rhythm), or a basis of tempo (the speed of the beat is the tempo, which determines the speed of the song). A beat may be a time signature. A time signature is the basis of the rhythm of an entire song and indicates how many beats there are in a measure. For example, in a 4 / 4 time signature, there are four beats in one measure, and each beat is the same length. In this disclosure, the description of "beat" can be replaced with "time signature" unless there is an explicit contradictory description. The device requests the AI ​​to generate music with a unit time being a beat (time signature). As a non-exhaustive example, if a video of a dance performance is three minutes long and the unit time is five seconds, the device requests the AI ​​to generate music with one beat per five seconds. In other embodiments, the device may generate the music by itself using the AI, as described above.

[0049] According to the above configuration, sound can be automatically created based on a video. The created sound or music has beats based on the features of the video, so there is a high degree of match between the video and the created sound. This has industrial applicability in that it reduces the workload of the sound creator, and enables the automatic generation of highly usable sound that has a high degree of match between the video and the created sound.

[0050] When requesting AI to generate music with a unit time being a beat, a method is disclosed for requesting AI to generate music of the same length as the video time (including the time for which the movement features should be identified. In this disclosure, unless there is an explicit contradictory description, "video time" can be understood as "video length" or "video time length"). The device requests AI to generate music of the same length as the video time. As a non-exhaustive example, when a video of a dance performance is a 3-minute video, the video length is 3 minutes, so the device requests that 3 minutes of music be generated. In another embodiment, the device sets the time for which the movement features should be identified as the length of the video. In at least one embodiment, the time for which the movement features should be identified refers to the entire or partial time period in the video during which the device or AI is allowed to identify the movement features. There are a method in which the user specifies the time for which the movement features should be identified, and a method in which the device automatically specifies the time. In the former, the user can specify the time for which the movement features should be specified within the time of the video. As a non-exhaustive example, when a video of a dance performance is three minutes long, the time for identifying movement features may be set to be between 0 minutes 0 seconds and 1 minute 30 seconds. The device identifies movement features only within the set time by the method described above. As an example of the latter, the device may set a predetermined portion of the time of the video as the time for identifying movement features. Of course, the device may set the entire time of the video as the time for identifying movement features. The device is requested to generate music of the same length as the time for identifying movement features.

[0051] A method is disclosed in which a device requests that music be generated so that the unit time in the video and the beat of the music are the same. The above description of "become the same" can be replaced with "synchronize". In at least one embodiment, the device generates music of the same duration as the time of the video, and requests that the music be generated so that the unit time in the video and the unit time of the music are the same. As a non-exhaustive example, assume that a video of a dance performance is a 3-minute video. If the dancer repeats a movement feature at 1 minute 12 seconds and 1 minute 17 seconds, the unit time is 5 seconds. Furthermore, the device requests the AI ​​to generate music with 5 seconds as one beat, and to generate music that is 3 minutes long. And since the device requests that the music be generated so that the unit time in the video and the unit time of the music are the same, the generated sound generates beats so that they are the same (synchronized) as the beats at 1 minute 12 seconds and 1 minute 17 seconds in the video. In this case, if the generated music has beats in 5-second units of time from 0:02, it will be identical (synchronized) with the beats at 1:12 and 1:17 in the video. In other words, the device recognizes the beats of the music that correspond to the unit of time in the video as specific beats, and requests the generation of music with the overall beat determined based on the specific beats (or without contradicting the specific beats). In the previous example, the beats at 1:12 and 1:17 correspond to the specific beats, and music with the overall beat determined based on the specific beats is generated.

[0052] According to the above configuration, it is possible to automatically create sound that is synchronized with the beat of a video. The created sound or music not only has a beat based on the features of the video, but is also synchronized with at least the beat of the video, so there is a higher degree of match between the video and the generated sound. In this case, the generated sound can be used industrially as it is by playing the video and the generated sound simultaneously. This has industrial applicability in that it reduces the workload of the sound creator, and can automatically generate highly usable sound that matches the video and the generated sound to a high degree.

[0053] A method is disclosed for requesting an AI to generate music with a unit time as a beat, by requesting a relatively short unit time to be a beat from among two or more unit times resulting from similar motion features. In at least one embodiment, the device may recognize three or more similar motion features from a video. In this case, the device may recognize two or more unit times. The device recognizes the shortest unit time from among the two or more unit times resulting from similar motion features as the unit time to be a beat. The device can transmit such a unit time to be a beat to another device or an AI, or request the AI ​​to recognize or identify such a unit time to be a beat. As already described, the AI ​​learns to set the shortest unit time as a beat by supervised learning using a video having two or more unit times resulting from similar motion features. As a non-comprehensive example, suppose a video of a dance performance is a three-minute video. For example, the A melody, B melody, and chorus are often used mainly in popular songs such as pop and rock, and each is composed of a unique melody. In a music video, the composition is a combination of an A-verse, a chorus, a B-verse, and a chorus, and the choreography of the chorus may be similar. Suppose the dancer repeats the movement features at 1 minute 12 seconds, 1 minute 17 seconds, 2 minutes 12 seconds, and 2 minutes 17 seconds. The device will recognize four types of unit time: 5 seconds, 55 seconds, 1 minute, and 1 minute 5 seconds. In this case, the device will recognize 5 seconds as the shortest unit time. The device will recognize the shortest unit time as a beat, or request the device or AI to recognize the shortest unit time as a beat.

[0054] According to the above configuration, even for complex video in which similar motion features are scattered throughout the video, it is possible to automatically create sound synchronized with the beat of the video. The created sound or music not only has a beat based on the features of the video, but is also synchronized at least with the beat of the video, so there is a higher degree of match between the video and the generated sound. In this case, the generated sound can be used industrially as it is by playing the video and the generated sound simultaneously. This has industrial applicability in that it reduces the workload of the sound creator, or automatically generates highly usable sound that matches the video and the generated sound to a high degree.

[0055] A method is disclosed in which, when an AI is requested to generate music with a unit time as a beat, the AI ​​is requested to set the identified unit time with a relatively high occurrence frequency as a beat from among two or more unit times resulting from similar motion features. In at least one embodiment, the device calculates the occurrence frequency for the recognized unit time. The occurrence frequency includes a numerical value indicating how many times an event or object appears within a specific period or range. As an example, the occurrence frequency includes an event (a specific word, number, action, phenomenon, etc.), a range (a specific sentence, data set, time, space, etc.), and a word occurrence frequency in a sentence (e.g., how many times the word "cat" appears in a sentence). The device calculates the occurrence frequency of the unit time within the time range of the video or the time in which the motion feature is to be identified. As a non-exhaustive example, assume that a video of a dance performance is a three-minute video, and the dancer's choreography is recognized as a motion feature. Then, assume that the device recognizes that there are 50 unit times of 5 seconds, 10 unit times of 10 seconds, 3 unit times of 20 seconds, and 1 unit time of 1 minute. In this case, the device recognizes the most frequently occurring 5 seconds as the unit of time that should be the beat. The device generates music by itself with this unit of time as the beat, or requests the AI ​​to generate it in this way.

[0056] According to the above configuration, as already explained, even for a complex video in which similar motion features are scattered within the video, it is possible to automatically create a sound synchronized with the beat of the video. Furthermore, since the most frequently occurring unit time is the beat, the beat of the generated music is unlikely to be inconsistent with the beat of the video. Therefore, the created sound or music not only has a beat based on the feature amount of the video, but is at least synchronized with the beat of the video, so that the degree of match between the video and the generated sound is high. In this case, the generated sound can be used industrially as it is if the video and the generated sound are played simultaneously. This has industrial applicability in that it reduces the workload of the sound creator, or automatically generates a highly usable sound that is highly consistent between the video and the generated sound.

[0057] A method is disclosed in which, when an AI is requested to generate music with a unit time as a beat, music of the same time as the time of the video or the time to identify the action feature is generated, and when two or more unit times due to dissimilar action features are identified, two or more pieces of music are generated and the music is requested to be connected. In at least one embodiment, the device recognizes the unit time based on similar action features as already described. In this case, the device recognizes a unit time based on an action feature A and a unit time based on an action feature B. These are two or more unit times due to dissimilar action features. When two or more unit times due to dissimilar action features are identified, the device generates two or more pieces of music. In another embodiment, the device requests the AI ​​or another device to generate two or more pieces of music in this way. The device connects the generated music. The connection includes a state in which multiple melodies are connected. The connection may be connected in a smooth and natural flow. As a non-exhaustive example, the device generates music A generated in a unit time based on an action feature A from a video (including a music video or a movie) and music B generated in a unit time based on an action feature B. The device links music A and music B. One method of linking is to link music data so that one music can be played as soon as another music ends. Another method is to link a portion of music data with a portion of other music data so that one music can be ended midway and another music can be played. In at least one embodiment, when two or more pieces of music are generated, the second piece of music can be a modulated piece of music from one piece of music.

[0058] According to the above configuration, even for a complex video in which two or more unit times are specified by dissimilar motion features, it is possible to automatically create a sound synchronized with the beat of the video. Furthermore, since the sound is based on two or more unit times, the possibility of it not matching the beat of the video is further reduced. Therefore, the created sound or music is mostly synchronized with the beat of the video, so there is a higher degree of match between the video and the generated sound. In this case, the generated sound can be used industrially as it is if the video and the generated sound are played simultaneously. This has industrial applicability in that it reduces the workload of the sound creator, or automatically generates a highly usable sound that matches the video and the generated sound to a high degree.

[0059] The method discloses a method for requesting that the music be linked at a time point identified based on the frequency of occurrence of two or more action features. In at least one embodiment, the device generates music for a time period equal to the time of the video or the time period when the action features are to be identified, and identifies a time point when the unit time changes for two or more unit times. As a non-exhaustive example, the device calculates the frequency or probability of occurrence for each unit time based on the action features in a time period of the video. For example, the device identifies a time period in which the unit time based on action feature A occurs relatively frequently and a time period in which the unit time based on action feature B occurs relatively frequently within a range of the time of the video or the time period when the action features are to be identified. The device generates music based on the unit time with the highest frequency of occurrence based on these frequencies of occurrence. When the unit time with the highest frequency of occurrence changes, the device generates music based on the new unit time. The device links these two pieces of music. In another embodiment, the device generates music for a time period equal to the time of the video (the time period when the action features are to be identified), and generates two or more pieces of music when two or more dissimilar action features are identified. The device identifies a time point (in this paragraph, unless explicitly stated to the contrary, the terms "time point" and "time" can be replaced with "time period") identified based on the frequency of occurrence of two or more dissimilar action features as the change time. For example, within the time range of the video or the time in which the action features are to be identified, the device identifies a time period in which the occurrence of unit times based on action feature A is relatively high and a time period in which the occurrence of unit times based on action feature B is relatively high. The device can identify a time point at which the most frequently occurring action feature changes, or a time point between or approximately midway between time periods with high occurrence frequencies based on different action features as the change time. As a non-exhaustive example, assume that a video of a dance performance is three minutes long. In the time period from 0 minutes 0 seconds to 1 minute 30 seconds, the occurrence of unit times based on action feature A is relatively high, and in the time period from 1 minute 30 seconds to 3 minutes, the occurrence of unit times based on action feature B is relatively high, and the device identifies 1 minute 30 seconds, which is approximately midway, based on the frequency of occurrence of two or more action features. Once identified, the device will request that the music be linked.That is, the generated music is music A based on movement feature A from 0:00 to 1:30, and music B based on movement feature B from 1:30 to 3:00. These are connected at the 1:30 point.

[0060] According to the above configuration, as already explained, even for a complex video in which two or more unit times are specified by dissimilar motion features, it is possible to automatically create a sound synchronized with the beat of the video. Furthermore, since the sound is based on two or more unit times, the possibility of it not matching the beat of the video is further reduced. Therefore, the created sound or music is mostly synchronized with the beat of the video, so there is a higher degree of match between the video and the generated sound. In this case, the generated sound can be used industrially as it is if the video and the generated sound are played simultaneously. This has industrial applicability in that it reduces the workload of the sound creator, or automatically generates a highly usable sound that matches the video and the generated sound to a high degree.

[0061] In at least one embodiment, the video includes a movie, a dance, and a choreography. As an example, the video includes a video published on Youtube (registered trademark). In this case, music can be automatically generated to match the published video. The video includes a video that is live-streamed. In this case, the device generates music according to the conditions or configurations described in any of the present disclosure at the same time as the live stream. The device can stream the generated music at the same time as the live stream video. According to the above configuration, music that matches the beat of the video can be provided at the same time as the live stream. Since such a flexible generation method is not possible for a human composer, it has a superior effect compared to conventional techniques and has industrial applicability. In at least one embodiment, the device can request the AI ​​to analyze environmental sounds and voices of the audience during the stream in real time, recognize one or more of them as specific sounds, features, or feature amounts, and generate sounds and sound effects according to them.

[0062] In at least one embodiment, the video includes a video used in VR or AR. VR (Virtual Reality) is a technology that provides an experience of being completely immersed in a virtual world. By wearing a device such as VR goggles, a user can feel as if they are in another world. AR (Augmented Reality) is a technology that overlays digital information on the real world. Through a smartphone camera, virtual objects can be displayed on a real landscape and information can be added. The device can generate music for a video to be presented to a user through VR or AR according to any of the conditions or configurations described in the present disclosure. The device can output the generated music as sound at the same time as the video is presented to the user through VR or AR. The audio output includes an embodiment in which audio is output from a user terminal. According to the above configuration, music that matches the beat of a VR or AR video can be provided. This has industrial applicability in that it reduces the workload of a person who creates sound, or automatically generates a highly usable sound that has a high degree of matching between a video and the generated sound.

[0063] In at least one embodiment, the AI ​​repeats learning to predict the next note through unsupervised learning. As already explained, the flow of a composition is hierarchical according to its structure. The elements that make up music are at least scales, chords, and melodies. Once a scale is decided, chords are also decided according to the constituent notes. There is a theory for chord progressions, and many patterns that are considered to be good progressions in songs to date have been accumulated. Chords are said to greatly affect the mood of a song and greatly affect the emotions and feelings of people at that time. The AI ​​repeats learning to predict the next note through unsupervised learning using music data. Furthermore, the AI ​​performs supervised learning on the music data as to what emotions the music indicates.

[0064] In at least one embodiment, the device or AI generates music based on music or information related thereto that has already been uploaded to the Internet or recorded in a database as completed, regardless of whether the author is registered or not. As a non-exhaustive example, when a unit time based on a motion characteristic of A is 5 seconds, the device identifies music with a beat of 5 seconds that has already been uploaded to the Internet as completed. If multiple pieces of music are identified, the device identifies music that matches any of the conditions described in this disclosure. The device or AI modifies or creates the identified music.

[0065] In at least one embodiment, the device or AI identifies one piece of music based on music or related information that has already been uploaded to the Internet or recorded in a database as completed music, regardless of whether the author is registered. In this embodiment, the device or AI is configured to identify existing music that meets conditions, rather than generate music. The device presents the identified music or information about the music to the user.

[0066] In at least one embodiment, the device or AI generates music using music or related information that has already been uploaded to the internet or recorded in a database as completed work, regardless of whether the work has been registered by a copyright holder.

[0067] In at least one embodiment, the device adds data indicating that the music was generated by AI to the data of the music generated by AI. When providing a video to a user together with the generated music, if the device recognizes that the music has the data, it presents to the user that the music was generated by AI, or displays that fact on the user's terminal. In another embodiment, when the device generates music using music that has already been uploaded to the Internet or recorded in a database as a completed work, it adds information about the name of the music or the copyright holder to the data of the generated music. If the device recognizes that the music contains the information, it presents to the user the name of the music, that the copyright holder created it, or that the AI ​​generated it based on the music, or displays that fact on the user's terminal. In at least one embodiment, the device generates data or a prompt that displays a copyright in a format compatible with the distribution platform. As an example, in Youtube (registered trademark), data or a prompt that displays a copyright as specified by the platform is generated, or the data is generated in association with the generated video. These are sometimes called copyright management tools. The above configuration has industrial applicability in that it can reduce the risk of copyright infringement in music data added or generated by AI.

[0068] In at least one embodiment, the user can specify to the device the sound to be relied upon when generating music. As an example, the device accepts sound data to be relied upon. The device stores two or more sound sources that can be specified by the user, and the user can select any sound source. "Two or more sound sources are stored" includes not only embodiments in which the device itself stores these sound sources, but also embodiments in which the device obtains these sound sources from an external storage medium. For example, these sound sources are pre-made music. The user selects one or more sound data to be relied upon, and the device generates music based on the selected music. As a non-exhaustive example, when the unit time of a video is 5 seconds and the selected sound data is pre-made music with a beat of 7 seconds, the device adjusts the playback speed of the pre-made music to be relied upon and changes it to music with a beat of 5 seconds. In this example, the changed music corresponds to the "generated music" in the above embodiment. According to the above configuration, pre-made music that is synchronized with the beat of the video can be automatically identified. Furthermore, since the identified pre-made music is a sound based on the unit time, it matches the beat of the video. In this case, the presented sound can be used industrially as it is if the video and the generated sound are played simultaneously. This has industrial applicability in that it reduces the workload of the sound creator, reduces the workload of searching for music with a matching beat, and automatically prepares a highly usable sound that matches the video and the generated sound to a high degree. In these embodiments, the device can generate or store data or prompts that should display a license for the used pre-made sound within the generated music data or accompany the data. The advantage of such an embodiment is at least that it provides a system that smoothly performs copyright management and licensing.

[0069] In at least one embodiment, the device performs a more detailed analysis of motion features (e.g., dance motions) in the video and generates different musical elements (e.g., beats and melodies) for each motion feature (e.g., dance steps). These musical elements may be adapted or concatenated according to any of the methods described herein. These musical elements may be added to the generated music as long as they do not contradict the beat of the generated music.

[0070] In at least one embodiment, the device recognizes a predetermined motion feature of an object in a video as a specific motion feature, and generates a specific sound, which is a predetermined sound, at the time when the specific motion feature occurs. The device can generate the specific sound in addition to the generated music. In another embodiment, the device can generate the specific sound independently of the generated music. The user registers information about the specific motion feature or the specific sound to the device. As a non-exhaustive example, the user registers a person's surprised appearance or a predetermined reaction to the device as information about the specific sound, the appearance of surprise or a prompt. The specific sound includes pre-made music or sound effects specified by the user. The device can store two or more of such pre-made music or sound effects by itself or receive them from an external storage device. The device can present two or more of such available specific sounds to the user. The user can specify the specific sound to be used from among them. When two or more specific sounds are specified, the device uses the specific sound to be generated alternately, randomly, or according to an embodiment of the present disclosure. When the video is a two-hour movie, and at 1 hour, 20 minutes, and 43 seconds, a person in the movie shows signs of surprise, the device generates a specific sound representing the sound of surprise at that time. According to the above configuration, a specific sound based on a specific action feature can be automatically generated. Furthermore, since the generated specific sound is based on the time when the specific action feature occurs, it coincides with the timing of the occurrence of the specific action feature in the video. In this case, the presented sound can be used industrially as it is if the video and the generated sound are played simultaneously. This reduces the workload of the person who creates the specific sound, or can automatically prepare a highly usable sound that matches the video and the generated sound, and has industrial applicability. As already described, this configuration can be combined with or replaced with one or more aspects in another embodiment disclosed herein, as long as there is no contradiction. That is, it is also possible to analyze the environmental sound and the voice of the audience during distribution in real time and generate background music and sound effects (specific sounds) corresponding to the analyzed sounds. In another embodiment, it can be implemented as an application that automatically inserts a surprised sound effect for a surprised action.

[0071] In at least one embodiment, the moving image includes a movie. According to this embodiment, optimal music or specific sounds can be generated separately for each sequence of the movie.

[0072] An embodiment different from the above will be disclosed.

[0073] A method is disclosed in which an apparatus identifies features of an object in a video and requests an AI to generate music based on the features. The method includes a method of identifying motion features that are characteristic of a motion and requesting an AI to generate music based on the motion features. According to at least one embodiment, the apparatus or AI represents a video or image as a collection of visual patches, which are small data units similar to text tokens of an LLM. The apparatus combines a video recognition AI and an LLM to understand each scene in the video. Specifically, the video recognition AI is utilized to individually recognize various objects and environments that make up a scene, such as people, cars, buildings, animals, natural objects such as trees, and weather, and changes therein. The AI ​​learns emotions corresponding to the visual patches by the learning method described above (including supervised learning). Furthermore, the AI ​​may be fine-tuned in advance by using sample videos in the target field. For example, the AI ​​learns that a particular visual patch corresponds to a mood or atmosphere, and an emotion (collectively referred to as a "concept" in this disclosure). The apparatus requests the AI ​​to generate music based on the features based on the motion features. As an example, if the video is a three-minute music video and the dancer's facial expression is a smile, the device requests the AI ​​to identify motion characteristics such as cheerful and fun, and generate music based on the characteristics. In another embodiment, the device also requests the AI ​​to identify motion characteristics that are characteristics of the motion of objects in the video. As another example, if the video is a three-minute music video and the back of the dancer is a ruin for the performance, the device requests the AI ​​to identify auxiliary features such as ruins and darkness as auxiliary features and generate music based on the auxiliary features. In another example, the video is a movie, and the scene and emotional flow of the video can be analyzed and a corresponding music composition (intro, climax, ending) can be generated. In addition, the music corresponding to the concept includes an embodiment corresponding to a music genre (classical, EDM, hip hop, etc.). According to this embodiment, music that matches the taste or concept of the video can be automatically generated.This has industrial applicability in that it reduces the workload of the sound creator and automatically prepares highly usable sound that closely matches the concept of the video and the generated sound.

[0074] In at least one embodiment, the device may also set the feature specified by the user as the action feature or the auxiliary feature. The user inputs the concept of the music to be generated into the device in the form of characters. These characters may be prompts. The device requests the AI ​​to generate music based on the concept input by the user. Furthermore, the device may request the AI ​​to generate music of the same time as the time of the video or the time at which the action feature is to be specified. In one embodiment, the user can select a mood or atmosphere, and music that matches the selected mood can be generated. According to this embodiment, the user can specify his / her preferred taste, so that music that matches the user's taste can be automatically generated. This has industrial applicability in that it reduces the workload of the person who creates the sound, or automatically prepares a highly usable sound that has a high degree of match between the concept of the video and the generated sound.

[0075] In at least one embodiment, the device analyzes lyrics as features specified by the user. The device or AI recognizes concepts contained in the lyrics as features. For example, the device or AI extracts concepts such as love, heartbreak, and sadness based on the lyrics. The device or AI generates sound based on the extracted concepts. When two or more concepts are recognized based on the lyrics, the AI ​​is requested to base the sound on the feature with the highest occurrence frequency in which the feature is recognized, as described below. According to this method, even if the lyrics are complex, music can be generated based on the main concept of the lyrics. This has industrial applicability in that it reduces the workload of the person creating the sound, or automatically prepares a highly usable sound that has a high degree of match between the concept of the video and the generated sound.

[0076] In at least one embodiment, the device requests the AI ​​to convert objects in a video into text. As already described, the AI ​​or the device represents a video or image as a collection of visual patches, which are small data units similar to text tokens in an LLM. The device combines a video recognition AI with an LLM to understand each scene in the video. Specifically, the video recognition AI is used to individually recognize various objects and environments that make up a scene, such as people, cars, buildings, animals, natural objects such as trees, and weather, and changes therein. The AI ​​learns characters that correspond to the visual patches by the learning method (including supervised learning) as described above. This method allows the AI ​​to describe a given video in text (convert it into text). The AI ​​generates music based on the converted characters. Specifically, the device identifies features based on the converted characters and requests the AI ​​to generate music based on the features. The method of generating music based on features is as disclosed elsewhere in this disclosure.

[0077] A method is disclosed in which, when requesting AI to generate music based on features, when there are two or more features, the AI ​​is requested to generate music based on at least the feature for which the feature is recognized for a relatively long time. In at least one embodiment, the device may recognize two or more features. In this case, the AI ​​is requested to generate music based on the feature for which the feature is recognized for the longest time in the video. The device identifies the time for which each feature is recognized. As a non-comprehensive example, if a video is a three-minute movie, of which two minutes and 40 seconds depict a broken heart, but 20 seconds depict a fun and uplifting love affair as a flashback to the past, the feature of "broken heart" is two minutes and 40 seconds, and the feature of "fun" is 20 seconds. In other words, the device requests the AI ​​to create music based on the feature of "broken heart", which is the longest feature. Note that this method is not intended to be limited to one feature, and it is also possible to generate music based on two or more features. According to the above configuration, even if a video has multiple concepts, music based on the main concept can be generated. Therefore, even if the video contains contradictory concepts, since the music is based on the main concept, there is a high possibility that the concept of the video will match. This has industrial applicability in that it reduces the workload of the sound creator and automatically prepares highly usable sound that closely matches the concept of the video and the generated sound.

[0078] A method is disclosed in which, when requesting AI to generate music based on features, when there are two or more features, the AI ​​is requested to generate music based on at least the feature with a relatively high occurrence frequency in which the feature is recognized. As already described, in at least one embodiment, the device may recognize two or more features. In this case, the AI ​​is requested to generate music based on the feature with the highest occurrence frequency in the video. The device identifies the number of times each feature is recognized. As a non-comprehensive example, if the video is a three-minute movie in which the actor laughs 15 times, cries once, and is angry once, the most frequent feature is "laughter." In other words, the device requests the AI ​​to create music based on the feature of "laughter," which is the longest feature. Note that this method is not intended to be limited to one feature, and it is also possible to generate music based on two or more features. According to the above configuration, even if a video has multiple concepts, music based on the main concept can be generated. Therefore, even if the video contains contradictory concepts, since the music is based on the main concept, there is a high possibility that the concept of the video will match. This has industrial applicability in that it reduces the workload of the sound creator and automatically prepares highly usable sound that closely matches the concept of the video and the generated sound.

[0079] A method is disclosed in which, when requesting AI to generate music based on features, music is generated with the same time as the time of the video or the time to identify the action feature, and when two or more dissimilar features are identified, two or more pieces of music are generated and the music is requested to be linked. As already explained, when two or more features are identified, the device or AI can generate two or more pieces of music. Also, as already explained, the device or AI can link two or more pieces of music. According to the above configuration, even for a complex video having two or more dissimilar features, a sound similar to the concept of the video can be automatically created. Furthermore, since the sound is based on two or more concepts, the possibility of it not matching the concept of the video is further reduced. Therefore, the created sound or music is largely synchronized with the concept of the video, so the degree of match between the video and the generated sound is higher. In this case, the generated sound can be used industrially as it is if the video and the generated sound are played simultaneously. This reduces the workload of the sound creator, or automatically generates a highly usable sound that has a high degree of match between the concept of the video and the generated sound, and thus has industrial applicability.

[0080] In the method, a method is disclosed in which the music is requested to be linked at a time point identified based on the frequency of appearance of two or more features. As already described, the device can calculate the frequency or probability of appearance of each feature in a time period of a video. For example, the device identifies a time period in which unit times based on feature A occur relatively frequently and a time period in which unit times based on feature B occur relatively frequently within a range of the time of the video or the time period in which the action feature is to be identified. The device generates music based on the unit time with the highest frequency of appearance based on these frequencies of appearance. When the unit time with the highest frequency of appearance is changed, the device generates music based on the new unit time. The device links these two pieces of music. In another embodiment, the device generates music for the same time as the time of the video (the time in which the action feature is to be identified), and generates two or more pieces of music when two or more dissimilar action features are identified. The device identifies the time point identified based on the frequency of appearance of two or more dissimilar action features as the change time. For example, the device identifies, within the time range of the video or the time for which the action feature is to be identified, a time period in which the occurrence of unit times based on feature A is relatively high and a time period in which the occurrence of unit times based on feature B is relatively high. The device can identify, as the change time, a time point at which the most frequently occurring action feature changes, or a time point between or approximately midway between time periods with high occurrence frequencies based on different action features. As a non-exhaustive example, assume that a video of a movie is 10 minutes long. In the time period from 0 minutes 00 seconds to 1 minute 30 seconds, the occurrence of unit times based on feature A (e.g., the feature that the setting is in ruins) is relatively high, and in the time period from 1 minute 30 seconds to 3 minutes, the occurrence of unit times based on feature B (e.g., the feature that the setting is in a gorgeous building (such as a castle)) is relatively high, and the device identifies 1 minute 30 seconds, which is approximately midway, based on the occurrence frequencies of two or more action features. At the identified time point, the device requests that the music be linked. That is, the generated music is music A based on feature A from 0:00 to 1:30, and music B based on feature B from 1:30 to 3:00. These are connected at the 1:30 point.

[0081] According to the above configuration, as already explained, even for a complex video in which two or more unit times are specified by dissimilar features, it is possible to automatically create a sound similar to the concept of the video. Furthermore, since the sound is based on two or more unit times, the possibility of it not matching the concept of the video is further reduced. Therefore, the created sound or music is largely synchronized with the concept of the video, so there is a higher degree of match between the video and the generated sound. In this case, the generated sound can be used industrially as it is if the video and the generated sound are played simultaneously. This has industrial applicability in that it reduces the workload of the sound creator, or automatically generates a highly usable sound that is highly consistent between the concept of the video and the generated sound.

[0082] In at least one embodiment, when generating music based on lyrics or characters, the device recognizes characters or concepts in a sentence that indicate rhythm, and requests the AI ​​to generate music with a rhythm based on the characters or concept. With the above configuration, music can be generated with a specified rhythm, further reducing the possibility that the generated music will not match the concept of the video. Therefore, the generated sound can be used industrially as it is if the video and the generated sound are played simultaneously. This has industrial applicability in that it reduces the workload of the sound creator, or automatically generates highly usable sound that is highly consistent between the video and the generated sound in concept.

[0083] An embodiment different from the above will be disclosed. The device requests the AI ​​to create lyrics based on music data. The device or AI breaks down the music data into units such as scales, chords, or melodies. The AI ​​learns the mood, atmosphere, or emotion that each unit represents through supervised learning. Furthermore, the AI ​​learns and fine-tunes by combinations of already published music scales, chords, or melodies and lyrics. This allows the AI ​​to predict what characters or lyrics will come when a scale, chord, or melody is given. Furthermore, it can predict what word will come next for the characters or lyrics that have come. In this way, the AI ​​can create lyrics based on music data. The device requests the AI ​​to create lyrics based on music data. According to the above configuration, lyrics based on traditional concepts of music can be generated, so the generated lyrics are less likely to be inconsistent with the concept of music. Furthermore, since AI processes and learns natural language, the possibility that the generated lyrics will contain grammatical errors is certainly low. Therefore, the generated lyrics can be directly used industrially together with the music. This has industrial applicability in that it reduces the workload of the person who creates the lyrics, or automatically generates lyrics with high applicability that have a high degree of match between the concept of the music and the generated lyrics. In at least one embodiment, the device further requests the AI ​​to generate a singing voice based on the generated lyrics. In at least one embodiment, the device requests the AI ​​to generate music, requests the AI ​​to create lyrics based on the generated music, requests the AI ​​to generate a singing voice based on the generated lyrics, and requests the AI ​​to match (integrate, combine) the generated singing voice and music data (or the device itself matches the data). An application of the AI ​​that generates a singing voice includes Vocaloid (registered trademark). This method has industrial applicability in that it can automatically generate a singing voice that matches the concept of the video.

[0084] The following provides an overview of the above-described embodiment.

[0085] 1. A method of producing a sound, comprising: The device, Identifying motion features that are characteristic of the motion of an object in a video; The AI ​​is required to generate music with the time between similar motion features as a unit of time, with the unit time being one beat. method.

[0086] When you ask an AI to generate music with a unit time being a beat, Generate music of the same duration as the time of the video or the time at which the action feature is to be identified; Requires that music be generated so that the unit time in the video and the beat of the music are the same. method.

[0087] When you ask an AI to generate music with a unit time being a beat, Among two or more unit times resulting from similar motion characteristics, a relatively short unit time is required to be defined as a beat. method.

[0088] When you ask an AI to generate music with a unit time being a beat, A request is made to select a unit time having a relatively high occurrence frequency from among two or more unit times resulting from similar motion characteristics as a beat. method.

[0089] When you ask an AI to generate music with a unit time being a beat, Generate music of the same duration as the time of the video or the time at which the action feature is to be identified; When two or more time units with dissimilar motion features are identified, two or more pieces of music are generated; Request that the music be linked; method.

[0090] In the method, requesting that the music be linked when identified based on the frequency of occurrence of two or more motion characteristics; method.

[0091] 1. A method of producing a sound, comprising: The device, Identify the features (feature values) of objects in the video, Ask the AI ​​to generate music based on features, method.

[0092] When you ask an AI to generate music based on features, When there are two or more characteristics, require the AI ​​to base its selection on at least the characteristic that has been observed for a relatively long time; method

[0093] When you ask an AI to generate music based on features, When there are two or more features, require the AI ​​to base its selection on at least the feature with a relatively high frequency of occurrence in which the feature is recognized; method

[0094] When you ask an AI to generate music based on features, Generate music of the same duration as the time of the video or the time at which the feature is to be identified; When two or more dissimilar features are identified, two or more pieces of music are generated; Request that the music be linked; method.

[0095] In the method, requesting that the music be linked once identified based on the frequency of occurrence of two or more characteristics; method.

[0096] In at least one embodiment, the description "requesting that, from among two or more unit times resulting from similar motion features, a relatively short unit time be set as a beat" can be replaced with "requesting that, from among two or more unit times resulting from similar motion features, a shortest unit time be set as a beat." The description "requesting that, from among two or more unit times resulting from similar motion features, a unit time with a relatively high occurrence frequency of a specified unit time be set as a beat" can be replaced with "requesting that, from among two or more unit times resulting from similar motion features, a unit time with a highest occurrence frequency of a specified unit time be set as a beat." The description "requesting that, when two or more features are present, the AI ​​be based on at least a feature with a relatively long time in which the feature is recognized" can be replaced with "requesting that, when two or more features are present, the AI ​​be based on at least a feature with a longest time in which the feature is recognized." The description "requesting that, when two or more features are present, the AI ​​be based on at least a feature with a relatively high occurrence frequency in which the feature is recognized" can be replaced with "requesting that, when two or more features are present, the AI ​​be based on at least a feature with a highest occurrence frequency in which the feature is recognized."

[0097] 1. A method of producing a sound, comprising: The device, Generate music with the same duration as the video or a specified duration, Ask the AI ​​to generate music based on the characteristics of the lyrics that correspond to the video, method.

[0098] In the above configuration, the specified time includes the time specified by the user. The device requests the AI ​​to generate music within the time range specified by the user. The method of generating music based on the characteristics of lyrics and its advantages are as described above.

[0099] In at least one embodiment, the user can make detailed musical adjustments (tempo, key, effects) to the AI-generated music or vocals through the device. The device changes the tempo, key, effects, etc. of the generated music or vocal data in response to the user's adjustments. Such adjustments can be made even if the video is a real-time video or a distributed video.

[0100] In at least one embodiment, two or more users can collaborate through the device to input rhythms and concepts to be generated and fine-tune music (tempo, key, effects) for AI-generated music or vocals. The device accepts information input from two or more users. Two or more users can collaborate on editing a piece of data through the device. Such editing can be done even if the video is a real-time video or a streaming video. As a non-exhaustive example, this editing function is referred to as a "collaboration mode" or "real-time mode."

[0101] In at least one embodiment, the device requests the AI ​​to create lyrics in one or more foreign languages ​​in response to lyrics or a concept. The device requests the AI ​​to create music in response to the input lyrics or concept in a foreign language. The AI ​​fine-tunes the concept and music in response to the foreign language. As a non-exhaustive example, fine-tunes concepts related to ethnic music, such as a koto for a Japanese concept, or bagpipes for a European or Irish concept. The device also requests the AI ​​to generate music for a foreign concept when lyrics in a foreign language are input.

[0102] In at least one embodiment, the device requests the AI ​​to generate music or video in a format compatible with major distribution platforms (non-inclusively, YouTube, TikTok, Instagram, etc., all of which are registered trademarks) so that the generated music or video will work on major distribution platforms. In another embodiment, the device converts the AI-generated music or video into a format compatible with major distribution platforms so that the generated music or video will work on major distribution platforms. The advantage of such an embodiment is at least that it creates platform-compatible data and improves user convenience.

[0103] It is sufficient for the invention according to the present disclosure to achieve at least one of the effects described above.

Claims

1. 1. A method of producing a sound, comprising: The device, Identifying motion features that are characteristic of the motion of an object in a video; The AI ​​is required to generate music with beats in a relatively short unit time from two or more unit times resulting from similar motion features, with the time between similar motion features being the unit time. method.

2. 1. A method of producing a sound, comprising: The device, Identifying motion features that are characteristic of the motion of an object in a video; The AI ​​is required to generate music with beats based on the time between similar motion features, which is determined as a unit time and which is determined to have a relatively high occurrence frequency among two or more unit times resulting from similar motion features. method.

3. 1. A method of producing a sound, comprising: The device, Identify features of objects in the video, When there are two or more features, require the AI ​​to generate music based on at least the feature that is present for a relatively long period of time. method.

4. 1. A method of producing a sound, comprising: The device, Identify features of objects in the video, When there are two or more features, the AI ​​is required to generate music based on at least the features that appear relatively frequently. method.

5. An apparatus for producing sound, comprising: Identifying motion features that are characteristic of the motion of an object in a video; The AI ​​is required to generate music with beats in a relatively short unit time from two or more unit times resulting from similar motion features, with the time between similar motion features being the unit time. Device.

6. An apparatus for producing sound, comprising: Identifying motion features that are characteristic of the motion of an object in a video; The AI ​​is required to generate music with beats based on the time between similar motion features, which is determined as a unit time and which is determined to have a relatively high occurrence frequency among two or more unit times resulting from similar motion features. Device.

7. An apparatus for producing sound, comprising: Identify features of objects in the video, When there are two or more features, require the AI ​​to generate music based on at least the feature that has a relatively long duration of presence; Device.

8. An apparatus for producing sound, comprising: Identify features of objects in the video, When there are two or more features, the AI ​​is required to generate music based on at least the features that appear relatively frequently. Device.