How to create sounds

Abstract

This provides a method for creating industrially usable music based on video. [Solution] A method for creating sound, wherein the device identifies motion features, which are characteristics of the movement of objects in a video, and requests the AI ​​to generate music in which the time between similar motion features is used as the unit time, and the unit time is used as the beat.

Description

Technical Field

[0001] The present invention relates to a method for creating sound.

Background Art

[0002] The statements in this section only provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] ,Patent Document 1 discloses a music generation system.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0009] The following disclosure provides many different embodiments and examples for carrying out different features of the presented subject matter. For the sake of simplicity, specific examples of components and arrangements are disclosed below. Of course, these are merely examples and are not intended to be limiting. For example, a structure in which a first feature is covered by or in contact with a second feature subsequently disclosed may include embodiments in which an additional feature is formed between the first and second features so that they do not come into direct contact, as well as embodiments in which the first and second features are formed so that they do not come into direct contact. Furthermore, the disclosure may repeat reference numbers and / or letters in various examples. This repetition is for the sake of brevity and clarity and does not require in itself to be related to the various embodiments and / or configurations described. Furthermore, when describing the first element as being "connected" or "joined" to the second element, such description includes embodiments in which the first and second elements are directly connected or joined to each other, as well as embodiments in which the first and second elements are indirectly connected or joined to each other by having one or more other elements interposed between them.

[0010] As used herein, the phrase "at least one of" encompasses all the exemplary variations. For example, the phrase "comprises at least one of A, B, or C" is synonymous with "consisting of A, B, C and combinations thereof," and encompasses all conceivable variations of A, B, C, A+B, A+C, B+C, and A+B+C.

[0011] In this disclosure, disclosures involving the use of machines, electronic operators, or computers may include embodiments of methods, recording media, apparatus, or programs. Any statement used herein that “A is B” may be replaced with “A includes B” unless otherwise stated herein, to the extent that it does not conflict with the statement.

[0012] The terms used in this disclosure, including those used in the claims, may be interpreted in light of the descriptions and drawings in the specification, and, to the extent that they do not contradict the implications of this disclosure, they may also be interpreted in relation to what one or more citizens have called, represented, understood or practiced, or may have done in the past, present, or future, as such. The following embodiments can be used for the operating method in at least one embodiment. The following will be explained by reference to the description in JP6456303, which describes at least one embodiment in detail (beginning of reference).

[0013] As used herein, the term “computer” schematically includes, as known in the art, a processor, memory, at least one information storage / retrieval device such as a hard drive, disk drive or flash drive or memory stick, or other non-temporary computer-readable medium or non-temporary storage device, at least one input device such as a keyboard, mouse, point and touch device, touchscreen or microphone, and a display structure such as a well-known computer screen. In addition, a computer may include one or more network connections, such as wired or wireless connections. As known in the art, such a computer or computer system may more or less include, but is not limited to, for example, tablet computers or smart devices, other electronic media and electronic devices.

[0014] As used herein, the terms “cloud” or “cloud computing” refer to centralized and virtualized computing facilities where all computing resources are shared. For application systems and subsystems, since they all reside within the “cloud,” it is no longer possible to refer to specific machines.

[0015] As used herein, the term “Distributed Internet Services System” refers to a distributed internet services platform that translates internet applications for execution in various computing environments. A DIS system distributes internet applications, including content, data, and logic, to any number and type of device, to whatever extent appropriate, and along a network, via Component Distribution Servers / Asset Distribution Servers. Through a DIS, internet applications can be hosted and centrally managed as a service tailored to each user's needs, and can be locally cached and executed on the user's device or nearby locations while maintaining their integrity. Web-enabled computing devices can be upgraded with DIS software to become DIS-enabled, capable of enjoying and running distributed internet services. The Distributed Internet Service System is fully described in any one of the following patent families: U.S. Patents No. 7,136857, 7150015, 7181731, 7209921, 7430610, 7685183, 7685577, 7752214, 8326883, 8386525, 8443035, 8458142, 8458222, 8473468, 8527545, and 8650226, and U.S. Patent Publications 20120005205 ​​and 20130091252, all of which, like the present invention, are jointly owned by OP40 Holdings, Inc., and all of which are incorporated by reference. (End of quote)

[0016] Regarding the operating method used in at least one embodiment, the following embodiments can be taken for conventional internet systems that do not use a distributed internet. The following will be explained by reference to the description in JP7113047, which describes at least one embodiment in detail (beginning of reference).

[0017] Embodiments including those specifically disclosed herein can provide an automated response system based on artificial intelligence that is implemented in a manner that mimics actual human conversation, thereby enabling more natural communication with users while quickly and conveniently handling inquiries, reservations, delivery orders, and more.

[0018] The multiple electronic devices 110, 120, 130, and 140 may be fixed or mobile terminals implemented by a computer system. Examples of the multiple electronic devices 110, 120, 130, and 140 include AI speakers, smartphones, mobile phones, navigation systems, PCs (personal computers), notebook PCs, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablets, game consoles, wearable devices, IoT (Internet of Things) devices, VR (virtual reality) devices, and AR (augmented reality) devices. As an example, Figure 1 shows an AI speaker as electronic device 110, but in embodiments of the present invention, electronic device 110 may mean one of a variety of physical computer systems that can communicate with other electronic devices 120, 130, 140 and / or servers 150, 160 via a network 170 using substantially wireless or wired communication methods.

[0019] The communication method is not limited, and may include not only communication methods that utilize communication networks that can be included in network 170 (for example, mobile communication networks, wired internet, wireless internet, broadcasting networks, satellite networks, etc.), but also short-range wireless communication between devices. For example, network 170 may include one or more arbitrary networks such as PAN (personal area network), LAN (local area network), CAN (campus area network), MAN (metropolitan area network), WAN (wide area network), BBN (broadband network), and the Internet. Furthermore, network 170 may include, but is not limited to, one or more network topologies, including bus networks, star networks, ring networks, mesh networks, starbus networks, tree or hierarchical networks.

[0020] Servers 150 and 160 may each be implemented by one or more computer devices that communicate with a plurality of electronic devices 110, 120, 130, 140 via a network 170 to provide commands, codes, files, contents, services, etc. For example, server 150 may be a system that provides a first service to a plurality of electronic devices 110, 120, 130, 140 connected via network 170, and server 160 may also be a system that provides a second service to a plurality of electronic devices 110, 120, 130, 140 connected via network 170. As a more specific example, server 150 may provide, as the first service, a service (such as an automatic response service, for example) targeted by the corresponding application to a plurality of electronic devices 110, 120, 130, 140 through an application that is a computer program installed and executed on the plurality of electronic devices 110, 120, 130, 140. As another example, server 160 may provide, as the second service, a service that distributes files for installation and execution of the above-described application to a plurality of electronic devices 110, 120, 130, 140.

[0021] FIG. 2 is a block diagram for explaining the internal configurations of an electronic device and a server in an embodiment of the present invention. In FIG. 2, the internal configuration of electronic device 110 and the internal configuration of server 150 are described as examples for an electronic device. Also, other electronic devices 120, 130, 140 and server 160 may have the same or similar internal configurations as those of the above-described electronic device 110 or server 150.

[0022] The electronic device 110 and the server 150 may include memory 211, 221, processors 212, 222, communication modules 213, 223, and input / output interfaces 214, 224. The memory 211, 221 may be a non-temporary computer-readable recording medium and may include non-temporary mass storage devices such as RAM (random access memory), ROM (read-only memory), disk drives, SSDs (solid-state drives), and flash memory. Here, non-temporary mass storage devices such as ROM, SSDs, flash memory, and disk drives may be included in the electronic device 110 and the server 150 as separate non-temporary storage devices distinct from the memory 211, 221. The memory 211, 221 may also store an operating system and at least one program code (for example, code for a browser installed and run on the electronic device 110, or code for an application installed on the electronic device 110 to provide a specific service). Such software components may be loaded from a computer-readable recording medium separate from the memory 211, 221. Such other computer-readable recording media may include computer-readable recording media such as floppy® drives, disks, tapes, DVD / CD-ROM drives, and memory cards. In other embodiments, software components may be loaded into memories 211, 221 through communication modules 213, 223 which are not computer-readable recording media. For example, at least one program may be loaded into memories 211, 221 based on a computer program (for example, the application described above) that is installed by a file provided over the network 170 by a file distribution system (for example, the server 160 described above) that distributes installation files for a developer or application.

[0023] The processors 212 and 222 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. The instructions may be provided to the processors 212 and 222 by the memories 211 and 221 or the communication modules 213 and 223. For example, the processors 212 and 222 may be configured to execute instructions received according to program codes recorded in a recording device such as the memories 211 and 221.

[0024] The communication modules 213 and 223 may provide a function for the electronic device 110 and the server 150 to communicate with each other via the network 170, or may provide a function for the electronic device 110 and / or the server 150 to communicate with other electronic devices (e.g., the electronic device 120) or other servers (e.g., the server 160). As an example, a request generated by the processor 212 of the electronic device 110 according to program codes recorded in a recording device such as the memory 211 may be transmitted to the server 150 via the network 170 under the control of the communication module 213. Conversely, control signals, instructions, contents, files, etc. provided under the control of the processor 222 of the server 150 may be received by the electronic device 110 through the communication module 213 of the electronic device 110 via the communication module 223 and the network 170. For example, the control signals, instructions, contents, files, etc. of the server 150 received through the communication module 213 may be transmitted to the processor 212 and the memory 211, and the contents and files, etc. may be recorded in a recording medium (the non-temporary recording device described above) that the electronic device 110 may further include.

[0025] The input / output interface 214 may be a means for interface with an input / output device 215. For example, an input device may include a keyboard, mouse, microphone, camera, etc., and an output device may include a display, speaker, haptic feedback device, etc. As another example, the input / output interface 214 may be a means for interface with a device that integrates input and output functions into one, such as a touchscreen. The input / output device 215 may consist of the electronic device 110 and one other device. Also, the input / output interface 224 of the server 150 may be a means for interface with an input or output device (not shown) that connects to or can be included in the server 150. As a more specific example, when the processor 212 of the electronic device 110 processes instructions for a computer program loaded into memory 211, a service screen or content configured using data provided by the server 150 or electronic device 120 may be displayed on the display via the input / output interface 214.

[0026] Furthermore, in other embodiments, the electronic device 110 and the server 150 may include more components than those shown in Figure 2. However, it is not necessary to explicitly show most of the conventional components in the figure. For example, the electronic device 110 may be implemented to include at least some of the input / output devices 215 described above, and may further include other components such as transceivers, cameras, various sensors, and databases. As a more specific example, if the electronic device 110 is an AI speaker, the electronic device 110 may be implemented to further include a variety of components that are generally included in an AI speaker, such as various sensors, camera modules, various physical buttons, buttons using a touch panel, input / output ports, and vibrators for vibration. (End of quote)

[0027] The following disclosure concerns a machine. According to at least one embodiment, the user terminal comprises a control unit, RAM, storage unit, graphics processing unit, communication interface, and interface unit, each connected by an internal bus.

[0028] According to at least one embodiment, the control unit consists of a CPU and ROM. The control unit executes programs stored in the storage unit and controls the user terminal. RAM is the work area of ​​the control unit. The storage unit is a memory area for saving programs and data. The control unit reads programs and data from RAM and processes them. By processing the programs and data loaded into RAM, the control unit outputs drawing commands to the graphics processing unit.

[0029] According to at least one embodiment, the graphics processing unit is connected to the display unit. The display unit has a display screen. When the control unit outputs a drawing command to the graphics processing unit, the graphics processing unit outputs a video signal for displaying an image on the display screen. Here, the display unit may be a touch panel equipped with a touch sensor. The touch panel of this display unit functions as an input unit.

[0030] According to at least one embodiment, the communication interface can be connected to a communication network wirelessly or via a wired connection, and can send and receive data with a server device via the communication network. The data received via the communication interface is loaded into RAM and processed by the control unit. External memory (e.g., an SD card) is connected to the interface unit.

[0031] According to at least one embodiment, the user terminal is not particularly limited as long as it is a computer device having a display screen and an input unit. Examples of user terminals include conventional mobile phones, tablet devices, smartphones, and desktop or notebook personal computers. A VR goggle, i.e., a screen (or two display panels, one for each eye) attached to a frame (or headset) that is fixed or attached to the head with a strap, may also be used. The user terminal has an audio output unit.

[0032] According to at least one embodiment, a user terminal can communicate with a server device via a communication network. It can transmit or receive information by establishing a communication connection via the communication network.

[0033] According to at least one embodiment, the server device comprises at least a control unit, RAM, a storage unit, and a communication interface, each connected by an internal bus.

[0034] According to at least one embodiment, the control unit consists of a CPU and ROM, executes programs stored in the storage unit, and controls the server device. The control unit also has an internal timer for timing. RAM is the work area of ​​the control unit. The storage unit is a memory area for saving programs and data. The control unit reads programs and data from RAM and performs program execution processing based on information received from the user terminal, etc.

[0035] This document discloses AI. According to at least one embodiment, artificial intelligence includes machine learning, deep learning, generative AI, large-scale language models (LLMs), foundational models, and generative AI. Generative AI uses transformers and employs numerous attention mechanisms. It uses self-supervised learning and Extract Prediction. In this case, the AI ​​can predict the next word. Given a sentence, it predicts the next word from the sentence up to that point. It generates a large number of supervised learning problems. These enable an AI that can predict the next word. Generative AI can predict grammatical structure, topic connections, and what kind of sentences people with a certain writing style are likely to write. Furthermore, by simply predicting the next sentence, generative AI can learn the underlying structure, causal relationships, and knowledge. Generative AI scales quickly, and its accuracy improves as the number of parameters increases. Ordinary statistics and machine learning overfit if the model parameters are too large compared to the data sample size. LLMs become more accurate as the number of parameters increases. One generative AI has 175 billion parameters. Generative AI is overlaid with supervised learning to facilitate smooth conversation. They are taught not to say anything strange. They write essays and act as call center operators.

[0036] According to at least one embodiment, Large Language Models (LLMs) are, in a non-inclusive sense, machine learning natural language processing models built using large datasets and deep learning techniques. Generally, they are adapted to various natural language processing (NLP) tasks such as text classification and generation, sentiment analysis, text summarization, and question answering using a technique called "fine-tuning," which involves training them on specific tasks. According to at least one embodiment, self-supervised learning is close to intrinsic human intelligence. When humans act, they are always predicting what will happen next and predicting the next input. In the process, they can learn the structure of the external world. Predicting the next word is considered intrinsic intelligence and is close to what is done in the cerebral cortex. According to at least one embodiment, Large Language Models memorize all the input information but generalize it to the extent necessary to predict the next word. They do not try to generalize all the information from the beginning. Large Language Models require capacity to remember information, and therefore require parameters. According to at least one embodiment, a large-scale language model incorporates eight models, each with 175 billion parameters or 220 billion parameters.

[0037] According to at least one embodiment, videos and images are represented as a collection of visual patches, which are small data units similar to text tokens in LLMs. Patches effectively represent models of visual data and are used as highly scalable and effective representations for training generative models on various types of videos and images. Videos are transformed into patches by first compressing the video into a low-dimensional latent space and then decomposing the representation into spatiotemporal patches.

[0038] According to at least one embodiment, a video compression network is a network that reduces the dimensionality of visual data, taking raw video as input and outputting a temporally and spatially compressed latent representation. An AI is trained in this compressed latent space and then generates video within this compressed latent space.

[0039] According to at least one embodiment, Spacetime Latent Patches extract a series of spatiotemporal patches that function as transformer tokens when given a compressed input video. The patch-based representation allows Sora to be trained on videos and images of varying resolutions, lengths, and aspect ratios, and controls the size of the resulting video by arranging randomly initialized patches into a grid of appropriate size during inference.

[0040] According to at least one embodiment, the AI ​​is a diffusion model that, given a noisy patch (and conditioning information such as text prompts) as input, is trained to predict the original "clean" patch. The AI ​​is a diffusion transformer, which exhibits remarkable scaling properties in various domains, including language modeling, computer vision, and image generation. Diffusion transformers are also effective as video generation models. The AI's sample quality improves significantly as the amount of training computation increases.

[0041] According to at least one embodiment, the AI ​​applies caption regeneration technology to train a highly descriptive caption model, which is then used to generate text captions for all videos in the training set. Training highly descriptive captions improves not only the overall quality of the generated videos but also the fidelity of the text. GPT is used to convert short user prompts into long, detailed captions, which are then sent to the model. This allows the AI ​​to generate high-quality videos that precisely follow the user prompts.

[0042] According to at least one embodiment, AI can perform vectorization in natural language processing according to the following flow. First, as a preprocessing step, the given text is cleaned. In the cleaning process, unnecessary words such as JavaScript code and HTML tags contained in the text are removed. These codes are used to display on the internet and are therefore not generally used in natural language processing. Next, the text is divided into words using morphological analysis. Morphological analysis is the classification of natural language sentences written in characters into the smallest meaningful linguistic units. "MeCab," "JUMAN," and "JANOME" can be used as morphological analysis tools. In normalization, words with the same meaning, such as variations in spelling, are unified into a single word. Stop words are words that are excluded from processing for reasons such as not being usable in natural language processing. Examples of stop words include particles and auxiliary verbs, which do not have meaning on their own. When calculating vectors, these may be removed, and only meaningful words may be targeted. Vectorization may also be performed without removing these stop words. Vectorization is the process of converting strings of words into vectors. Vectorization transforms word data into numerical data. When converting words to vectors, methods such as Bag of Words and distributed representations are used. Bag of Words is a method of vectorizing a given text using the number of occurrences of each word. It focuses on how often each word appears in the text, and does not consider the order of words or sentences. Distributed representation is a method of vectorizing by focusing on the meaning of words. By vectorizing the meaning of words, it is possible to assign similar vectors to words with similar meanings or usages, and the relationships between words can also be represented by vectors. With vector representation, it is possible to add and subtract the meanings of words. In applied processing, natural language converted into numerical data can be used as input for machine learning. Specifically, vectorized natural language is fed into a classifier to perform text classification.Tools used here include TensorFlow, scikit-learn, and PyTorch.

[0043] This document discloses information about sound. In at least one embodiment, sound includes music or sound effects. Music is a combination of length, pitch, intensity, timbre, etc., of sounds that expresses various emotions and stories, and includes singing, instrumental performances, and natural sounds. Music can be composed by determining a scale (key), chords (harmonies), or melody. In at least one embodiment, once a scale is determined, the chords are also determined according to its constituent notes. There are theories about chord progressions, and there are patterns that have been considered good progressions in songs to date. Chords greatly influence the mood of a song and can have a significant impact on a person's emotions and feelings at that time. Similarly, while melodies can not deviate too much if they are based on the constituent notes of chords, they tend to become monotonous, so some randomness may be necessary. By varying the notes within a range that does not deviate too much from the chords, monotony can sometimes be prevented.

[0044] This disclosure provides a method for creating sound, wherein the apparatus identifies motion features that characterize the movement of objects in a video (in this disclosure, the term “method” may be interpreted as “steps” unless otherwise explicitly contradicted). In at least one embodiment, a video is a sequence of still images that create movement or data. Videos may include sound, and the synchronization of the video and sound may result in richer expression. An object is data or related processing that is manipulated within a video. Examples of objects include moving objects in a video (including living and non-living things; non-inclusive examples of the former include people, people dancing, animals, plants swaying in the wind, etc.; non-inclusive examples of the latter include moving objects (such as cars), the ebb and flow of waves, flashing lights, changes in the wavelength or intensity of light, and digital representations drawn with computer graphics). In this disclosure, any object that a person can perceive as moving when viewing a video is considered to be this “object” or data corresponding to an “object”. In this disclosure, "operational features" may also mean "operational feature quantities" (unless otherwise explicitly stated throughout this disclosure). Feature quantities are numerical values ​​that represent the characteristics of data contained within a dataset. For example, when representing a person's face in a video, feature quantities could include "eye size," "nose height," and "skin color." In addition, when a person is dancing, the movement of organs (including hands, feet, and hips) can also be considered feature quantities. By providing these numerical values ​​to a computer, the computer can distinguish between people's faces, recognize specific individuals, or recognize dance choreography. Furthermore, in the case of tides, the way waves rise and the way they fall can naturally be considered feature quantities. In addition, in the case of computer graphics, the movement of lines or surfaces, and the way colors change, can naturally be considered feature quantities. Non-comprehensively, the types of feature quantities include the following: Numerical data is data that can be expressed numerically, such as height, weight, and age. Categorical data is data that can be expressed in categories, such as gender, nationality, and occupation. Text data refers to data that can be represented in text form, such as sentences, words, and keywords.Image data (including still images in videos) consists of features extracted from images, such as color, shape, and texture. Video data consists of features extracted from videos, such as color, shape, texture, and the presence or change of these over time. Audio data consists of features extracted from audio, such as pitch, frequency, and timbre.

[0045] In at least one embodiment, the device can be created by any of the methods, machines, or AI described above. In this disclosure, the term "device" may be replaced with "AI" unless otherwise expressly contradictory. The device analyzes the data of the target video. The device identifies objects in the video. The user can identify the target objects. In another embodiment, the device identifies objects that appear relatively frequently from among the objects present in the video. The device may also request the AI ​​to do so (throughout this disclosure, unless otherwise expressly contradictory, the phrase "the device performs a process" may be replaced with "the device requests the AI ​​to perform that process." In this case, the AI ​​may be one that has learned the process and judgment by the learning methods described above (including supervised learning)). In such cases, the target objects are not specified by the user, but are automatically identified by the device. The device identifies the motion characteristics of the objects. For example, in a video of a dancer dancing, the dancer waving their hands is a motion characteristic of an object. In this embodiment, the movements of the dancer's head, hips, feet, and fingers can naturally be used as motion features. As already explained, motion features can be identified by the user. Furthermore, the device can also identify motions that occur relatively frequently from among the movements of the object as motion features.

[0046] This disclosure describes a method for requesting an AI to generate music where the time between similar motion features is used as the unit time, and the unit time is used as the beat (time signature). In this disclosure, unless otherwise explicitly stated, the phrase "generate music" may be replaced with "generate sound." In at least one embodiment, a video contains similar motion features. As a non-exclusive example, in the case of dance, a dancer may repeatedly perform the same dance choreography. In this case, the device compares the motion features of each movement and recognizes them as similar motion features. There are two methods for determining similarity: one where the user defines the criteria, and another where the machine automatically determines the criteria. In the former, the device displays videos or still images of two or more motion features on the screen of the user's terminal. The user specifies to the device whether two or more videos or still images of motion features are similar or not. Based on this specification, the device determines whether the motion features are similar or not. In the latter, the device determines whether two or more videos or still images of motion features contain clearly different motion features. If no clear features or differences in feature quantities can be detected, these videos can be determined to be similar. In this determination, the AI ​​can learn about similarity through supervised learning. The device can also utilize the functions of the AI, or request the AI ​​to determine similarity and obtain the determination result.

[0047] The device identifies the time between motion features. As a non-inclusive example, in the case of dance, if a dancer repeats the same dance choreography (similar motion features), the device identifies the difference (or distance) between the appearance times (or filing times) of each motion feature in the video. As another non-inclusive example, suppose a video of a dance performance is 3 minutes long. If the dancer repeats a motion feature at 1 minute 10 seconds and again at 1 minute 15 seconds, the time between the motion features is 5 seconds, and the unit time is 5 seconds. The device recognizes the time between motion features as unit time. The device can transmit the unit time information to other devices or AI. In other embodiments, as described above, the device may use an AI that has learned to recognize unit time through supervised learning, or may request that the AI ​​to recognize unit time.

[0048] In at least one embodiment, a beat in sound includes a unit of rhythm. Non-exclusive examples include a fixed time interval (a basic unit in music that repeats at a fixed interval), a contrast between strong and weak beats (there are strong and weak beats, and this contrast generates rhythm), or the basis of tempo (the speed of the beat becomes the tempo, which determines the speed of the song). A beat may also be a time signature. A time signature is the basis of the rhythm of the whole song and indicates how many beats there are in a measure. For example, in 4 / 4 time, there are four beats in a measure, and each beat is of the same length. In this disclosure, the term "beat" may be replaced with "time signature" unless otherwise explicitly contradictory. The device requests the AI ​​to generate music in which a unit of time is a beat (time signature). Non-exclusive examples include, if a video of a dance performance is 3 minutes long and the unit of time is 5 seconds, the device requests the AI ​​to generate music in which 5 seconds is one beat. In other embodiments, the device may generate the music itself using the AI ​​as described above.

[0049] According to the above configuration, sound can be automatically created based on video. The created sound or music has a beat based on the features of the video, resulting in a high degree of matching between the video and the generated sound. This reduces the workload of the person creating the sound, and allows for the automatic generation of highly usable sound with a high degree of matching between the video and the generated sound, thus having industrial applicability.

[0050] This disclosure describes a method for requesting an AI to generate music with a beat unit, where the music is of the same length as the video (including the time over which motion features should be identified; in this disclosure, unless otherwise explicitly stated, "video time" can be understood as "video length" or "video duration"). The device requests the AI ​​to generate music of the same length as the video. As a non-exclusive example, if a video of a dance performance is 3 minutes long, the video is 3 minutes long, so the device requests the AI ​​to generate 3 minutes of music. In other embodiments, the device defines the time over which motion features should be identified as the video length. The time over which motion features should be identified is, in at least one embodiment, the time period in the video over which the device or AI is permitted to identify motion features, in whole or in part. There are two methods for determining the time over which motion features should be identified: a user-defined method and an automatic method by the device. In the former, the user can define the time over which motion features should be identified, as long as it is within the video length. As a non-inclusive example, if a video of a dance performance is 3 minutes long, the period from 0 minutes 0 seconds to 1 minute 30 seconds can be designated as the time for identifying the motion characteristics. The device will identify the motion characteristics only within the designated time period using the method described above. As a latter example, the device can also designate a predetermined portion of the video's duration as the time for identifying the motion characteristics. Naturally, the entire duration of the video can also be designated as the time for identifying the motion characteristics. The device is required to generate music of the same length as the time for identifying the motion characteristics.

[0051] This invention discloses a method for requiring a device to generate music such that the beats of the music are identical to the unit time in a video. The phrase "identical" above can be replaced with "synchronized". In at least one embodiment, the device requires the device to generate music that is the same length as the video and to generate music such that the unit time of the music is identical to the unit time in the video. As a non-exclusive example, suppose a video of a dance performance is 3 minutes long. If the dancer repeats a motion feature at 1 minute 12 seconds and 1 minute 17 seconds, the unit time is 5 seconds. Furthermore, the device requires the AI ​​to generate music with 5 seconds as one beat and to generate music that is 3 minutes long. Since the device requires the device to generate music such that the unit time of the music is identical to the unit time in the video, the generated sound will have beats that are identical (synchronized) to the beats at 1 minute 12 seconds and 1 minute 17 seconds in the video. In this case, if the generated music has beats in 5-second units of time starting from 0 minutes 2 seconds, it will be identical (synchronized) to the beats at 1 minute 12 seconds and 1 minute 17 seconds in the video. In other words, the device recognizes the beats of the music corresponding to the unit time in the video as specific beats, and requires that the music be generated with a defined overall beat based on (or in a manner consistent with) these specific beats. Taking the previous example, the beats at 1 minute 12 seconds and 1 minute 17 seconds correspond to the specific beats, and the music is generated with a defined overall beat based on these specific beats.

[0052] According to the above configuration, it is possible to automatically create sound that is synchronized with the beat of a video. The created sound or music not only has a beat based on the video's features, but is also at least synchronized with the video's beat, resulting in a higher degree of matching between the video and the generated sound. In this case, the generated sound can be used industrially as is by playing the video and the generated sound simultaneously. This reduces the workload of sound creators and has industrial potential because it allows for the automatic generation of highly usable sound with a high degree of matching between the video and the generated sound.

[0053] This invention discloses a method for requesting an AI to generate music with beats based on unit time, by requesting the AI ​​to select a relatively short unit time from among two or more unit times arising from similar motion features as the beat. In at least one embodiment, the device may recognize three or more similar motion features from a video. In this case, the device may recognize two or more unit times. The device recognizes the shortest unit time from among two or more unit times arising from similar motion features as the unit time to be used as the beat. The device can transmit such a unit time to be used as the beat to another device or AI, or request the AI ​​to recognize or identify such a unit time to be used as the beat. As already explained, the AI ​​learns to use the shortest unit time as the beat through supervised learning using a video having two or more unit times arising from similar motion features. As a non-exclusive example, suppose a video of a dance performance is 3 minutes long. For example, the A section, B section, and chorus are commonly used in popular songs such as pop and rock, and each is composed of a different melody. In music videos, the structure is typically a combination of verse A, chorus, pre-chorus B, and chorus, with similar choreography for each chorus. Let's assume the dancers repeat a specific movement at 1 minute 12 seconds, 1 minute 17 seconds, 2 minutes 12 seconds, and 2 minutes 17 seconds. The device will recognize four different time units: 5 seconds, 55 seconds, 1 minute, and 1 minute 5 seconds. In this case, the device will recognize 5 seconds as the shortest possible time unit. The device will then recognize this shortest time unit as a beat, or request the device or AI to recognize the shortest time unit as a beat.

[0054] With the above configuration, even in complex videos where similar behavioral features are scattered throughout, it is possible to automatically create sound that is synchronized with the beat of the video. The created sound or music not only has a beat based on the video's features, but is also at least synchronized with the video's beat, resulting in a higher degree of matching between the video and the generated sound. In this case, the generated sound can be used industrially as is by playing the video and the generated sound simultaneously. This has industrial potential because it reduces the workload of sound creators and enables the automatic generation of highly usable sound with a high degree of matching between the video and the generated sound.

[0055] This invention discloses a method for requesting an AI to generate music with beats based on unit time, by requesting that it select a unit time that appears relatively frequently from among two or more unit times arising from similar motion features to be used as a beat. In at least one embodiment, the device calculates the frequency of occurrence for the recognized unit time. Frequency of occurrence includes a numerical value indicating how many times a certain event or object appears within a specific period or range. As an example, frequency of occurrence includes events (specific words, numbers, actions, phenomena, etc.), ranges (specific sentences, datasets, time, space, etc.), and the frequency of occurrence of a word in a sentence (how many times the word "cat" appears in a sentence, etc.). The device calculates the frequency of occurrence of unit time within the time range of the video or the time for which motion features should be identified. As a non-exclusive example, suppose a video of a dance performance is 3 minutes long, and the dancer's choreography is recognized as a motion feature. Suppose the device recognizes that there are 50 units of 5 seconds, 10 units of 10 seconds, 3 units of 20 seconds, and 1 unit of 1 minute. In this case, the device recognizes the 5-second interval, which appears most frequently, as the unit of time to be considered a beat. The device then generates music using this unit of time as a beat, or requests the AI ​​to generate it in this way.

[0056] As described above, this configuration allows for the automatic creation of sounds synchronized with the beats of a video, even in complex videos where similar behavioral features are scattered throughout. Furthermore, since the most frequently occurring unit of time is used as the beat, the beats of the generated music are less likely to be inconsistent with the beats of the video. Therefore, the generated sound or music not only has beats based on the video's features, but is also at least synchronized with the video's beats, resulting in a higher degree of matching between the video and the generated sound. In this case, the generated sound can be used industrially as is by playing the video and the generated sound simultaneously. This reduces the workload of sound creators and allows for the automatic generation of highly usable sounds with a high degree of matching between the video and the generated sound, thus offering industrial potential.

[0057] This invention discloses a method for requesting an AI to generate music with a unit time of beat, wherein the AI ​​generates music of the same duration as the time of the video or the time of the motion feature to be identified, and when two or more unit times based on dissimilar motion features are identified, the AI ​​generates two or more pieces of music and concatenates them. In at least one embodiment, the device recognizes a unit time based on similar motion features, as already described. In this case, the device recognizes a unit time based on motion feature A and a unit time based on motion feature B. These are two or more unit times based on dissimilar motion features. When two or more unit times based on dissimilar motion features are identified, the device generates two or more pieces of music. In other embodiments, the AI ​​or other device is requested to generate two or more pieces of music in this manner. The device concatenates these generated pieces of music. Concatenation includes a state in which multiple melodies are connected. The concatenation may be a smooth and natural flow. As a non-exclusive example, the device generates music A, generated from a video (including a music video or a movie) with a unit time based on motion feature A, and music B, generated from a unit time based on motion feature B. The device links music A and music B. One method of linking them is to link the music data so that when one piece of music ends, the other piece of music can be played simultaneously. Another method is to link a portion of the music data with a portion of the data of another piece of music so that one piece of music ends midway and another piece of music is played. In at least one embodiment, when two or more pieces of music are generated, music 2 can be obtained by modulating music 1.

[0058] According to the above configuration, even for complex videos where two or more unit times are identified based on dissimilar motion features, it is possible to automatically create sound that synchronizes with the beat of the video. Furthermore, because the sound is based on two or more unit times, the possibility of mismatch with the video's beat is further reduced. Therefore, the created sound or music is largely synchronized with the video's beat, resulting in a higher degree of match between the video and the generated sound. In this case, the generated sound can be used industrially as is by playing the video and the generated sound simultaneously. This has industrial potential because it reduces the workload of sound creators and enables the automatic generation of highly usable sound with a high degree of match between the video and the generated sound.

[0059] The present invention discloses a method that requires the music to be concatenated at points in time identified based on the frequency of occurrence of two or more behavioral features. In at least one embodiment, the device generates music of the same duration as the video duration or the duration for which the behavioral features should be identified, and identifies points in time when the unit time changes for two or more unit time segments. As a non-exclusive example, the device calculates the frequency or probability of occurrence of each unit time segment based on each behavioral feature within the video's time frame. For example, within the video duration or the duration for which the behavioral features should be identified, the device identifies time segments where the occurrence of a unit time segment based on behavioral feature A is relatively high, and time segments where the occurrence of a unit time segment based on behavioral feature B is relatively high. Based on these frequencies, the device generates music based on the unit time segment with the highest frequency. If the unit time segment with the highest frequency changes, the device generates music based on the new unit time segment. The device concatenates these two pieces of music. In another embodiment, the device generates music of the same duration as the video duration (the duration for which the behavioral features should be identified), and generates two or more pieces of music when two or more dissimilar behavioral features are identified. The device identifies a point in time (in this paragraph, unless explicitly contradictory, the terms "point in time" or "time" can be replaced with "time range") determined based on the frequency of occurrence of two or more dissimilar behavioral features as the change time. For example, within the video duration or the time range for which behavioral features should be identified, the device identifies a time range where unit time based on behavioral feature A occurs relatively frequently, and a time range where unit time based on behavioral feature B occurs relatively frequently. The device can identify the point in time when the most frequently occurring behavioral feature changes, or a point in time between or approximately midway through the time ranges where different behavioral features occur frequently, as the change time. As a non-inclusive example, suppose a video of a dance performance is 3 minutes long. If, in the time range from 0 minutes 0 seconds to 1 minute 30 seconds, unit time based on behavioral feature A occurs relatively frequently, and in the time range from 1 minute 30 seconds to 3 minutes, unit time based on behavioral feature B occurs relatively frequently, the device identifies 1 minute 30 seconds, which is approximately midway between these two time ranges, based on the frequency of occurrence of the two or more behavioral features. The device will request that the music be linked once it has been identified.In other words, the generated music consists of two parts: from 0 minutes 0 seconds to 1 minute 30 seconds, it is music A based on operational characteristic A; and from 1 minute 30 seconds to 3 minutes, it is music B based on operational characteristic B. These two parts are joined together at 1 minute 30 seconds.

[0060] As described above, with this configuration, even for complex videos where two or more units of time are identified based on dissimilar motion features, it is possible to automatically create sound that synchronizes with the beat of the video. Furthermore, because the sound is based on two or more units of time, the possibility of mismatch with the video's beat is further reduced. Therefore, the created sound or music is largely synchronized with the video's beat, resulting in a higher degree of match between the video and the generated sound. In this case, the generated sound can be used industrially as is by playing the video and the generated sound simultaneously. This has industrial potential because it reduces the workload of sound creators and enables the automatic generation of highly usable sound with a high degree of match between the video and the generated sound.

[0061] In at least one embodiment, the video includes films, dances, and choreography. For example, it includes videos published on YouTube®. In this case, music can be automatically generated to match the published video. The video also includes live-streamed videos. In this case, the device generates music simultaneously with the live stream, according to the conditions or configuration described in any of the disclosures. The device can deliver this generated music simultaneously with the live-streamed video. According to the above configuration, music that matches the beat of the video can be provided simultaneously with the live stream. Such a highly flexible generation method is superior to conventional technologies and has industrial applicability, as it is something that human composers cannot do. In at least one embodiment, the device can request the AI ​​to analyze ambient sounds and audience voices during streaming in real time, recognize one or more of them as specific sounds, features, or feature quantities, and generate sounds or sound effects corresponding to them.

[0062] In at least one embodiment, the video includes a video used in VR or AR. VR (Virtual Reality) is a technology that provides an experience of being completely immersed in a virtual world. By wearing a device such as VR goggles, users can feel as if they are in another world. AR (Augmented Reality) is a technology that overlays digital information onto the real world. Through a smartphone camera, virtual objects can be displayed on real-world scenery, or information can be added. The device can generate music for a video presented to the user through VR or AR, in accordance with the conditions or configuration described in any of the disclosures. The device can output the generated music as audio at the same time as it is presented to the user through VR or AR. The audio output includes embodiments where the audio is output from a user terminal. According to the above configuration, music that matches the beat of the VR or AR video can be provided. This has industrial applicability in that it reduces the workload of the sound creator, or enables the automatic generation of highly usable sound with a high degree of matching between the video and the generated sound.

[0063] In at least one embodiment, the AI ​​repeatedly learns to predict the next note using unsupervised learning. As already explained, the composition process is hierarchical according to its structure. The elements that make up music are at least a scale (key), chords (harmonies), and melody. Once the scale is determined, the chords are also determined according to its constituent notes. There is a theory to chord progressions, and there are many patterns that have been accumulated that are considered good progressions in songs to date. Chords greatly influence the mood of a song and are said to have a significant impact on a person's emotions and feelings at that time. The AI ​​repeatedly learns to predict the next note using unsupervised learning based on music data. Furthermore, the AI ​​performs supervised learning on the music data to determine what kind of emotions the music expresses.

[0064] In at least one embodiment, the device or AI generates music based on music or related information that has already been uploaded to the internet or recorded in a database as completed, regardless of whether the author is registered or not. As a non-exclusive example, given that the unit time based on operational feature A is 5 seconds, the device identifies music that has a beat of 5 seconds and has already been uploaded to the internet as completed. If multiple pieces of music are identified, the device identifies the music that best matches any of the conditions described in this disclosure. The device or AI generates music by, or by modifying, the identified music.

[0065] In at least one embodiment, the device or AI identifies one piece of music based on music or related information that has already been uploaded to the internet or recorded in a database as completed, regardless of whether the author is registered or not. In this embodiment, the device or AI is configured to identify existing music that meets certain criteria, rather than generating music. The device presents the identified music or information about music to the user.

[0066] In at least one embodiment, the device or AI generates music using music or related information that has already been uploaded to the internet or recorded in a database as completed, regardless of whether the author is registered or not.

[0067] In at least one embodiment, the device adds data indicating that the music was generated by AI to the music data generated by AI. When providing a video to a user along with the generated music, if the device recognizes that the music contains such data, it informs the user that the music was generated by AI, or displays this information on the user's terminal. In another embodiment, when the device generates music using music that has already been uploaded to the internet or recorded in a database as completed, it adds information about the name of the source music or the copyright holder to the generated music data. If the device recognizes that the music contains such information, it informs the user that the music was created by the copyright holder, or that the music was generated by AI based on that music, or displays this information on the user's terminal. In at least one embodiment, the device generates data or prompts that display copyright notices in a format compatible with the distribution platform. For example, on YouTube®, it generates data or prompts that display copyright notices as defined by the platform, or generates such data accompanying the generated video. These may be referred to as copyright management tools. The above configuration has industrial applicability because it can reduce the risk of copyright infringement in music data added or generated by AI.

[0068] In at least one embodiment, the user can specify a sound to be relied upon when generating music. As an example, the device accepts sound data to be relied upon. The device has two or more sound sources that the user can specify, and the user can select any sound source. "Having two or more sound sources recorded" includes not only embodiments in which the device itself stores these sound sources, but also embodiments in which the device obtains these sound sources from an external storage medium. For example, these sound sources are pre-existing music. The user selects one or more sound data to be relied upon, and the device generates music based on the selected music. As a non-exclusive example, if the video has a unit time of 5 seconds and the selected sound data is pre-existing music with 7 beats, the device adjusts the playback speed of the pre-existing music to be relied upon to change it to music with 5 beats. In this example, the changed music corresponds to the "generated music" in the above embodiment. According to the above configuration, pre-existing music that synchronizes with the beat of the video can be automatically identified. Furthermore, since the identified pre-existing music is a sound based on a unit time, it matches the beat of the video. In this case, the presented sound can be used industrially as is by playing the video and the generated sound simultaneously. This reduces the workload of the sound creator, alleviates the burden of searching for music with matching beats, and automatically provides highly usable sounds with a high degree of matching between the video and the generated sound, thus having industrial applicability. In these embodiments, the device can generate or store data or prompts for license indications regarding the pre-existing sounds used within the generated music data or attached to the data. The advantage of such embodiments is that they can provide a system that facilitates copyright management and licensing.

[0069] In at least one embodiment, the device analyzes motion features in a video (for example, dance movements) in more detail and generates different musical elements (beats and melodies) for each motion feature (for example, dance steps). These musical elements are employed or linked together based on any of the methods of this disclosure. These musical elements may be added to the generated music, provided that they do not conflict with the beats of the generated music.

[0070] In at least one embodiment, the device recognizes a predetermined motion characteristic of an object in a video as a specific motion characteristic and generates a specific sound, which is a predetermined sound, at the time the specific motion characteristic occurs. The device can generate the specific sound in addition to the music to be generated. In another embodiment, the device can generate the specific sound independently of the music to be generated. The user registers information about specific motion characteristics or specific sounds with the device. As a non-exclusive example, the user registers a person's surprised expression or a predetermined reaction as a specific motion characteristic, and an expression of surprise or a prompt as information about a specific sound. Specific sounds include pre-existing music and sound effects specified by the user. The device can store or receive two or more such pre-existing music and sound effects. The device can present two or more such available specific sounds to the user. The user can specify which specific sounds to use from among them. If two or more specific sounds are specified, the device uses them alternately, randomly, or, according to the embodiments of this disclosure, as the specific sounds to be generated. If the video is a two-hour movie, and at 1 hour, 20 minutes, and 43 seconds, a character in the movie shows surprise, the device generates a specific sound representing surprise at that point. According to the above configuration, a specific sound based on a specific action feature can be automatically generated. Furthermore, since the generated specific sound is based on the time the specific action feature occurred, it matches the timing of the specific action feature in the video. In this case, the presented sound can be used industrially as is by playing the video and the generated sound simultaneously. This reduces the workload for those creating the specific sound, and allows for the automatic provision of highly usable sounds with a high degree of matching between the video and the generated sound, thus offering industrial applicability. As already explained, this configuration can be combined with or replaced with one or more embodiments of other embodiments disclosed herein, as long as they do not contradict each other. That is, it is also possible to analyze ambient sounds and audience voices during distribution in real time and generate background music and sound effects (specific sounds) accordingly. In other embodiments, it can be implemented as an application that automatically inserts a surprised sound effect in response to a surprised action.

[0071] In at least one embodiment, the video includes a movie. According to this embodiment, optimal music or specific sounds can be generated separately for each sequence of the movie.

[0072] We will now disclose an embodiment different from the one described above.

[0073] This disclosure describes a method by which a device identifies features of objects in a video and requests an AI to generate music based on those features. This method includes identifying motion features, which are characteristics of actions, and requesting an AI to generate music based on those motion features. According to at least one embodiment, the device or AI represents a video or image as a collection of visual patches, which are small data units similar to text tokens in LLM. The device combines a video recognition AI with LLM to understand each scene of the video. Specifically, it utilizes the video recognition AI to individually recognize various objects and environments that make up the scene, such as people, cars, buildings, animals, natural objects such as trees, and weather, as well as changes in them. The AI ​​learns emotions corresponding to the visual patches by the learning methods described above (including supervised learning). Furthermore, the AI ​​may have the LLM pre-tuned using sample videos of the subject area. For example, the AI ​​learns to correspond to a mood or atmosphere or emotion (collectively referred to as "concept" in this disclosure) for a particular visual patch. Based on the motion features, the device requests an AI to generate music based on those features. For example, if the video is a 3-minute music video and the dancer's expression is smiling, the device identifies behavioral characteristics such as bright and cheerful, and requests the AI ​​to generate music based on those characteristics. In another embodiment, the device also requests the AI ​​to identify behavioral characteristics, which are the characteristics of the movement of objects in the video. In yet another embodiment, if the video is a 3-minute music video and the background behind the dancer is a ruined building for dramatic effect, the device identifies auxiliary characteristics such as ruins and darkness, and requests the AI ​​to generate music based on those auxiliary characteristics. In yet another embodiment, the video is a film, and the device can analyze the flow of scenes and emotions in the video and generate a corresponding musical structure (intro, climax, ending). Furthermore, the concept-based music includes embodiments corresponding to music genres (classical, EDM, hip hop, etc.). According to this embodiment, music that matches the taste or concept of the video can be automatically generated.This has industrial potential because it reduces the workload of those who create the sound, and allows for the automatic creation of highly usable sounds that have a high degree of conceptual consistency with the video.

[0074] In at least one embodiment, the device may also use user-specified features as operational features or auxiliary features. The user inputs a concept for the music to be generated into the device in text. These characters may be prompts. The device requests the AI ​​to generate music based on the user-inputted concept. Furthermore, the device may request the AI ​​to generate music that is the same length as the video or the time at which the operational features should be identified. In one embodiment, the user can select a mood or atmosphere, and music matching the selected mood can be generated. According to this embodiment, since the user can specify their preferences, music that matches the user's preferences can be automatically generated. This has industrial applicability in that it reduces the workload of the sound creator or automatically provides highly usable sound with a high degree of conceptual agreement between the video and the generated sound.

[0075] In at least one embodiment, the device analyzes lyrics as features specified by the user. The device or AI recognizes concepts contained in the lyrics as features. For example, the device or AI extracts concepts such as love, heartbreak, and sadness based on the lyrics. The device or AI generates sound based on the extracted concepts. If two or more concepts are recognized based on the lyrics, the AI ​​is requested to base the generation on the feature with the highest frequency of occurrence, as described later. This method allows music to be generated based on the main lyric concepts, even with complex lyrics. This has industrial potential because it reduces the workload of sound creators and automatically provides highly usable sound with a high degree of conceptual match between the video and the generated sound.

[0076] In at least one embodiment, the device requests the AI ​​to convert objects in a video into text. As already described, the AI, or the device, represents the video or image as a collection of visual patches, which are small data units similar to text tokens in LLM. The device combines the video recognition AI and LLM to understand each scene in the video. Specifically, it utilizes the video recognition AI to individually recognize various objects and environments that make up the scene, such as people, cars, buildings, animals, natural objects such as trees, and weather, as well as their changes. The AI ​​learns the characters corresponding to the visual patches by the learning methods described above (including supervised learning). In this way, the AI ​​can describe (convert to text) a given video into text. The AI ​​generates music based on the converted text. Specifically, the device requests the AI ​​to identify features based on the converted text and generate music based on the features. The method for generating music based on features is disclosed elsewhere in this disclosure.

[0077] This document discloses a method for requesting an AI to generate music based on features, where, if there are two or more features, the AI ​​is required to base the music on the feature that is relatively longer in duration. In at least one embodiment, the device may recognize two or more features. In this case, the AI ​​is required to base the music on the feature that is longest in duration in the video. The device identifies the duration for which each feature is recognized. As a non-inclusive example, if the video is a 3-minute movie, and 2 minutes and 40 seconds of it depict a breakup, but 20 seconds depict a fun and exciting time in love as a flashback, then the "breakup" feature is 2 minutes and 40 seconds, and the "fun" feature is 20 seconds. In other words, the device requests the AI ​​to create music based on the longest feature, the "breakup" feature. Note that this method is not intended to be limited to one feature; it is also possible to generate music based on two or more features. According to the above configuration, even if a video has multiple concepts, music can be generated based on the main concept. Therefore, even if a video contains conflicting concepts, the music is based on the main concept, so there is a high probability that the concepts of the video and the music will match. This has industrial potential because it reduces the workload of those who create the sound, and allows for the automatic creation of highly usable sounds that have a high degree of conceptual consistency with the video.

[0078] This document discloses a method for requesting an AI to generate music based on features, where, if there are two or more features, the AI ​​is required to base the music on the feature with the relatively higher frequency of occurrence. As already explained, in at least one embodiment, the device may recognize two or more features. In this case, the AI ​​is required to base the music on the feature that appears most frequently in the video. The device identifies the number of times each feature is recognized. As a non-inclusive example, if the video is a 3-minute movie in which the actors laugh 15 times, cry once, and are angry once, the most frequent feature is "laughing." In other words, the device requests the AI ​​to create music based on the longest-running feature, "laughing." Note that this method is not intended to be limited to one feature; it is also possible to generate music based on two or more features. According to the above configuration, even if a video has multiple concepts, music based on the main concept can be generated. Therefore, even if a video contains conflicting concepts, the music is based on the main concept, so there is a high probability that the concepts of the video will match. This has industrial potential because it reduces the workload of those who create the sound, and allows for the automatic creation of highly usable sounds that have a high degree of conceptual consistency with the video.

[0079] This invention discloses a method for requesting an AI to generate music based on features, wherein the AI ​​generates music for the same duration as the video's duration or the duration of the motion features to be identified, and if two or more dissimilar features are identified, it generates two or more pieces of music and concatenates them. As already explained, the device or AI can generate two or more pieces of music when two or more features are identified. Also, as already explained, the device or AI can concatenate two or more pieces of music. With the above configuration, even for complex videos with two or more dissimilar features, it is possible to automatically create sounds that are similar to the concept of the video. Furthermore, because the sounds are based on two or more concepts, the possibility of inconsistency with the video's concept is further reduced. Therefore, since the created sounds or music are largely synchronized with the video's concept, the degree of agreement between the video and the generated sounds is higher. In this case, the generated sounds can be used industrially as is by playing the video and the generated sounds simultaneously. This reduces the workload of sound creators and has industrial applicability in that it can automatically generate highly usable sounds with a high degree of conceptual agreement between the video and the generated sounds.

[0080] The present invention discloses a method that requires the music to be concatenated at a point in time identified based on the frequency of occurrence of two or more features. As already described, the device can calculate the frequency or probability of occurrence of each feature within the time frame of the video. For example, within the time frame of the video or the time frame in which the motion features should be identified, the device identifies a time frame in which the occurrence of a unit time based on feature A is relatively high, and a time frame in which the occurrence of a unit time based on feature B is relatively high. Based on these frequencies, the device generates music based on the unit time with the highest frequency of occurrence. If the unit time with the highest frequency of occurrence changes, the device generates music based on the new unit time. The device concatenates these two pieces of music. In another embodiment, the device generates music for the same time frame as the time frame of the video (the time in which the motion features should be identified), and generates two or more pieces of music when two or more dissimilar motion features are identified. The device identifies the point in time identified based on the frequency of occurrence of two or more dissimilar motion features as the change time. For example, the device identifies time periods within the time frame where the duration or operational features of a video should be identified, where the occurrence of unit time based on feature A is relatively high, and where the occurrence of unit time based on feature B is relatively high. The device can identify the point in time when the most frequently occurring operational feature changes, or a point in time between or approximately midway through time periods where different operational features are frequently occurring, as the change time. As a non-inclusive example, suppose a movie video is 10 minutes long. If, in the time period from 0 minutes 00 seconds to 1 minute 30 seconds, the occurrence of unit time based on feature A (for example, the feature that the setting is in a ruin) is relatively high, and in the time period from 1 minute 30 seconds to 3 minutes, the occurrence of unit time based on feature B (for example, the feature that the setting is in a magnificent building (such as a castle)) is relatively high, the device identifies 1 minute 30 seconds, which is approximately midway between the two or more operational features, based on their respective frequencies. The device then requests that the music be linked at the identified point in time. In other words, the generated music consists of two parts: from 0 minutes 0 seconds to 1 minute 30 seconds, it is music A based on characteristic A; and from 1 minute 30 seconds to 3 minutes, it is music B based on characteristic B. These two parts are joined together at 1 minute 30 seconds.

[0081] As described above, this configuration allows for the automatic creation of sounds similar to the video's concept, even for complex videos where two or more units of time are identified based on dissimilar features. Furthermore, because the sounds are based on two or more units of time, the possibility of mismatch with the video's concept is further reduced. Therefore, the created sounds or music are largely synchronized with the video's concept, resulting in a higher degree of match between the video and the generated sounds. In this case, the generated sounds can be used industrially as is by playing the video and the generated sounds simultaneously. This has industrial potential because it reduces the workload of sound creators and enables the automatic generation of highly usable sounds with a high degree of conceptual match between the video and the generated sounds.

[0082] In at least one embodiment, when generating music based on lyrics or text, the device recognizes characters or concepts indicating rhythm within the text and requests the AI ​​to generate music with a rhythm based on those characters or concepts. With the above configuration, music with a specified rhythm can be generated, further reducing the possibility of the generated music being inconsistent with the concept of the video. Therefore, the generated sound can be used industrially as is when the video and the generated sound are played simultaneously. This has industrial applicability in that it reduces the workload of those who create the sound, or in that it can automatically generate highly usable sound with a high degree of conceptual agreement between the video and the generated sound.

[0083] A different embodiment from the above is disclosed. The device requests the AI ​​to create lyrics based on music data. The device or AI breaks down the music data into units such as scales, chords, or melodies. The AI ​​learns the mood, atmosphere, or emotion represented by each unit through supervised learning. Furthermore, the AI ​​learns and fine-tunes using already published music scales, chords, or melodies and lyrics combinations. This allows the AI ​​to predict what characters or lyrics will follow given a scale, chords, or melody. It can also predict what word will follow the characters or lyrics that have been received. In this way, the AI ​​can create lyrics based on music data. The device requests the AI ​​to create lyrics based on music data. According to the above configuration, since lyrics based on traditional musical concepts can be generated, the generated lyrics are less likely to be inconsistent with the musical concept. Furthermore, since the AI ​​performs natural language processing and learning, the likelihood of the generated lyrics containing grammatical errors is indeed low. Therefore, the generated lyrics can be used industrially as is, along with the music. This has industrial applicability in that it reduces the workload of those who create lyrics, or in that it can automatically generate highly usable lyrics with a high degree of conceptual agreement between the music and the generated lyrics. In at least one embodiment, the device further requests the AI ​​to generate a singing voice based on the generated lyrics. In at least one embodiment, the device requests the AI ​​to generate music, requests the AI ​​to create lyrics based on the generated music, requests the AI ​​to generate a singing voice based on the generated lyrics, and requests the AI ​​to combine (integrate, combine) the generated singing voice and music data (or the device itself combines the data). Vocaloid® is included as an application of the AI ​​for generating the singing voice. This method has industrial applicability in that it can automatically generate a singing voice that matches the concept of the video.

[0084] The following is an overview of the embodiments described above.

[0085] A method for creating sound, The device, Identify the behavioral features of objects in the video, The AI ​​is asked to generate music where the time between similar behavioral features is used as the unit of time, and each unit of time is used as a beat. method.

[0086] When requesting an AI to generate music where each unit of time is a beat, Generate music that is the same length as the time in the video or the time at which the motion characteristics should be identified. This requires generating music so that the unit time within the video matches the beat of the music. method.

[0087] When requesting an AI to generate music where each unit of time is a beat, The system requires that a relatively shorter unit of time be selected from two or more unit times arising from similar operational characteristics. method.

[0088] When requesting an AI to generate music where each unit of time is a beat, From among two or more unit times arising from similar operational characteristics, the unit time that has a relatively high frequency of occurrence of a specified unit time is designated as a beat. method.

[0089] When requesting an AI to generate music where each unit of time is a beat, Generate music that is the same length as the time in the video or the time at which the motion characteristics should be identified. When two or more unit times with dissimilar operational characteristics are identified, two or more pieces of music are generated. Request to link the music in question. method.

[0090] In the above method, The system requests that the music be linked together at a point determined based on the frequency of occurrence of two or more behavioral features. method.

[0091] A method for creating sound, The device, Identify the features (feature quantities) of objects in the video, The AI ​​is asked to generate music based on its characteristics. method.

[0092] When requesting an AI to generate music based on its characteristics, When there are two or more features, the AI ​​is required to prioritize the feature that is observed for a relatively longer period of time. method

[0093] When requesting an AI to generate music based on its characteristics, When there are two or more features, the AI ​​is required to base its selection on at least the feature that appears relatively frequently. method

[0094] When requesting an AI to generate music based on its characteristics, Generate music of the same duration as the time or characteristic of the video that should be identified. When two or more dissimilar features are identified, two or more pieces of music are generated. Request to link the music in question. method.

[0095] In the above method, The system requests that the music be linked together once it has been identified based on the frequency of occurrence of two or more features. method.

[0096] In at least one embodiment, the statement "requiring the AI ​​to select a relatively shorter unit time from among two or more unit time periods arising from similar operational features" can be replaced with "requiring the AI ​​to select the shortest unit time from among two or more unit time periods arising from similar operational features." The statement "requiring the AI ​​to select a unit time from among two or more unit time periods arising from similar operational features that has a relatively high frequency of occurrence of a specified unit time" can be replaced with "requiring the AI ​​to select a unit time from among two or more unit time periods arising from similar operational features that has the highest frequency of occurrence of a specified unit time." The statement "when there are two or more features, requiring the AI ​​to base its selection on the feature with a relatively long duration" can be replaced with "when there are two or more features, requiring the AI ​​to base its selection on the feature with the longest duration of occurrence of a specified feature." The statement "when there are two or more features, requiring the AI ​​to base its selection on the feature with a relatively high frequency of occurrence of a specified feature" can be replaced with "when there are two or more features, requiring the AI ​​to base its selection on the feature with the highest frequency of occurrence of a specified feature."

[0097] A method for creating sound, The device, Generate music that is the same length as the video or a specified length. The AI ​​is asked to generate music based on the characteristics of the lyrics corresponding to the video. method.

[0098] In the above configuration, the specified time includes the time specified by the user. The device requests the AI ​​to generate music within the time range specified by the user. The method of generating music based on the characteristics of the lyrics and its advantages are as described above.

[0099] In at least one embodiment, the user can make fine adjustments (tempo, key, effects) to the music or vocals generated by the AI ​​through the device. The device modifies the tempo, key, effects, etc., of the generated music or vocal data according to the user's adjustments. Such adjustments can be made even if the video is a real-time video or a streamed video.

[0100] In at least one embodiment, two or more users can collaboratively input rhythms and concepts to be generated, and make fine adjustments to the music or vocals generated by the AI, through the device. The device accepts information input from two or more users. Two or more users can collaboratively combine data through the device. This editing can be performed even if the video is a real-time video or a streamed video. In a non-exclusive example, this editing function is referred to as "collaboration mode" or "real-time mode".

[0101] In at least one embodiment, the device requests the AI ​​to create lyrics in one or more foreign languages, depending on the lyrics or concept. The device requests the AI ​​to create music corresponding to the input foreign language lyrics or concept. The AI ​​fine-tunes the concept and music according to the foreign language. As a non-exclusive example, it fine-tunes concepts related to ethnic music, such as the koto for the concept of Japan, or the bagpipes for the concepts of Europe or Ireland. The device also requests the AI ​​to generate music in the foreign language concept when foreign language lyrics are input.

[0102] In at least one embodiment, the device requests the AI ​​to generate music or video in a compatible format so that the generated music or video will work on major distribution platforms (non-exclusively, YouTube, TikTok, Instagram, etc., all registered trademarks). In other embodiments, the device converts the music or video generated by the AI ​​into a compatible format so that it will work on major distribution platforms. The advantage of such embodiments is that they create platform-compatible data and improve user convenience.

[0103] The invention disclosed herein only needs to achieve at least one of the effects described above.

Claims

1. A method for creating sound, The device, Identify the behavioral features of objects in the video, The AI ​​is asked to generate music where the time between similar behavioral features is used as the unit of time, and each unit of time is used as a beat. method.

2. The method according to claim 1, When requesting an AI to generate music where each unit of time is a beat, Generate music that is the same length as the time in the video or the time at which the motion characteristics should be identified. This requires generating music so that the unit time within the video matches the beat of the music. method.

3. The method according to claim 1, When requesting an AI to generate music where each unit of time is a beat, The system requires that a relatively shorter unit of time be selected from two or more unit times arising from similar operational characteristics. method.

4. The method according to claim 1, When requesting an AI to generate music where each unit of time is a beat, From among two or more unit times arising from similar operational characteristics, the unit time that has a relatively high frequency of occurrence of a specified unit time is designated as a beat. method.

5. A method for creating sound, The device, Identify the features of objects in the video, The AI ​​is asked to generate music based on its characteristics. method.

6. The method according to claim 5, When requesting an AI to generate music based on its characteristics, When there are two or more features, the AI ​​is required to prioritize the feature that is observed for a relatively longer period of time. method.

7. The method according to claim 5, When requesting an AI to generate music based on its characteristics, When there are two or more features, the AI ​​is required to base its selection on at least the feature that appears relatively frequently. method

8. The method according to claim 5, When requesting an AI to generate music based on its characteristics, Generate music of the same duration as the time or characteristic of the video that should be identified. When two or more dissimilar features are identified, two or more pieces of music are generated. Request to link the music in question. method.

9. A method for creating sound, The device, Generate music that is the same length as the video or a specified length. The AI ​​is asked to generate music based on the characteristics of the lyrics corresponding to the video. method.