Method, device, recording medium, and program

Abstract

One or more embodiments provide a method for creating sound, wherein a device identifies motion features that are features of motion of an object in a video, and requests an AI to generate music having a unit time as a beat, where the unit time is the time between similar motion features.

Description

Method, apparatus, recording medium, program

[0001] The present invention relates to a method, an apparatus, a recording medium, and a program.

[0002] The statements in this section only provide background information related to this disclosure and do not necessarily constitute prior art.

[0003] Patent Document 1 discloses a music generation system.

[0004] Japanese Patent Application Laid-Open No. 2006-154777

[0005] However, the inventor recognized that at least the above-described embodiment has a shortcoming that there is no method for creating sound based on a moving image.

[0006] At least one disclosure provides a method for creating sound, in which an apparatus specifies an operation feature that is a feature of the operation of an object in a moving image, and requires AI to generate music with the time between similar operation features as a unit time and the unit time as a beat.

[0007] In this configuration, there is at least a utility that industrially applicable music can be generated based on a moving image.

[0008] These and other aspects, features, and advantages of the present disclosure will become apparent from the following detailed written description of the preferred embodiments and aspects taken in conjunction with the following drawings, but modifications and alterations thereof can be made without departing from the spirit and scope of the novel concept of the present disclosure. Aspects in certain embodiments of the present disclosure can be combined with, or replaced by, one or more of the aspects in another embodiment of the present disclosure as long as they do not conflict.

[0009] Block diagrams according to one or more embodiments Block diagrams according to one or more embodiments

[0010] The following disclosure provides many different embodiments and examples for carrying out different features of the presented subject matter. For the sake of simplicity, specific examples of components and arrangements are disclosed below. Of course, these are merely examples and are not intended to be limiting. For example, a structure in which a first feature is covered by or in contact with a second feature subsequently disclosed may include embodiments in which an additional feature is formed between the first and second features so that they do not come into direct contact, as well as embodiments in which the first and second features are formed so that they do not come into direct contact. Furthermore, the disclosure may repeat reference numbers and / or letters in various examples. This repetition is for the sake of brevity and clarity and does not require in itself to be related to the various embodiments and / or configurations described. Furthermore, when a first element is described as being “connected” or “joined” with a second element, such description includes embodiments in which the first and second elements are directly connected or joined to each other, as well as embodiments in which the first and second elements are indirectly connected or joined to each other by having one or more other elements interposed between them. As used herein, the phrase “at least one of” encompasses all exemplary variations. For example, the phrase “comprises at least one of A, B, or C” is synonymous with “consisting of A, B, C and combinations thereof,” and encompasses all conceivable variations of A, B, C, A+B, A+C, B+C, and A+B+C.

[0011] In this disclosure, disclosures involving the use of machines, electronic operators, or computers may include embodiments of methods, recording media, apparatus, or programs. Any statement used herein that “A is B” may be replaced with “A includes B” unless otherwise stated herein, to the extent that it does not conflict with the statement.

[0012] The terms used in this disclosure, including those used in the claims, can be interpreted in light of the descriptions and drawings in the specification, and, to the extent that they do not contradict the implications of this disclosure, they can be interpreted based on what one or more citizens have called, represented, understood or practiced, or may have done in the past, present, or future, as such. The following embodiments can be taken for the operating method used in at least one embodiment. These embodiments will be described by reference to the description in JP6456303, which describes at least one embodiment well (beginning of reference).

[0013] As used herein, the term “computer” schematically includes, as known in the art, a processor, memory, at least one information storage / retrieval device such as a hard drive, disk drive or flash drive or memory stick, or other non-temporary computer-readable medium or non-temporary storage device, at least one input device such as a keyboard, mouse, point and touch device, touchscreen or microphone, and a display structure such as a well-known computer screen. In addition, a computer may include one or more network connections, such as wired or wireless connections. As known in the art, such a computer or computer system may more or less include, but is not limited to, for example, tablet computers or smart devices, other electronic media and electronic devices.

[0014] As used herein, the terms “cloud” or “cloud computing” refer to centralized and virtualized computing facilities where all computing resources are shared. For application systems and subsystems, since they all reside within the “cloud,” it is no longer possible to refer to specific machines.

[0015] As used herein, the term “Distributed Internet Services System” refers to a distributed internet services platform that transforms internet applications for execution in various computing environments. A DIS system delivers internet applications, including content, data, and logic, to whatever extent applicable, and along a network, to any number and type of devices, via Component Distribution Servers / Asset Distribution Servers. Through DIS, internet applications can be hosted and centrally managed as services tailored to each user's needs, and can be locally cached and executed on the user's device or nearby locations while maintaining their integrity. Web-enabled computing devices can be upgraded with DIS software to become DIS-enabled, capable of enjoying and running distributed internet services. The distributed internet service system is fully described in any one of the patent families U.S. Patents 7,136,857, 7150015, 7181731, 7209921, 7430610, 7685183, 7685577, 7752214, 8326883, 8386525, 8443035, 8458142, 8458222, 8473468, 8527545, and 8650226, and U.S. Patent Publications 20120005205 ​​and 20130091252, all of which, like the present invention, are jointly owned by OP40, Holdings, Inc., and all of which are incorporated by reference. (End of quote)

[0016] Regarding the operating method used in at least one embodiment, the following embodiments can be taken for conventional internet systems that do not use a distributed internet. The following will be explained by reference to the description in JP7113047, which describes at least one embodiment in detail (beginning of reference).

[0017] Embodiments including those specifically disclosed herein can provide an automated response system based on artificial intelligence that is implemented in a manner that mimics actual human conversation, thereby enabling more natural communication with users while quickly and conveniently handling inquiries, reservations, delivery orders, and more.

[0018] The multiple electronic devices 110, 120, 130, and 140 may be fixed terminals or mobile terminals implemented by a computer system. Examples of the multiple electronic devices 110, 120, 130, and 140 include AI speakers, smartphones, mobile phones, navigation systems, PCs (personal computers), notebook PCs, digital broadcasting terminals, PDAs (Personal Digital Assistants), PMPs (Portable Multimedia Players), tablets, game consoles, wearable devices, IoT (Internet of Things) devices, VR (virtual reality) devices, and AR (augmented reality) devices. As an example, Figure 1 shows an AI speaker as the electronic device 110, but in embodiments of the present invention, the electronic device 110 may mean one of a variety of physical computer systems that can communicate with other electronic devices 120, 130, 140 and / or servers 150, 160 via the network 170 using substantially wireless or wired communication methods.

[0019] The communication method is not limited, and the network 170 may include not only communication methods that utilize communication networks that can be included (for example, mobile communication networks, wired internet, wireless internet, broadcasting networks, satellite networks, etc.), but also short-range wireless communication between devices. For example, the network 170 may include one or more arbitrary networks such as PAN (personal area network), LAN (local area network), CAN (campus area network), MAN (metropolitan area network), WAN (wide area network), BBN (broadband network), and the Internet. Furthermore, the network 170 may include, but is not limited to, one or more network topologies, including bus networks, star networks, ring networks, mesh networks, starbus networks, tree or hierarchical networks.

[0020] Servers 150 and 160 may each be implemented by one or more computer devices that communicate with multiple electronic devices 110, 120, 130, and 140 via a network 170 to provide instructions, code, files, content, services, etc. For example, server 150 may be a system that provides a first service to multiple electronic devices 110, 120, 130, and 140 connected via the network 170, and server 160 may be a system that provides a second service to multiple electronic devices 110, 120, 130, and 140 connected via the network 170. As a more specific example, server 150 may provide the multiple electronic devices 110, 120, 130, and 140 as a first service through an application, which is a computer program installed and executed on the multiple electronic devices 110, 120, 130, and 140, with the service targeted by that application (for example, an automated response service). As another example, server 160 may provide a second service that distributes files for installing and running the aforementioned application to multiple electronic devices 110, 120, 130, and 140.

[0021] Figure 2 is a block diagram illustrating the internal configuration of an electronic device and a server in one embodiment of the present invention. In Figure 2, the internal configuration of electronic device 110 and the internal configuration of server 150 are described as examples of electronic devices. Other electronic devices 120, 130, 140 and server 160 may also have the same or similar internal configuration as the electronic device 110 or server 150 described above.

[0022] The electronic devices 110 and server 150 may include memories 211, 221, processors 212, 222, communication modules 213, 223, and input / output interfaces 214, 224. The memories 211, 221 may be non-temporary computer-readable recording media and may include non-temporary mass storage devices such as RAM (random access memory), ROM (read-only memory), disk drives, SSDs (solid state drives), and flash memory. Here, non-temporary mass storage devices such as ROM, SSDs, flash memory, and disk drives may be included in the electronic devices 110 and server 150 as separate non-temporary storage devices distinct from the memories 211, 221. Furthermore, the memories 211 and 221 may contain an operating system and at least one program code (for example, code for a browser installed and run on the electronic device 110, or code for an application installed on the electronic device 110 to provide a specific service). Such software components may be loaded from a computer-readable recording medium other than the memories 211 and 221. Such a separate computer-readable recording medium may include computer-readable recording media such as floppy® drives, disks, tapes, DVD / CD-ROM drives, and memory cards. In other embodiments, the software components may be loaded into the memories 211 and 221 through communication modules 213 and 223 that are not computer-readable recording media. For example, at least one program may be loaded into the memories 211 and 221 based on a computer program (for example, the application described above) that is installed by a file provided via the network 170 by a developer or a file distribution system (for example, the server 160 described above) that distributes installation files for an application.

[0023] Processors 212 and 222 may be configured to process computer program instructions by performing basic arithmetic, logic, and input / output operations. Instructions may be provided to processors 212 and 222 by memory 211 and 221 or by communication modules 213 and 223. For example, processors 212 and 222 may be configured to execute instructions received according to program code recorded in a recording device such as memory 211 and 221.

[0024] Communication modules 213 and 223 may provide functions for the electronic device 110 and the server 150 to communicate with each other via the network 170, or they may provide functions for the electronic device 110 and / or the server 150 to communicate with other electronic devices (for example, electronic device 120) or other servers (for example, server 160). For example, a request generated by the processor 212 of the electronic device 110 according to program code recorded in a recording device such as memory 211 may be transmitted to the server 150 via the network 170 under the control of the communication module 213. Conversely, control signals, instructions, content, files, etc., provided under the control of the processor 222 of the server 150 may be received by the electronic device 110 via the communication module 223 and the network 170 through the communication module 213 of the electronic device 110. For example, control signals, commands, content, and files from the server 150 received through the communication module 213 may be transmitted to the processor 212 and memory 211, and the content and files may be recorded on a recording medium (the non-temporary recording device described above) that the electronic device 110 may further include.

[0025] The input / output interface 214 may be a means for interface with an input / output device 215. For example, the input device may include devices such as a keyboard, mouse, microphone, or camera, and the output device may include devices such as a display, speaker, or haptic feedback device. As another example, the input / output interface 214 may be a means for interface with a device that integrates input and output functions into one, such as a touchscreen. The input / output device 215 may consist of the electronic device 110 and one other device. The input / output interface 224 of the server 150 may be a means for interface with an input or output device (not shown) that is connected to or can be included in the server 150. As a more specific example, when the processor 212 of the electronic device 110 processes program instructions loaded into memory 211, a service screen or content configured using data provided by the server 150 or the electronic device 120 may be displayed on the display via the input / output interface 214.

[0026] Furthermore, in other embodiments, the electronic device 110 and the server 150 may include more components than those shown in Figure 2. However, it is not necessary to explicitly show most of the conventional components in the figure. For example, the electronic device 110 may be implemented to include at least some of the input / output devices 215 described above, and may further include other components such as transceivers, cameras, various sensors, and databases. As a more specific example, if the electronic device 110 is an AI speaker, the electronic device 110 may be implemented to further include a variety of components that are generally included in an AI speaker, such as various sensors, camera modules, various physical buttons, buttons using a touch panel, input / output ports, and vibrators for vibration. (End of quote)

[0027] The following disclosure concerns a machine. According to at least one embodiment, the user terminal comprises a control unit, RAM, storage unit, graphics processing unit, communication interface, and interface unit, each connected by an internal bus.

[0028] According to at least one embodiment, the control unit consists of a CPU and ROM. The control unit executes programs stored in the storage unit and controls the user terminal. RAM is the work area of ​​the control unit. The storage unit is a memory area for saving programs and data. The control unit reads programs and data from RAM and processes them. By processing the programs and data loaded into RAM, the control unit outputs drawing commands to the graphics processing unit.

[0029] According to at least one embodiment, the graphics processing unit is connected to the display unit. The display unit has a display screen. When the control unit outputs a drawing command to the graphics processing unit, the graphics processing unit outputs a video signal for displaying an image on the display screen. Here, the display unit may be a touch panel equipped with a touch sensor. The touch panel of this display unit functions as an input unit.

[0030] According to at least one embodiment, the communication interface can be connected to a communication network wirelessly or via a wired connection, and can send and receive data with a server device via the communication network. The data received via the communication interface is loaded into RAM and processed by the control unit. External memory (e.g., an SD card) is connected to the interface unit.

[0031] According to at least one embodiment, the user terminal is not particularly limited as long as it is a computer device having a display screen and an input unit. Examples of user terminals include conventional mobile phones, tablet devices, smartphones, and desktop or notebook personal computers. It may also consist of a screen (or two display panels, one for each eye) attached to a frame (or headset) that is fixed or attached to the head with a strap, i.e., a VR goggle. The user terminal has an audio output unit.

[0032] According to at least one embodiment, a user terminal can communicate with a server device via a communication network. It can transmit or receive information by establishing a communication connection via the communication network.

[0033] According to at least one embodiment, the server device comprises at least a control unit, RAM, a storage unit, and a communication interface, each connected by an internal bus.

[0034] According to at least one embodiment, the control unit consists of a CPU and ROM, executes programs stored in the storage unit, and controls the server device. The control unit also has an internal timer for timing. RAM is the work area of ​​the control unit. The storage unit is a memory area for saving programs and data. The control unit reads programs and data from RAM and performs program execution processing based on information received from the user terminal, etc.

[0035] This document discloses AI. According to at least one embodiment, artificial intelligence includes machine learning, deep learning, generative AI, large-scale language models (LLMs), foundational models, and generative AI. Generative AI uses transformers and employs numerous attention mechanisms. It uses self-supervised learning and Extract Prediction. In this case, the AI ​​can predict the next word. Given a sentence, it predicts the next word from the sentence up to that point. A large number of supervised learning problems are generated. These create an AI that can predict the next word. Generative AI can predict grammatical structure, topic connections, and what kind of sentences people with a certain writing style are likely to write. Furthermore, by simply predicting the next sentence, generative AI can learn the underlying structure, causal relationships, and knowledge. Generative AI scales quickly, and its accuracy improves as the number of parameters increases. Ordinary statistics and machine learning overfit if the model parameters are too large compared to the data sample size. LLMs become more accurate as the number of parameters increases. One generative AI has 175 billion parameters. Generative AI is overlaid with supervised learning to facilitate smooth conversation. They are taught not to say anything strange. They write essays and act as call center operators.

[0036] According to at least one embodiment, Large Language Models (LLMs) are, in a non-inclusive sense, machine learning natural language processing models built using large datasets and deep learning techniques. Generally, they are adapted to various natural language processing (NLP) tasks such as text classification and generation, sentiment analysis, text summarization, and question answering using a technique called "fine-tuning," which involves training them on specific tasks. According to at least one embodiment, self-supervised learning is close to intrinsic human intelligence. When humans act, they are always predicting what will happen next and predicting the next input. In the process, they can learn the structure of the external world. Predicting the next word is considered intrinsic intelligence and is close to what is done in the cerebral cortex. According to at least one embodiment, Large Language Models memorize all the input information but generalize it to the extent necessary to predict the next word. They do not try to generalize all the information from the beginning. Large Language Models require capacity to remember information, and therefore require parameters. According to at least one embodiment, the large-scale language model incorporates eight models, each with 175 billion parameters or 220 billion parameters.

[0037] According to at least one embodiment, videos and images are represented as a collection of visual patches, which are small data units similar to text tokens in LLMs. Patches effectively represent models of visual data and are used as highly scalable and effective representations for training generative models on various types of videos and images. Videos are transformed into patches by first compressing the video into a low-dimensional latent space and then decomposing the representation into spatiotemporal patches.

[0038] According to at least one embodiment, a video compression network is a network that reduces the dimensionality of visual data, taking raw video as input and outputting a temporally and spatially compressed latent representation. An AI is trained in this compressed latent space and then generates video within this compressed latent space.

[0039] According to at least one embodiment, Spacetime Latent Patches extract a series of spatiotemporal patches that function as transformer tokens when given a compressed input video. The patch-based representation allows Sora to be trained on videos and images of varying resolutions, lengths, and aspect ratios, and controls the size of the resulting video by arranging randomly initialized patches into a grid of appropriate size during inference.

[0040] According to at least one embodiment, the AI ​​is a diffusion model that, given a noisy patch (and conditioning information such as text prompts) as input, is trained to predict the original "clean" patch. The AI ​​is a diffusion transformer, which exhibits remarkable scaling properties in various domains, including language modeling, computer vision, and image generation. Diffusion transformers are also effective as video generation models. The AI's sample quality improves significantly as the training computation increases.

[0041] According to at least one embodiment, the AI ​​applies caption regeneration technology to train a highly descriptive caption model, which is then used to generate text captions for all videos in the training set. Training highly descriptive captions improves not only the overall quality of the generated videos but also the fidelity of the text. GPT is used to convert short user prompts into long, detailed captions, which are then sent to the model. This allows the AI ​​to generate high-quality videos that precisely follow the user prompts.

[0042] According to at least one embodiment, the AI ​​can perform vectorization in natural language processing according to the following flow. First, as a preprocessing step, it performs a cleaning process on the given text. In the cleaning process, unnecessary words such as JavaScript code and HTML tags contained in the text are removed. These codes are used to display on the internet and are therefore not generally used in natural language processing. Next, the text is divided into words using morphological analysis. Morphological analysis is the classification of a natural language sentence written in characters into the smallest meaningful linguistic unit. "MeCab," "JUMAN," and "JANOME" can be used as morphological analysis tools. In normalization, words with the same meaning, such as variations in spelling, are unified into a single word. Stop words are words that are excluded from processing for reasons such as not being usable in natural language processing. Examples of stop words include particles and auxiliary verbs, which do not have meaning on their own. When calculating vectors, these may be removed, and only meaningful words may be targeted. Vectorization may also be performed without removing these stop words. Vectorization is the process of converting strings of words into vectors. Vectorization transforms word data into numerical data. When converting words to vectors, methods such as Bag of Words and distributed representations are used. Bag of Words is a method of vectorizing a given text using the number of occurrences of each word. It focuses on how often each word appears in the text, and does not consider the order of words or sentences. Distributed representation is a method of vectorizing by focusing on the meaning of words. By vectorizing the meaning of words, it is possible to assign similar vectors to words with similar meanings or usages, and the relationships between words can also be represented by vectors. With vector representation, it is possible to add and subtract the meanings of words. In applied processing, natural language converted into numerical data can be used as input for machine learning. Specifically, vectorized natural language is fed into a classifier to perform text classification.Tools used here include TensorFlow, scikit-learn, and PyTorch.

[0043] This document discloses information about sound. In at least one embodiment, sound includes music or sound effects. Music is a combination of length, pitch, intensity, timbre, etc., of sounds that expresses various emotions and stories, and includes singing, instrumental performances, and natural sounds. Music can be composed by determining a scale (key), chords (harmonies), or melody. In at least one embodiment, once a scale is determined, the chords are also determined according to its constituent notes. There are theories about chord progressions, and there are patterns that have been considered good progressions in songs to date. Chords greatly influence the mood of a song and can have a significant impact on a person's emotions and feelings at that time. Similarly, while melodies can be based on the constituent notes of chords without deviating too much, they tend to become monotonous, so some randomness may be necessary. By varying the notes within a range that does not deviate too much from the chords, monotony can sometimes be prevented.

[0044] This disclosure provides a method for creating sound, wherein the apparatus identifies motion features that are characteristic of the movement of objects in a video (in this disclosure, the term “method” may be interpreted as “steps” unless otherwise explicitly contradicted). In at least one embodiment, a video is a sequence of still images displayed in order to create movement, or data. Videos may include sound, and the synchronization of the video and sound may result in richer expression. An object is data or related processing that is manipulated within a video. Examples of objects include moving objects in a video (including living and non-living things; non-inclusive examples of the former include people, people dancing, animals, plants swaying in the wind, etc.; non-inclusive examples of the latter include moving objects (such as cars), the ebb and flow of waves, flashing lights, changes in the wavelength or intensity of light, and digital representations drawn with computer graphics). In this disclosure, any object that a person can perceive as moving when viewing a video is considered to be this “object” or data corresponding to an “object”. In this disclosure, "operational features" may refer to "operational feature quantities" (unless otherwise explicitly stated throughout this disclosure). Feature quantities are numerical values ​​that represent the characteristics of data contained within a dataset. For example, when representing a person's face in a video, feature quantities could include "eye size," "nose height," and "skin color." In addition, when a person is dancing, the movement of organs (including hands, feet, and hips) can also be considered feature quantities. By providing these numerical values ​​to a computer, the computer can distinguish between people's faces, recognize specific individuals, or recognize dance choreography. Furthermore, in the case of tides, the way waves rise and the way they fall can naturally be considered feature quantities. In addition, in the case of computer graphics, the movement of lines or surfaces, and the way colors change, can naturally be considered feature quantities. Non-comprehensively, the types of feature quantities include the following: Numerical data is data that can be expressed numerically, such as height, weight, and age. Categorical data is data that can be expressed in categories, such as gender, nationality, and occupation. Text data refers to data that can be represented in text form, such as sentences, words, and keywords.Image data (including still images in videos) are features extracted from images, such as color, shape, texture, etc. Video data are features extracted from videos, such as color, shape, texture, the presence or change of these at the playback time. Audio data are features extracted from audio, such as pitch, frequency, timbre, etc.

[0045] In at least one embodiment, the device can be created by any of the methods, machines or AIs described above. In the present disclosure, the description of "device" can be replaced by "AI" unless there is an explicitly contradictory description. The device analyzes the data of the target video. The device identifies the objects in the video. The user can identify the target object. As another embodiment, the device identifies the object with a relatively high appearance frequency from the objects existing in the video. The device can also request the AI to identify it in this way (throughout the present disclosure, unless there is an explicitly contradictory description, the description that "the device performs a certain process" can be replaced by "the device requests the AI to perform the process". In this case, the AI can use the AI that has learned the process and judgment by the learning method as described above (including supervised learning)). In such a case, the target object is automatically identified by the device rather than specified by the user. The device identifies the motion features of the object. As an example, for a video of a dancer dancing, the way the dancer waves their hand is a motion feature of the object. In this embodiment, it is of course also possible to use the way the dancer shakes their head, the way they sway their waist, the movement of their feet, and the movement of their fingers as motion features. As already described, the motion features can be identified by the user. Furthermore, the device can also identify the motion with a relatively high appearance frequency from the motions of the object as a motion feature.

[0046] This disclosure describes a method for requesting an AI to generate music in which the time between similar motion features is used as the unit time, and the unit time is used as the beat (time signature). In this disclosure, unless otherwise explicitly stated, the phrase "generate music" may be replaced with "generate sound." In at least one embodiment, a video contains similar motion features. As a non-exclusive example, in the case of dance, a dancer may repeatedly perform the same dance choreography. In this case, the device compares the motion features of each movement and recognizes them as similar motion features. There are two methods for determining similarity: one where the user defines the criteria, and another where the machine automatically determines the criteria. In the former, the device displays videos or still images of two or more motion features on the screen of the user's terminal. The user specifies to the device whether two or more videos or still images of motion features are similar or not. Based on this specification, the device determines whether the motion features are similar or not. In the latter, the device determines whether two or more videos or still images of motion features contain clearly different motion features. If no clear features or differences in feature quantities can be detected, these videos can be determined to be similar. In this determination, the AI ​​can learn about similarity through supervised learning. The device can also utilize the functions of the AI, or request the AI ​​to determine similarity and obtain the determination result.

[0047] The device identifies the time between operation characteristics. As a non-exclusive example, regarding dance, when a dancer repeats the choreography of the same dance (similar operation characteristics), the difference (or distance) in the appearance time (or application time) of each operation characteristic in the video is identified. As a non-exclusive example, assume that a video of a certain dance performance is a 3-minute video. And when the dancer repeats the operation characteristic at the 1 minute 10 seconds point and the 1 minute 15 seconds point, the time between the operation characteristics is 5 seconds, and the unit time is 5 seconds. The device recognizes the time between operation characteristics as the unit time. The device can transmit the information of the unit time to other devices or AI. In other embodiments, as described above, the device can also use an AI that has learned to recognize the unit time through supervised learning or request the AI to recognize the unit time.

[0048] In at least one embodiment, the beat in sound includes the unit of rhythm. As a non-exclusive example, the beat includes a certain time interval (the basic unit that repeats at a certain period in music), the contrast between strong and weak (there are strong beats and weak beats, and the rhythm is generated by this contrast), or the basis of tempo (the speed of the beat becomes the tempo, which determines the speed of the song). The beat may be the meter. The meter is the basis of the rhythm of the whole song and indicates how many beats are in a measure. For example, in the case of a four-four time signature, there are 4 beats in 1 measure, and the length of each beat is the same. In the present disclosure, the description of "beat" can be replaced by "meter" unless there is an explicitly contradictory description. The device requests the AI to generate music with the unit time as the beat (meter). As a non-exclusive example, assume that a video of a certain dance performance is a 3-minute video and the unit time is 5 seconds. The device requests the AI to generate music with 5 seconds as 1 beat. In other embodiments, as described above, the device may generate the music itself using the AI.

[0049] According to the above configuration, sound can be automatically created based on video. The created sound or music has a beat based on the features of the video, resulting in a high degree of matching between the video and the generated sound. This reduces the workload of the person creating the sound, and allows for the automatic generation of highly usable sound with a high degree of matching between the video and the generated sound, thus having industrial applicability.

[0050] This disclosure describes a method for requesting an AI to generate music with a beat unit, and for requesting that the music be the same length as the video duration (including the time over which motion features should be identified; in this disclosure, unless otherwise explicitly stated, "video duration" can be understood as "video length" or "video duration"). The device requests the AI ​​to generate music of the same length as the video duration. As a non-exclusive example, if a video of a dance performance is three minutes long, the video duration is three minutes, so the device requests that the AI ​​generate three minutes of music. In other embodiments, the device defines the time over which motion features should be identified as the video duration. The time over which motion features should be identified is, in at least one embodiment, the time period in the video over which the device or AI is permitted to identify motion features, in whole or in part. There are two methods for determining the time over which motion features should be identified: a user-defined method and an automatic method by the device. In the former, the user can define the time over which motion features should be identified, as long as it is within the video duration. As a non-inclusive example, if a video of a dance performance is three minutes long, the period from 0 minutes 0 seconds to 1 minute 30 seconds can be designated as the time for identifying the motion characteristics. The device identifies the motion characteristics only within the designated time period using the method described above. As a latter example, the device can also designate a predetermined portion of the video's duration as the time for identifying the motion characteristics. Naturally, the entire duration of the video can also be designated as the time for identifying the motion characteristics. The device is required to generate music of the same length as the time for identifying the motion characteristics.

[0051] This invention discloses a method for requiring a device to generate music such that the beats of the music are identical to the unit time in a video. The phrase "identical" above can be replaced with "synchronized". In at least one embodiment, the device requires the device to generate music that is the same length as the video and to generate music such that the unit time of the music is identical to the unit time in the video. As a non-exclusive example, suppose a video of a dance performance is 3 minutes long. If the dancer repeats a motion feature at 1 minute 12 seconds and 1 minute 17 seconds, the unit time is 5 seconds. Furthermore, the device requires the AI ​​to generate music where 5 seconds equals one beat and to generate music that is 3 minutes long. Since the device requires the device to generate music such that the unit time of the music is identical to the unit time in the video, the generated sound will have beats that are identical (synchronized) to the beats at 1 minute 12 seconds and 1 minute 17 seconds in the video. In this case, if the generated music has beats in 5-second units of time starting from 0 minutes 2 seconds, it will be identical (synchronized) to the beats at 1 minute 12 seconds and 1 minute 17 seconds in the video. In other words, the device recognizes the beats of the music corresponding to the unit time in the video as specific beats, and generates music with a defined overall beat based on (or in a manner consistent with) these specific beats. Taking the previous example, the beats at 1 minute 12 seconds and 1 minute 17 seconds correspond to the specific beats, and music with a defined overall beat is generated based on these specific beats.

[0052] According to the above configuration, it is possible to automatically create sound that is synchronized with the beat of a video. The created sound or music not only has a beat based on the video's features, but is also at least synchronized with the video's beat, resulting in a higher degree of matching between the video and the generated sound. In this case, the generated sound can be used industrially as is by playing the video and the generated sound simultaneously. This reduces the workload of sound creators and has industrial potential because it allows for the automatic generation of highly usable sound with a high degree of matching between the video and the generated sound.

[0053] This invention discloses a method for requesting an AI to generate music with beats based on unit time, by requesting the AI ​​to select a relatively short unit time from among two or more unit times arising from similar motion features as the beat. In at least one embodiment, the device may recognize three or more similar motion features from a video. In this case, the device may recognize two or more unit times. The device recognizes the shortest unit time from among two or more unit times arising from similar motion features as the unit time to be used as the beat. The device can transmit such a unit time to be used as the beat to another device or AI, or request the AI ​​to recognize or identify such a unit time to be used as the beat. As already explained, the AI ​​learns to use the shortest unit time as the beat through supervised learning using a video having two or more unit times arising from similar motion features. As a non-exclusive example, suppose a video of a dance performance is three minutes long. For example, the A section, B section, and chorus are commonly used in pop and rock songs, and each is composed of a different melody. In music videos, the structure is typically a combination of verse A, chorus, pre-chorus B, and chorus, with similar choreography for each chorus. Let's assume the dancers repeat a specific movement at 1 minute 12 seconds, 1 minute 17 seconds, 2 minutes 12 seconds, and 2 minutes 17 seconds. The device will recognize four different time units: 5 seconds, 55 seconds, 1 minute, and 1 minute 5 seconds. In this case, the device will recognize 5 seconds as the shortest time unit. The device will then recognize this shortest time unit as a beat, or request the device or AI to recognize the shortest time unit as a beat.

[0054] With the above configuration, even in complex videos where similar behavioral features are scattered throughout, it is possible to automatically create sound that is synchronized with the beat of the video. The created sound or music not only has a beat based on the video's features, but is also at least synchronized with the video's beat, resulting in a higher degree of matching between the video and the generated sound. In this case, the generated sound can be used industrially as is by playing the video and the generated sound simultaneously. This has industrial potential because it reduces the workload of sound creators and enables the automatic generation of highly usable sound with a high degree of matching between the video and the generated sound.

[0055] This invention discloses a method for requesting an AI to generate music with beats based on unit time, by requesting that it select a unit time that has a relatively high frequency of occurrence from two or more unit time units arising from similar motion features to be used as a beat. In at least one embodiment, the device calculates the frequency of occurrence for the recognized unit time. Frequency of occurrence includes a numerical value that indicates how many times a certain event or object appears within a specific period or range. As an example, frequency of occurrence includes events (specific words, numbers, actions, phenomena, etc.), ranges (specific sentences, datasets, time, space, etc.), and the frequency of occurrence of a word in a sentence (how many times the word "cat" appears in a sentence, etc.). The device calculates the frequency of occurrence of unit time within the time range of the video or the time for which motion features should be identified. As a non-exclusive example, suppose a video of a dance performance is 3 minutes long, and the dancer's choreography is recognized as a motion feature. Suppose the device recognizes that there are 50 units of 5 seconds, 10 units of 10 seconds, 3 units of 20 seconds, and 1 unit of 1 minute. In this case, the device recognizes the 5-second interval, which appears most frequently, as the unit of time to be considered a beat. The device then generates music using this unit of time as a beat, or requests the AI ​​to generate it in this way.

[0056] As described above, this configuration allows for the automatic creation of sounds synchronized with the beats of a video, even in complex videos where similar behavioral features are scattered throughout. Furthermore, since the most frequently occurring unit of time is used as the beat, the beats of the generated music are less likely to be inconsistent with the beats of the video. Therefore, the generated sound or music not only has beats based on the video's features, but is also at least synchronized with the video's beats, resulting in a higher degree of matching between the video and the generated sound. In this case, the generated sound can be used industrially as is by playing the video and the generated sound simultaneously. This reduces the workload of sound creators and allows for the automatic generation of highly usable sounds with a high degree of matching between the video and the generated sound, thus offering industrial potential.

[0057] This invention discloses a method for requesting an AI to generate music with a unit time of beat, wherein the AI ​​generates music of the same duration as the duration of the video or the duration of the motion features to be identified, and when two or more unit times based on dissimilar motion features are identified, the AI ​​generates two or more pieces of music and concatenates them. In at least one embodiment, the device recognizes a unit time based on similar motion features, as already described. In this case, the device recognizes a unit time based on motion feature A and a unit time based on motion feature B. These are two or more unit times based on dissimilar motion features. When two or more unit times based on dissimilar motion features are identified, the device generates two or more pieces of music. In other embodiments, the AI ​​or other device is requested to generate two or more pieces of music in this manner. The device concatenates these generated pieces of music. Concatenation includes a state in which multiple melodies are connected. The concatenation may be smooth and natural. As a non-exclusive example, the device generates music A, generated from a video (including a music video or a film) with a unit time based on motion feature A, and music B, generated from a unit time based on motion feature B. The device links music A and music B. One method of linking them is to link the music data so that when one piece of music ends, the other piece of music can be played simultaneously. Another method is to link a portion of the music data with a portion of the data of another piece of music so that one piece of music ends midway and another piece of music is played. In at least one embodiment, when two or more pieces of music are generated, music 2 can be obtained by modulating music 1.

[0058] According to the above configuration, even for complex videos where two or more unit times are identified based on dissimilar motion features, it is possible to automatically create sound that synchronizes with the beat of the video. Furthermore, because the sound is based on two or more unit times, the possibility of mismatch with the video's beat is further reduced. Therefore, the created sound or music is largely synchronized with the video's beat, resulting in a higher degree of match between the video and the generated sound. In this case, the generated sound can be used industrially as is by playing the video and the generated sound simultaneously. This has industrial potential because it reduces the workload of sound creators and enables the automatic generation of highly usable sound with a high degree of match between the video and the generated sound.

[0059] The present invention discloses a method that requires the music to be concatenated at a point in time identified based on the frequency of occurrence of two or more operational features. In at least one embodiment, the device generates music of the same duration as the video duration or the duration for which the operational features should be identified, and identifies the point in time when the unit time changes for two or more unit time segments. As a non-exclusive example, the device calculates the frequency or probability of occurrence of each unit time segment based on each operational feature within the video's time frame. For example, within the video duration or the duration for which the operational features should be identified, the device identifies time segments where the unit time segment based on operational feature A appears relatively frequently, and time segments where the unit time segment based on operational feature B appears relatively frequently. Based on these frequencies, the device generates music based on the unit time segment with the highest frequency. If the unit time segment with the highest frequency changes, the device generates music based on the new unit time segment. The device concatenates these two pieces of music. In another embodiment, the device generates music of the same duration as the video duration (the duration for which the operational features should be identified), and generates two or more pieces of music when two or more dissimilar operational features are identified. The device identifies a point in time (in this paragraph, unless explicitly contradictory, the terms "point in time" or "time" can be replaced with "time range") determined based on the frequency of occurrence of two or more dissimilar behavioral features as the change time. For example, within the video duration or the time range for which behavioral features should be identified, the device identifies a time range where unit time based on behavioral feature A appears relatively frequently, and a time range where unit time based on behavioral feature B appears relatively frequently. The device can identify the point in time when the most frequently occurring behavioral feature changes, or a point in time between or approximately midway through the time ranges where different behavioral features appear frequently, as the change time. As a non-inclusive example, suppose a video of a dance performance is 3 minutes long. If, in the time range from 0 minutes 0 seconds to 1 minute 30 seconds, unit time based on behavioral feature A appears relatively frequently, and in the time range from 1 minute 30 seconds to 3 minutes, unit time based on behavioral feature B appears relatively frequently, the device identifies 1 minute 30 seconds, which is approximately midway between these two time ranges, based on the frequency of occurrence of the two or more behavioral features. The device will request that the music be linked once it has been identified.In other words, the generated music consists of two parts: from 0 minutes 0 seconds to 1 minute 30 seconds, it is music A based on operational characteristic A; and from 1 minute 30 seconds to 3 minutes, it is music B based on operational characteristic B. These two parts are joined together at 1 minute 30 seconds.

[0060] As described above, with this configuration, even for complex videos where two or more units of time are identified based on dissimilar motion features, it is possible to automatically create sound that synchronizes with the beat of the video. Furthermore, because the sound is based on two or more units of time, the possibility of mismatch with the video's beat is further reduced. Therefore, the created sound or music is largely synchronized with the video's beat, resulting in a higher degree of match between the video and the generated sound. In this case, the generated sound can be used industrially as is by playing the video and the generated sound simultaneously. This has industrial potential because it reduces the workload of sound creators and enables the automatic generation of highly usable sound with a high degree of match between the video and the generated sound.

[0061] In at least one embodiment, the video includes films, dances, and choreography. For example, it includes videos published on YouTube®. In this case, music can be automatically generated to match the published video. The video also includes live-streamed videos. In this case, the device generates music simultaneously with the live stream, according to the conditions or configuration described in any of the disclosures. The device can distribute this generated music simultaneously with the live-streamed video. According to the above configuration, music that matches the beat of the video can be provided simultaneously with the live stream. Such a highly flexible generation method is something that human composers cannot do, and therefore has superior effects compared to conventional technologies, making it potentially industrially applicable. In at least one embodiment, the device can request the AI ​​to analyze ambient sounds and audience voices during streaming in real time, recognize one or more of them as specific sounds, features, or feature quantities, and generate sounds or sound effects corresponding to them.

[0062] In at least one embodiment, the video includes a video used in VR or AR. VR (Virtual Reality) is a technology that provides an experience of being completely immersed in a virtual world. By wearing a device such as VR goggles, users can feel as if they are in another world. AR (Augmented Reality) is a technology that overlays digital information onto the real world. Through a smartphone camera, virtual objects can be displayed on real-world scenery, or information can be added. The device can generate music for a video presented to the user through VR or AR, in accordance with the conditions or configuration described in any of the disclosures. The device can output the generated music as audio at the same time as it is presented to the user through VR or AR. The audio output includes embodiments in which audio is output from a user terminal. According to the above configuration, music that matches the beat of the VR or AR video can be provided. This has industrial applicability in that it reduces the workload of the sound creator, or enables the automatic generation of highly usable sound with a high degree of matching between the video and the generated sound.

[0063] In at least one embodiment, the AI ​​repeatedly learns to predict the next note using unsupervised learning. As already explained, the composition process is hierarchical according to its structure. The elements that make up music are at least a scale (key), chords (harmonies), and melody. Once the scale is determined, the chords are also determined according to its constituent notes. There is a theory to chord progressions, and many patterns that have been considered good progressions in songs have been accumulated. Chords greatly influence the mood of a song and are said to have a significant impact on a person's emotions and feelings at that time. The AI ​​repeatedly learns to predict the next note using unsupervised learning based on music data. Furthermore, the AI ​​performs supervised learning on the music data to determine what kind of emotions the music expresses.

[0064] In at least one embodiment, the device or AI generates music based on music or related information that has already been uploaded to the internet or recorded in a database as completed, regardless of whether the author is registered or not. As a non-exclusive example, if the unit time based on the operational feature A is 5 seconds, the device identifies music that has been uploaded to the internet as completed and has a beat of 5 seconds. If multiple pieces of music are identified, the device identifies the music that best matches any of the conditions described in this disclosure. The device or AI generates music by, or by modifying the identified music.

[0065] In at least one embodiment, the device or AI identifies one piece of music based on music or related information that has already been uploaded to the internet or recorded in a database as completed, regardless of whether the author is registered or not. In this embodiment, the device or AI is configured to identify existing music that meets the criteria, rather than generating music. The device presents the identified music or information about music to the user.

[0066] In at least one embodiment, the device or AI generates music using music or related information that has already been uploaded to the internet or recorded in a database as completed, regardless of whether the author is registered or not.

[0067] In at least one embodiment, the device adds data indicating that the music was generated by AI to the music data generated by AI. When providing a video to a user along with the generated music, if the device recognizes that the music contains such data, it informs the user that the music was generated by AI, or displays this information on the user's terminal. In another embodiment, when the device generates music using music that has already been uploaded to the internet or recorded in a database as completed, it adds information about the name of the source music or the copyright holder to the generated music data. If the device recognizes that the music contains such information, it informs the user that the music was created by the copyright holder, or that the music was generated by AI based on that music, or displays this information on the user's terminal. In at least one embodiment, the device generates data or prompts that display copyright notices in a format compatible with the distribution platform. For example, on YouTube®, it generates data or prompts that display copyright notices as defined by the platform, or generates such data accompanying the generated video. These may be referred to as copyright management tools. The above configuration has industrial applicability because it can reduce the risk of copyright infringement in music data added or generated by AI.

[0068] In at least one embodiment, the user can specify to the device the sound to be relied upon when generating music. As an example, the device accepts sound data to be relied upon. The device has two or more sound sources that the user can specify, and the user can select any sound source. "Having two or more sound sources recorded" includes not only embodiments in which the device itself stores these sound sources, but also embodiments in which the device obtains these sound sources from an external storage medium. For example, these sound sources are pre-existing music. The user selects one or more sound data to be relied upon, and the device generates music based on the selected music. As a non-exclusive example, if the video has a unit time of 5 seconds and the selected sound data is pre-existing music with 7 beats, the device adjusts the playback speed of the pre-existing music to be relied upon to change it to music with 5 beats. In this example, the changed music corresponds to the "generated music" in the above embodiment. According to the above configuration, pre-existing music that synchronizes with the beat of the video can be automatically identified. Furthermore, since the identified pre-existing music is a sound based on a unit time, it matches the beat of the video. In this case, the presented sound can be used industrially as is by playing the video and the generated sound simultaneously. This reduces the workload of the sound creator, alleviates the burden of searching for music with matching beats, and automatically provides highly usable sounds with a high degree of matching between the video and the generated sound, thus having industrial applicability. In these embodiments, the device can generate or store data or prompts for license indications regarding the pre-existing sounds used within the generated music data or attached to the data. The advantage of such embodiments is that they can provide a system that facilitates copyright management and licensing.

[0069] In at least one embodiment, the device analyzes motion features in a video (for example, dance movements) in more detail and generates different musical elements (beats and melodies) for each motion feature (for example, dance steps). These musical elements are employed or linked together based on any of the methods of this disclosure. These musical elements may be added to the generated music, provided that they do not conflict with the beats of the generated music.

[0070] In at least one embodiment, the device recognizes a predetermined motion characteristic of an object in a video as a specific motion characteristic and generates a specific sound, which is a predetermined sound, at the time the specific motion characteristic occurs. The device can generate the specific sound in addition to the music to be generated. In another embodiment, the device can generate the specific sound independently of the music to be generated. The user registers information about specific motion characteristics or specific sounds with the device. As a non-exclusive example, the user registers a person's surprised appearance or a predetermined reaction as a specific motion characteristic, and a surprised appearance or prompt as information about a specific sound. Specific sounds include pre-existing music and sound effects specified by the user. The device can store or receive two or more such pre-existing music and sound effects. The device can present two or more such available specific sounds to the user. The user can specify which specific sounds to use from among them. If two or more specific sounds are specified, the device uses them alternately, randomly, or, according to the embodiments of this disclosure, as the specific sounds to be generated. If the video is a two-hour movie, and at 1 hour, 20 minutes, and 43 seconds, a character in the movie shows surprise, the device generates a specific sound representing surprise at that point. According to the above configuration, a specific sound based on a specific action feature can be automatically generated. Furthermore, since the generated specific sound is based on the time the specific action feature occurred, it matches the timing of the specific action feature in the video. In this case, the presented sound can be used industrially as is by playing the video and the generated sound simultaneously. This reduces the workload for those creating the specific sound, and allows for the automatic provision of highly usable sounds with a high degree of matching between the video and the generated sound, thus offering industrial applicability. As already explained, this configuration can be combined with or replaced with one or more embodiments of other embodiments disclosed herein, as long as they do not contradict each other. That is, it is also possible to analyze ambient sounds and audience voices during distribution in real time and generate background music and sound effects (specific sounds) accordingly. In other embodiments, it can be implemented as an application that automatically inserts a surprised sound effect in response to a surprised action.

[0071] In at least one embodiment, the video includes a movie. According to this embodiment, optimal music or specific sounds can be generated separately for each sequence of the movie.

[0072] We will now disclose an embodiment different from the one described above.

[0073] This disclosure describes a method by which a device identifies features of objects in a video and requests an AI to generate music based on those features. This method includes identifying motion features, which are characteristics of actions, and requesting an AI to generate music based on those motion features. According to at least one embodiment, the device or AI represents a video or image as a collection of visual patches, which are small data units similar to text tokens in LLM. The device combines a video recognition AI with LLM to understand each scene in the video. Specifically, it utilizes the video recognition AI to individually recognize various objects and environments that make up the scene, such as people, cars, buildings, animals, natural objects such as trees, and weather, as well as changes in them. The AI ​​learns emotions corresponding to the visual patches by the learning methods described above (including supervised learning). Furthermore, the AI ​​may have the LLM pre-tuned using sample videos of the subject area. For example, the AI ​​learns to correspond to a mood or atmosphere or emotion (collectively referred to as "concept" in this disclosure) for a particular visual patch. Based on the motion features, the device requests an AI to generate music based on those features. For example, if the video is a three-minute music video and the dancer's expression is smiling, the device identifies behavioral characteristics such as bright and cheerful, and requests the AI ​​to generate music based on those characteristics. In another embodiment, the device also requests the AI ​​to identify behavioral characteristics, which are the characteristics of the movement of objects in the video. In yet another embodiment, if the video is a three-minute music video and the background behind the dancer is a ruined building for dramatic effect, the device identifies auxiliary characteristics such as ruins and darkness, and requests the AI ​​to generate music based on those auxiliary characteristics. In yet another embodiment, the video is a film, and the device can analyze the flow of scenes and emotions in the video and generate a corresponding musical structure (intro, climax, ending). Furthermore, the concept-based music includes embodiments corresponding to music genres (classical, EDM, hip hop, etc.). According to this embodiment, music that matches the taste or concept of the video can be automatically generated.This has industrial potential because it reduces the workload of those who create the sound, and allows for the automatic creation of highly usable sounds that have a high degree of conceptual consistency with the video.

[0074] In at least one embodiment, the device may also use user-specified features as operational features or auxiliary features. The user inputs a concept for the music to be generated into the device in text. These characters may be prompts. The device requests the AI ​​to generate music based on the user-inputted concept. Furthermore, the device may request the AI ​​to generate music that is the same length as the video or the time at which the operational features should be identified. In one embodiment, the user can select a mood or atmosphere, and music matching the selected mood can be generated. According to this embodiment, since the user can specify their preferences, music that matches the user's preferences can be automatically generated. This has industrial applicability in that it reduces the workload of the sound creator and automatically provides highly usable sounds with a high degree of conceptual agreement between the video and the generated sound.

[0075] In at least one embodiment, the device analyzes the lyrics as features specified by the user. The device or AI recognizes the concepts contained in the lyrics as features. For example, the device or AI extracts concepts such as love, heartbreak, and sadness based on the lyrics. The device or AI generates sound based on the extracted concepts. If two or more concepts are recognized based on the lyrics, the AI ​​is requested to base the generation on the feature with the highest frequency of occurrence, as described later. This method allows music to be generated based on the main lyric concepts, even if the lyrics are complex. This has industrial applicability in that it reduces the workload of those who create the sound, or in that it can automatically provide highly usable sound with a high degree of conceptual agreement between the video and the generated sound.

[0076] In at least one embodiment, the device requests the AI ​​to convert objects in a video into text. As already described, the AI, or the device, represents videos and images as a collection of visual patches, which are small data units similar to text tokens in LLM. The device combines video recognition AI and LLM to understand each scene in a video. Specifically, it utilizes video recognition AI to individually recognize various objects and environments that make up a scene, such as people, cars, buildings, animals, natural objects like trees, and weather, as well as changes in them. The AI ​​learns the characters corresponding to the visual patches by the learning methods described above (including supervised learning). In this way, the AI ​​can describe (convert to text) a given video into text. The AI ​​generates music based on the converted text. Specifically, the device requests the AI ​​to identify features based on the converted text and generate music based on those features. The method for generating music based on features is disclosed elsewhere in this disclosure.

[0077] This invention discloses a method for requesting an AI to generate music based on features, where, if there are two or more features, the AI ​​is required to base the music on the feature that is relatively longer in duration. In at least one embodiment, the device may recognize two or more features. In this case, the AI ​​is required to base the music on the feature that is longest in duration in the video. The device identifies the duration for which each feature is recognized. As a non-inclusive example, if the video is a three-minute film, and 2 minutes and 40 seconds of it depict a breakup, but 20 seconds depict a fun and exciting time in love as a flashback, then the "breakup" feature is 2 minutes and 40 seconds, and the "fun" feature is 20 seconds. In other words, the device requests the AI ​​to create music based on the longest feature, the "breakup" feature. Note that this method is not intended to be limited to one feature; it is also possible to generate music based on two or more features. According to the above configuration, even if a video has multiple concepts, music can be generated based on the main concept. Therefore, even if a video contains conflicting concepts, the music is based on the main concept, so there is a high probability that the concepts of the video and the music will match. This has industrial potential because it reduces the workload of those who create the sound, and allows for the automatic creation of highly usable sounds that have a high degree of conceptual consistency with the video.

[0078] This invention discloses a method for requesting an AI to generate music based on features, where, if there are two or more features, the AI ​​is required to base the music on the feature that appears relatively more frequently. As already explained, in at least one embodiment, the device may recognize two or more features. In this case, the AI ​​is required to base the music on the feature that appears most frequently in the video. The device identifies the number of times each feature is recognized. As a non-inclusive example, if the video is a three-minute film in which the actors laugh 15 times, cry once, and are angry once, the most frequent feature is "laughing." In other words, the device requests the AI ​​to create music based on the "laughing" feature, which is the longest-running feature. Note that this method is not intended to be limited to one feature; it is also possible to generate music based on two or more features. According to the above configuration, even if a video has multiple concepts, music based on the main concept can be generated. Therefore, even if a video contains conflicting concepts, the music is based on the main concept, so there is a high probability that the concepts of the video will match. This has industrial potential because it reduces the workload of those who create the sound, and allows for the automatic creation of highly usable sounds that have a high degree of conceptual consistency with the video.

[0079] This invention discloses a method for requesting an AI to generate music based on features, wherein the AI ​​generates music for the same duration as the video's duration or the duration of the motion features to be identified, and when two or more dissimilar features are identified, it generates two or more pieces of music and concatenates them. As already explained, the device or AI can generate two or more pieces of music when two or more features are identified. Also, as already explained, the device or AI can concatenate two or more pieces of music. With the above configuration, even for complex videos with two or more dissimilar features, it is possible to automatically create sounds that are similar to the concept of the video. Furthermore, because the sounds are based on two or more concepts, the possibility of inconsistency with the video's concept is further reduced. Therefore, since the created sounds or music are largely synchronized with the video's concept, the degree of agreement between the video and the generated sounds is higher. In this case, the generated sounds can be used industrially as is by playing the video and the generated sounds simultaneously. This reduces the workload of those who create the sounds, and has industrial applicability in that it can automatically generate highly usable sounds with a high degree of conceptual agreement between the video and the generated sounds.

[0080] The present invention discloses a method that requires the music to be concatenated at a point in time identified based on the frequency of occurrence of two or more features. As already explained, the device can calculate the frequency or probability of occurrence of each feature within the time frame of the video. For example, within the time frame of the video or the time frame in which the motion features should be identified, the device identifies a time frame in which the occurrence of a unit time based on feature A is relatively high, and a time frame in which the occurrence of a unit time based on feature B is relatively high. Based on these frequencies, the device generates music based on the unit time with the highest frequency of occurrence. If the unit time with the highest frequency of occurrence changes, the device generates music based on the new unit time. The device concatenates these two pieces of music. In another embodiment, the device generates music for the same time frame as the time frame of the video (the time in which the motion features should be identified), and generates two or more pieces of music when two or more dissimilar motion features are identified. The device identifies the point in time identified based on the frequency of occurrence of two or more dissimilar motion features as the change time. For example, the device identifies time zones within the time frame where the video duration or operational features should be identified, where unit time based on feature A appears relatively frequently, and where unit time based on feature B appears relatively frequently. The device can identify the point in time when the most frequently occurring operational feature changes, or a point in time between or approximately midway through time zones where different operational features appear frequently, as the change time. As a non-inclusive example, suppose a movie video is 10 minutes long. If, in the time zone from 0 minutes 00 seconds to 1 minute 30 seconds, unit time based on feature A (for example, the feature that the setting is in ruins) appears relatively frequently, and in the time zone from 1 minute 30 seconds to 3 minutes, unit time based on feature B (for example, the feature that the setting is in a magnificent building (such as a castle)) appears relatively frequently, the device identifies 1 minute 30 seconds, which is approximately midway between the two or more operational features, based on their respective frequencies. The device then requests that the music be linked at the identified point in time. In other words, the generated music consists of two parts: from 0 minutes 0 seconds to 1 minute 30 seconds, it is music A based on characteristic A; and from 1 minute 30 seconds to 3 minutes, it is music B based on characteristic B. These two parts are joined together at 1 minute 30 seconds.

[0081] As described above, even for complex videos where two or more units of time are identified based on dissimilar features, it is possible to automatically create sounds that are similar to the video's concept. Furthermore, because the sounds are based on two or more units of time, the possibility of mismatch with the video's concept is further reduced. Therefore, the created sounds or music are largely synchronized with the video's concept, resulting in a higher degree of match between the video and the generated sounds. In this case, the generated sounds can be used industrially as is by playing the video and the generated sounds simultaneously. This has industrial potential because it reduces the workload of sound creators and enables the automatic generation of highly usable sounds with a high degree of match between the video and the generated sounds.

[0082] In at least one embodiment, when generating music based on lyrics or text, the device recognizes characters or concepts indicating rhythm within the text and requests the AI ​​to generate music with a rhythm based on those characters or concepts. With the above configuration, music with a specified rhythm can be generated, further reducing the possibility that the generated music will not match the concept of the video. Therefore, the generated sound can be used industrially as is when the video and the generated sound are played simultaneously. This has industrial applicability in that it reduces the workload of those who create the sound, or in that it can automatically generate highly usable sound with a high degree of conceptual match between the video and the generated sound.

[0083] A different embodiment from the above is disclosed. The device requests the AI ​​to create lyrics based on music data. The device or AI breaks down the music data into units such as scales, chords, or melodies. The AI ​​learns the mood, atmosphere, or emotion represented by each unit through supervised learning. Furthermore, the AI ​​learns and fine-tunes using already published combinations of scales, chords, melodies, and lyrics of music. This allows the AI ​​to predict what characters or lyrics will follow given a scale, chords, or melody. It can also predict what words will follow the characters or lyrics that have been given. In this way, the AI ​​can create lyrics based on music data. The device requests the AI ​​to create lyrics based on music data. According to the above configuration, since lyrics based on traditional musical concepts can be generated, the generated lyrics are less likely to be inconsistent with the musical concepts. Furthermore, since the AI ​​performs natural language processing and learning, the generated lyrics are less likely to contain grammatical errors. Therefore, the generated lyrics can be used industrially as is, along with the music. This has industrial applicability in that it reduces the workload of those who create lyrics, or in that it can automatically generate highly usable lyrics with a high degree of conceptual agreement between the music and the generated lyrics. In at least one embodiment, the device further requests the AI ​​to generate a singing voice based on the generated lyrics. In at least one embodiment, the device requests the AI ​​to generate music, requests the AI ​​to create lyrics based on the generated music, requests the AI ​​to generate a singing voice based on the generated lyrics, and requests the AI ​​to combine (integrate, combine) the generated singing voice and music data (or the device itself combines the data). Vocaloid® is included as an application for the AI ​​that generates the singing voice. This method has industrial applicability in that it can automatically generate a singing voice that matches the concept of the video.

[0084] The following is an overview of the embodiments described above.

[0085] A method for creating sound, wherein the device identifies motion features, which are characteristic features of the movement of objects in a video, and requests the AI ​​to generate music in which the time between similar motion features is used as the unit of time, with each unit of time being a beat.

[0086] A method for requesting an AI to generate music with a unit of time as the beat, wherein the AI ​​generates music of the same duration as the time or motion features to be identified in the video, and generates music so that the unit of time in the video and the beats of the music are the same.

[0087] A method for requesting an AI to generate music where each unit of time is a beat, by selecting the relatively shorter unit of time from two or more unit times arising from similar operational characteristics and using that as the beat.

[0088] A method for requesting an AI to generate music with beats based on unit time, which involves selecting a unit time that appears relatively frequently from among two or more unit times arising from similar behavioral characteristics and using that as the beat.

[0089] A method for requesting an AI to generate music with a unit time of beat, wherein the AI ​​generates music of the same duration as the duration or duration of the video where the motion features should be identified, and if two or more units of duration are identified based on dissimilar motion features, the AI ​​generates two or more pieces of music and requests the AI ​​to concatenate them.

[0090] A method comprising the above-mentioned method, wherein the music is requested to be linked at a point determined based on the frequency of occurrence of two or more operational features.

[0091] A method for creating sound, wherein a device identifies the features (feature quantities) of objects in a video and requests an AI to generate music based on those features.

[0092] When requesting an AI to generate music based on features, if there are two or more features, the method requires the AI ​​to use at least the feature whose duration of presence is relatively longer.

[0093] When requesting an AI to generate music based on features, if there are two or more features, the method requires the AI ​​to use at least the feature that appears relatively frequently as the basis for generating the music.

[0094] A method for requesting an AI to generate music based on features, wherein the AI ​​generates music that is the same length as the video or the time at which the features should be identified, and if two or more dissimilar features are identified, the AI ​​generates two or more pieces of music and concatenates them.

[0095] A method comprising the above-mentioned method, wherein the music is requested to be linked at a point determined based on the frequency of occurrence of two or more features.

[0096] In at least one embodiment, the statement "requesting that the relatively shorter unit time be selected from among two or more unit time periods arising from similar operational features be a beat" can be replaced with "requesting that the shortest unit time be selected from among two or more unit time periods arising from similar operational features be a beat." The statement "requesting that the unit time for which a specified unit time occurs relatively frequently be selected from among two or more unit time periods arising from similar operational features be a beat" can be replaced with "requesting that the unit time for which a specified unit time occurs most frequently be selected from among two or more unit time periods arising from similar operational features be a beat." The statement "when there are two or more features, the AI ​​is required to base its selection on the feature for which the feature is observed relatively long" can be replaced with "when there are two or more features, the AI ​​is required to base its selection on the feature for which the feature is observed longest." The statement "when there are two or more features, the AI ​​is required to base its selection on the feature for which the feature is observed relatively frequently be a beat" can be replaced with "when there are two or more features, the AI ​​is required to base its selection on the feature for which the feature is observed most frequently be a beat."

[0097] A method for creating sound, wherein the device generates music of the same duration as the video or a specified duration, and requests an AI to generate music based on the characteristics of lyrics corresponding to the video.

[0098] In the above configuration, the specified time includes the time specified by the user. The device requests the AI ​​to generate music within the time range specified by the user. The method of generating music based on the characteristics of the lyrics and its advantages are as described above.

[0099] In at least one embodiment, the user can make fine adjustments (tempo, key, effects) to the music or singing voice generated by the AI ​​through the device. The device modifies the tempo, key, effects, etc., of the generated music or singing voice data according to the user's adjustments. Such adjustments can be made even if the video is a real-time video or a streamed video.

[0100] In at least one embodiment, two or more users can collaboratively input the rhythm and concept to be generated, and make fine adjustments to the music or vocals generated by the AI, through the device. The device accepts information input from two or more users. Two or more users can collaboratively combine a single piece of data through the device. This editing can be performed even if the video is a real-time video or a streamed video. As a non-exclusive example, this editing function is referred to as "collaboration mode" or "real-time mode".

[0101] In at least one embodiment, the device requests the AI ​​to create lyrics in one or more foreign languages, depending on the lyrics or concept. The device requests the AI ​​to create music corresponding to the input foreign language lyrics or concept. The AI ​​fine-tunes the concept and music according to the foreign language. As a non-exclusive example, it fine-tunes concepts related to ethnic music, such as using the koto for the concept of Japan, or bagpipes for the concepts of Europe or Ireland. The device also requests the AI ​​to generate music in the foreign language concept when foreign language lyrics are input.

[0102] In at least one embodiment, the device requests the AI ​​to generate music or video in a compatible format so that the generated music or video will work on major distribution platforms (non-exclusively, YouTube, TikTok, Instagram, etc., all registered trademarks). In other embodiments, the device converts the music or video generated by the AI ​​into a compatible format so that it will work on major distribution platforms. The advantage of such embodiments is that they create platform-compatible data and improve user convenience.

[0103] A method for creating sound, wherein the device identifies motion features, which are characteristics of the movement of objects that are moving bodies in a video, and requests the AI ​​to generate music in which the time between similar motion features is used as the unit of time, and the unit of time is used as the beat.

[0104] A method for creating sound, wherein the device identifies motion features, which are characteristic features of the movement of objects in a video, and requests the AI ​​to generate music in which the time between similar motion features is used as the unit of time, with each unit of time being one beat.

[0105] A method described above, in which, when requesting the AI ​​to generate music with a unit time of beat, the AI ​​is requested to generate music of the same duration as the time at which the video's duration or motion features should be identified, and to generate music such that the unit time within the video and the beats of the music are the same.

[0106] The above method, wherein the video includes a movie, and music is generated for each sequence or scene of the movie.

[0107] A device for creating sound, which identifies motion features—characteristics of the movement of moving objects within a video—and requests an AI to generate music where the time between similar motion features is used as the unit of time, with each unit of time being a beat.

[0108] A device for creating sound, which identifies motion features, which are characteristic movements of objects in a video, and requests an AI to generate music where the time between similar motion features is used as the unit of time, with each unit of time being one beat.

[0109] The above-described device, which, when requesting the AI ​​to generate music with a unit time of beat, generates music of the same duration as the time to be used to identify the duration or motion features of the video, and generates music such that the unit time in the video and the beats of the music are the same.

[0110] The above-mentioned device, wherein the video includes movies, and the device generates music for each sequence or scene of the movie.

[0111] A computer-readable recording medium in a device equipped with a processor, which records a program that causes the processor to execute the above method.

[0112] A program consisting of instructions that cause the processor to execute the above method.

[0113] The invention disclosed herein only needs to achieve at least one of the effects described above.

Claims

1. A method for creating sound, wherein the device identifies motion features, which are characteristics of the movement of an object that is a moving body in a video, and requests the AI ​​to generate music in which the time between similar motion features is used as the unit time, and the unit time is used as the beat.

2. A method for creating sound, wherein the device identifies motion features that are characteristic of the movement of objects in a video, and requests the AI ​​to generate music in which the time between similar motion features is used as the unit of time, with each unit of time being one beat.

3. A method according to either claim 1 or 2, wherein when requesting an AI to generate music in which the unit time is a beat, the AI ​​generates music of the same duration as the time to be used to identify the duration or motion features of the video, and generates music such that the unit time in the video and the beats of the music are the same.

4. A method according to either claim 1 or 2, wherein the video includes a film, and music is generated for each sequence or scene of the film.

5. A device for creating sound, which identifies motion features, which are characteristics of the movement of objects that are moving bodies in a video, and requests an AI to generate music in which the time between similar motion features is used as the unit time, and the unit time is used as the beat.

6. A device for creating sound, which identifies motion features that are characteristic of the movement of objects in a video, and requests an AI to generate music in which the time between similar motion features is used as the unit of time, with each unit of time being one beat.

7. The apparatus according to claim 5 or 6, wherein when requesting the AI ​​to generate music with a unit time of beats, the apparatus generates music of the same duration as the time at which the duration or motion features of the video should be identified, and requests the AI ​​to generate music such that the unit time in the video and the beats of the music are the same.

8. The apparatus according to claim 5 or 6, wherein the video includes a movie, and the apparatus generates music for each sequence or scene of the movie.

9. A computer-readable recording medium in a device equipped with a processor, which records a program causing the processor to execute the method according to either claim 1 or 2.

10. A program comprising instructions for causing a processor to perform the method according to either claim 1 or 2.

Images