A method and device for generating music scores based on multimodal input of music and / or graphs

Abstract

The present disclosure provides a method and device for generating a music score based on a multimodal input music AND / OR graph, wherein the method comprises: selecting node information of music data, where the node information may be a music segment, a phrase, a measure or a note, obtaining music attribute features corresponding to the node information, where the music attribute features may be a melody, an accompaniment, a harmony or a rhythm, and generating target music based on the node information and the music attribute features matched with the node information, thereby solving the problems of low quality and poor controllability of music generation in the prior art. The present disclosure allows the user to precisely control the process of music generation according to specific needs, including the selection of style, emotion, rhythm and timbre, so that the generated music is more in line with the user's expectations, so that the user can obtain music works that meet personal taste and emotional expression.

Description

Technical Field

[0001] The present invention relates to the field of audio data processing technology, and in particular to a method, device, electronic device, non-transitory computer-readable storage medium storing computer instructions, and computer program product for generating multimodal input music scores based on music and / or images. Background Art

[0002] In recent years, with the development of large pre-trained models, music generation has made significant progress. The current music score generation methods basically rely on deep neural networks and / or large models, which can be convolutional neural networks (CNNs) or transformers, which encode input information to form highly abstract features and constitute feature spaces or latent spaces; in this abstract encoded space, specialized learning of different targets is performed; and then different structures, CNNs or transformers, and different sample generation methods, such as adversarial generation and diffusion models, are used to sample new samples.

[0003] For formal music generation, the common practice is to segment musical elements such as notes and then send them into a deep network or large model based on Transformer for learning to achieve the purpose of music generation. However, this method has disadvantages. First, sending music segmentation into the network will cause the music to lose structural information. Music is an art form with rich structures, including melody, rhythm, harmony and other aspects. Segmentation may destroy the correlation between these structures, resulting in a lack of coherence and integrity in the generated music. Secondly, the massive parameters of the network require a massive amount of segmented music data as a basis. Collecting and organizing a large amount of high-quality music data is an arduous task, and the quality and diversity of the data will also affect the quality of the generated music. In addition, the large amount of data demand also increases computing resources and storage costs. Furthermore, this method can only control the generated music as a whole, lacking control over details. In music creation, details often determine the quality and expressiveness of music. The lack of control over details may make the generated music unsatisfactory in terms of timbre, rhythm changes, emotional expression, etc.

[0004] In summary, the music generated based on the current common methods often has many problems and cannot meet people's demand for high-quality music. Therefore, a new music score generation method is needed to overcome these drawbacks and improve the quality and controllability of music generation. Summary of the invention

[0005] In view of this, the technical problem to be solved by the present disclosure is to provide a method, device, electronic device, non-transitory computer-readable storage medium storing computer instructions and computer program product for generating multimodal input music scores based on music and / or images, so as to solve the problems of low quality and poor controllability of music generation in the prior art.

[0006] According to one aspect of the present disclosure, a method for generating a musical score is provided, the method comprising:

[0007] Selecting node information of the music data; wherein the node information at least includes: a music segment, a phrase, a measure, and a note; a plurality of notes constitute a measure node, a plurality of measures constitute a phrase node, and a plurality of phrases constitute a music segment node;

[0008] Acquire music attribute features corresponding to the node information; wherein the music attribute features include at least: melody, accompaniment, harmony, and rhythm;

[0009] Target music is generated based on the node information and the music attribute features matched with the node information.

[0010] Optionally, before selecting the node information of the music data, the method further includes:

[0011] Obtain an AND-OR graph; wherein the AND-OR graph includes a plurality of AND nodes, a plurality of OR nodes, and a plurality of edges; the AND nodes represent necessary elements in music, the OR nodes represent optional elements in music, and the edges represent relationships between music elements;

[0012] The music data is represented by the AND / OR graph, and the music data is divided into the music attribute features in space and the node information in time.

[0013] Optionally, obtaining the music attribute feature corresponding to the node information includes:

[0014] Collecting an input signal; the input signal reflects the music attribute characteristics;

[0015] The input signal is analyzed to obtain the music attribute feature.

[0016] Optionally, the input signal includes drawing information, and analyzing the input signal to obtain the music attribute feature includes:

[0017] Performing mathematical modeling on the drawing information to obtain a radar chart;

[0018] Calculating the rate of change of curvature of each point of the curve in the radar chart; and / or calculating the rate of change of the first-order derivative of the curve in the radar chart;

[0019] The music attribute feature corresponding to the node information is determined according to the curvature change rate and / or the change rate of the first-order derivative of the curve.

[0020] Optionally, the input signal includes user visual information, and analyzing the input signal to obtain the music attribute feature includes:

[0021] Collecting user visual information including user hand shape information;

[0022] Extracting the hand shape information from the user visual information through a deep neural network; wherein the hand shape information includes at least one of the following: overall hand area information, hand joint position information, and hand coordinate information;

[0023] The music attribute feature corresponding to the node information is obtained according to the hand shape information.

[0024] Optionally, the input signal includes visual information of a user, and analyzing the input signal to obtain the music attribute feature includes:

[0025] Collecting user visual information including user facial information;

[0026] Extracting the user's facial information from the user's visual information through a deep neural network;

[0027] The music attribute feature corresponding to the node information is obtained according to the user face information.

[0028] According to another aspect of the present disclosure, a music score generating device is provided, the device comprising:

[0029] A selection module is used to select node information of music data; wherein the node information at least includes: a music segment, a phrase, a measure, and a note; a plurality of notes constitute a measure node, a plurality of measures constitute a phrase node, and a plurality of phrases constitute a music segment node;

[0030] A first acquisition module is used to acquire music attribute features corresponding to the node information; wherein the music attribute features include at least: melody, accompaniment, harmony, and rhythm;

[0031] A generation module is used to generate target music based on the node information and the music attribute features matching the node information.

[0032] Optionally, the device further comprises:

[0033] A second acquisition module is used to acquire an AND-OR graph; wherein the AND-OR graph includes a plurality of AND nodes, a plurality of OR nodes, and a plurality of edges; the AND nodes represent necessary elements in the music, the OR nodes represent optional elements in the music, and the edges represent the relationship between the music elements;

[0034] The representation module is used to represent the music data through the AND / OR graph, and divide the music data into the music attribute features in space and the node information in time.

[0035] Optionally, the first acquisition module includes:

[0036] A collection unit, used for collecting input signals; the input signals reflect the music attribute characteristics;

[0037] The analysis unit is used to analyze the input signal to obtain the music attribute characteristics.

[0038] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0039] Processor; and

[0040] Memory for storing programs,

[0041] The program includes instructions, which, when executed by the processor, enable the processor to perform the music score generating method according to any one of the above embodiments.

[0042] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the music score generating method according to any one of the above embodiments.

[0043] According to another aspect of the present disclosure, a computer program product is provided. The computer program product includes instructions. When the instructions are executed, a computer is enabled to execute the music score generating method according to any one of the above embodiments.

[0044] In the disclosed embodiment, node information of music data is selected, where the node information may be a musical segment, a musical phrase, a measure or a note, and music attribute features corresponding to the node information are obtained, where the music attribute features may be a melody, an accompaniment, a harmony or a rhythm. Target music is generated based on the node information and the music attribute features matching the node information, thereby solving the problems of low quality and poor controllability of music generation in the prior art. The disclosed embodiment allows the user to precisely control the process of music generation according to specific needs, including the selection of style, emotion, rhythm and timbre, so that the generated music is more in line with the user's expectations, so that the user can obtain music works that conform to personal taste and emotional expression.

[0045] It should be understood that the above general description and the following detailed description are merely exemplary and illustrative and are not intended to limit the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings below are part of the specification of the present invention, and illustrate exemplary embodiments of the present invention. Together with the description of the specification, the accompanying drawings are used to explain the principles of the present invention.

[0047] Figure 1 A flow chart of a music score generation method provided in a specific embodiment of the present invention.

[0048] Figure 2 A schematic diagram of an AOG structure provided in a specific embodiment of the present invention.

[0049] Figure 3 A schematic block diagram of a music score generating device is provided for a specific embodiment of the present invention.

[0050] Figure 4 A schematic block diagram of another music score generating device provided for a specific embodiment of the present invention.

[0051] Figure 5 A schematic block diagram of a first acquisition module provided for a specific embodiment of the present invention.

[0052] Figure 6 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0053] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention more clearly understood, the spirit of the contents disclosed by the present invention will be clearly explained with the accompanying drawings and detailed descriptions below. After understanding the embodiments of the contents of the present invention, any technician in the relevant technical field can change and modify the techniques taught by the contents of the present invention without departing from the spirit and scope of the contents of the present invention.

[0054] The exemplary embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. In addition, elements / components with the same or similar reference numerals used in the drawings and embodiments are used to represent the same or similar parts.

[0055] The terms “first”, “second”, etc. used in this document do not particularly refer to an order or sequence, nor are they used to limit the present invention. They are only used to distinguish elements or operations described with the same technical terms.

[0056] The directional terms used in this document, such as up, down, left, right, front or back, etc., are only used to refer to the directions of the drawings. Therefore, the directional terms used are used to illustrate and not to limit the present invention.

[0057] The words “include,” “including,” “have,” “contain,” etc. used in this document are open-ended terms, meaning including but not limited to.

[0058] As used herein, "and / or" includes any or all combinations of the items described.

[0059] As used herein, “plurality” includes “two” and “more than two”; as used herein, “plurality groups” includes “two groups” and “more than two groups”.

[0060] The terms "substantially" and "approximately" used herein are used to modify any quantity or error that may vary slightly, but these slight changes or errors do not change their essence. Generally speaking, the range of slight changes or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0061] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0062] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). Those skilled in the art should also understand that any transitional conjunctions and / or phrases that substantially represent two or more optional items, whether in the specification, claims, or drawings, should be understood to give the possibility of including one of these items, either of these items, or both of these items. For example, the phrase "A or B" should be understood to include the possibilities of "A" or "B", or "A and B".

[0063] The popularity of multimodal music generation is due to the development of large models in recent years. Its foundation is the Transformer developed in 2017, the development of the generative model paradigm, the development of autoencoders, and the application and development of contrastive learning. With the above theoretical reserve foundation, music generation applications based on it have gradually flourished. The current music generation technology basically relies on deep neural networks and / or large models. In order to obtain the effectiveness of the model, a large amount of training data is required. The volume (parameter quantity) of deep neural networks and large models has developed to billions, resulting in a large amount of data required to obtain usable effects. Obtaining a large amount of data, cleaning a large amount of data, and labeling a large amount of data all consume a lot of costs, such as storage, labor, time, etc. The training cost is positively correlated with the amount of training data. A large amount of training data corresponds to a high training cost. In addition, the existing methods can basically be classified as data-driven methods. Their essence is to use the nonlinear capabilities of deep models to fit the data distribution of existing data. The generation process is equivalent to sampling samples in the learned distribution. This poses a key problem: it is impossible to control, or to control the details. For example, when generating through language, people can only provide general guidance based on the overall style and emotion of the music, but have no way to deal with the details of the music itself. For example, whether the musical motive runs through the entire music and whether the model is used to enhance the musical effect. These common models cannot handle this.

[0064] The foundation of the current deep model method is based on the development of natural language processing, and the essence is to treat music as one-dimensional information along the time dimension. But this does not match the characteristics of music. Natural language certainly has its own internal rules, such as various languages ​​have their own grammar, but after forming paragraphs and articles, they no longer have obvious structural information. Music is exactly different from this. Whether it is more theoretical classical music or contemporary pop music that tends to be free, the structural information of music always exists. For example, the duology, trilogy, and sonata form in classical music; the verse and chorus in pop music. If you rely solely on the data itself to learn, it is difficult to achieve good results, especially for classical music. Because classical music is often long, and the current model learning will normalize the input data, making the input data into "fragments" of consistent length, thus losing the original information of the structure. Therefore, there is no way to solve this problem without changing the training method, or changing the model itself, or even breaking the current paradigm.

[0065] Based on the above problems existing in the prior art, a method for generating a music score is provided in this embodiment. Figure 1 A flow chart of a music score generation method provided by a specific embodiment of the present invention is as follows: Figure 1 As shown, the process includes the following steps:

[0066] Step S102: Selecting node information of the music data, wherein the node information may be a music segment, a phrase, a measure or a note, wherein a plurality of notes constitute a measure node, a plurality of measures constitute a phrase node, and a plurality of phrases constitute a music segment node.

[0067] When selecting the node information of music data, we are actually conducting a structured analysis and decomposition of the music work, which helps to better understand the structure and content of the music and also provides a basis for music generation, editing and parsing. Specific music nodes may include: (1) Notes. Notes are the basic building blocks of music. They represent a single sound in music. In music data, each note node may contain the following information: Pitch: the frequency of the note, usually represented by letters (AG) and sharp and flat symbols. Duration: the length of time the note lasts. Dynamics: the loudness of the note, such as piano or forte. Timbre: the sound quality or color of the note, which is usually related to the type of instrument played. (2) Measures: A measure is a musical structure composed of multiple notes, usually separated by beat lines. A measure node may contain the following information: Time Signature: the time signature of the measure, such as 4 / 4 or 3 / 4, indicating the number of beats in each measure and the duration of each beat. Chords: A combination of notes played simultaneously within a measure to form harmony. Melodic Line: The direction of the melody within a measure. (3) Phrases: A phrase is a musical expression unit consisting of multiple measures, which usually expresses a complete musical idea or theme. A phrase node may contain the following information: Melodic Pattern: The melodic characteristics and patterns in a phrase. Expression: The emotional expression of a phrase, such as excitement, tenderness, etc. Structural Function: The role of a phrase in the overall musical structure, such as theme, chorus or bridge. (4) Sections: A section is a larger musical structural unit consisting of multiple phrases, which usually represents a complete part of the music, such as the exposition, development or recapitulation. A section node may contain the following information: Musical Form: The musical structure and form of a section, such as AABA or ABAB. Key: The main tonality or mode of a section. Dynamic Changes: loudness changes and contrasts within a passage. Thematic Development: the presentation, changes, and development of themes within a passage. By accurately selecting and analyzing these node information, we can gain a deeper understanding of the composition of a musical work, thereby making more effective applications in creation, adaptation, analysis, and teaching. In addition, this structured approach also facilitates the storage, retrieval, and automatic generation of music information.

[0068] Step S104: Acquire the music attribute features corresponding to the node information; wherein the music attribute features include at least: melody, accompaniment, harmony, and rhythm.

[0069] Obtaining music attribute features corresponding to specific node information is a key step in the process of music analysis, creation and modification. Music attribute features may include: (1) Melody: Melody is the most easily recognizable and memorable element in music. It is a series of notes arranged in sequence with specific pitch and rhythm. When obtaining melody attribute features, the following aspects may be included: Pitch Sequence: The pitch sequence of notes in a melody. Melodic Contour: The overall trend of a melody line rising, falling or remaining unchanged. Melodic Motives: Short, characteristic melodic fragments in a melody, usually the basis of musical development. Melodic Rhythm: The arrangement of note durations and rests in a melody. (2) Accompaniment: Accompaniment is the harmonic support below the melody, which provides background and atmosphere for the melody. When obtaining accompaniment attribute features, the following aspects may be included: Chord Progression: The change and movement pattern of chords in the accompaniment. Accompaniment Patterns: The rhythmic and musical patterns in the accompaniment, such as staccato, legato, or a specific rhythmic pattern. Accompaniment Timbre: The timbre of the instruments or sounds used in the accompaniment, which affects the overall musical color. (3) Harmony: Harmony involves the combination of notes that sound simultaneously in music, which provides harmonic support and color to the melody. When obtaining harmonic attribute features, the following aspects can be included: Chord Structure: The vertical combination of notes in a chord, including triads, seventh chords, etc. Harmonic Function: The role of a chord in music, such as the tonic chord, dominant chord, or subdominant chord. Harmonic Density: The number of notes that sound simultaneously in a harmony, from simple to complex. (4) Rhythm: Rhythm is an element of time organization in music, which involves the length, strength, and pauses of notes. When obtaining rhythm attribute features, the following aspects may be included: Beat: The basic time unit of music, usually associated with a specific time signature. Rhythmic Patterns: Repeated rhythmic patterns or rhythmic sequences. Dynamically Rhythm: Changes in strength in rhythm, such as emphasis, rest, and dynamic contrast.

[0070] When obtaining these music attribute features, a variety of technologies and methods can be used, such as spectrum analysis, pattern recognition, machine learning, and deep learning. By analyzing these features, we can better understand the structure and style of music, and provide support for music creation, adaptation, teaching, and theoretical research. In addition, these features can also be used in music information retrieval systems to help users find similar music works based on specific music attributes.

[0071] Step S106: Generate target music based on the node information and the music attribute features matching the node information.

[0072] The choice of "node" is relatively free. If there are sub-nodes A, B, and C at the same level, such as the "sub-node" level, you can choose to add node P at any position in this level when generating music, so that it becomes A, P, B, and C. The effect of doing so is equivalent to adding a new phrase between phrases A and B during the progress of the music, so that the details of the music can be controlled. The above steps allow users to precisely control the process of music generation according to specific needs, including the selection of style, emotion, rhythm, and timbre, so that the generated music is more in line with the user's expectations, so that users can obtain music works that meet their personal tastes and emotional expressions.

[0073] In an optional embodiment, music generation is performed by constructing a music "AND-OR graph". The present invention proposes an AND-OR graph (AOG) using attributes to represent music data, such as Figure 2 As shown, music AOG is a random hierarchical grammar model that integrates music section structure grammar, music hierarchical dependency grammar and attribute grammar. Specifically, an AND-OR graph is obtained, wherein the AND-OR graph includes multiple AND nodes, multiple OR nodes and multiple edges, and the AND nodes represent necessary elements in music, the nodes represent optional elements in music, and the edges represent the relationship between music elements. The music data is represented by the AND-OR graph, and the music data is divided into the above-mentioned music attribute features in space and the above-mentioned node information in time.

[0074] Specifically, construct an AND / OR graph: convert the music data into quantifiable features, such as notes, rhythm, pitch, etc. to pre-process the music data, encode these features, and form a basic music feature vector. Use the music feature vector as the node of the graph, and construct the edge according to the logical relationship in the music data (such as parallel, progressive, transition, etc.). The specific steps can be: (1) Determine the node: map the music feature vector to the node of the graph. (2) Determine the edge: according to the logical relationship in the music data, set the AND, OR, NOT logic gates to connect the corresponding nodes. (3) Set the weight: assign weights to the edges according to the similarity between the music feature vectors.

[0075] The AND-OR graph displays music data in a graphical way, making the music structure more intuitive and convenient for music creation and appreciation. Based on the music data representation of the AND-OR graph, new music clips can be generated to provide inspiration for music creation. The AND-OR graph representation method is suitable for different types of music data, such as classical, popular, and folk. In practical applications, the AND-OR graph can be optimized and adjusted according to needs to achieve better music data representation effects.

[0076] The above embodiment involves obtaining the music attribute characteristics corresponding to the node information. In an optional embodiment, an input signal reflecting the music attribute characteristics is collected, and the input signal is analyzed to obtain the music attribute characteristics. Music attribute characteristics can come from the statistical laws of a large number of music, or can be obtained artificially through different technical means. They are the key to multimodal music generation based on music and / or graphs. This optional embodiment can perform different modal inputs through dynamic input signals to control the details of generated music.

[0077] Optionally, the input signal may be drawing information, and the drawing information is mathematically modeled to obtain a radar chart, and the curvature rate of each point of the curve in the radar chart is calculated, or the change rate of the first-order derivative of the curve in the radar chart is calculated, and the music attribute feature corresponding to the node information is determined according to the curvature change rate and / or the change rate of the first-order derivative of the curve. Specifically, with respect to the music attribute feature of melody, the first-order derivative may correspond to the pitch change rate of the melody line. A large change rate of the first-order derivative may correspond to a jump in the melody (for example, jumping from one note to another note that is far apart), while a small change rate may correspond to a glissando of the melody or a continuous semitone / full tone movement; the curvature change rate can describe the smoothness of the melody line. A smooth melody line will have a smaller curvature change rate, while a melody line containing many abrupt turns and changes will have a larger curvature change rate. Regarding the musical attribute feature of accompaniment, the curvature change rate and the first-order derivative change rate can reflect the complexity and degree of change of the accompaniment. A more complex accompaniment may have a larger curvature change rate and first-order derivative change rate because it contains more notes and changes. For example, in a rock song, a strong accompaniment may have a larger curvature change rate and first-order derivative change rate to match the intense rhythm and melody. In a soft folk song, the accompaniment may be simpler, and the curvature change rate and first-order derivative change rate are relatively small. Regarding the musical attribute feature of harmony, the change of harmony can also be captured by the curvature change rate and the first-order derivative change rate. The transformation of harmony may cause the change of the curve, and the curvature change rate and the first-order derivative change rate can reflect the degree and speed of this change. For example, in a harmonious chord progression, the curvature change rate and the first-order derivative change rate may be relatively stable, showing a smooth curve feature, while in the part where the harmony is disharmonic or tense, the curvature change rate and the first-order derivative change rate may increase to express this tension. Regarding the musical attribute characteristic of rhythm, the curvature change rate and the first-order derivative change rate can directly reflect the change of rhythm. A faster rhythm may have a larger curvature change rate and the first-order derivative change rate because the interval between notes is shorter and the change speed is faster. For example, in a fast-paced dance music, the curvature change rate and the first-order derivative change rate will be relatively large to reflect a strong sense of rhythm, while in a slow-paced lyrical song, the rhythm is relatively slow, and the curvature change rate and the first-order derivative change rate are relatively small. This optional embodiment obtains the corresponding musical attributes by defining these mathematical concepts. By calculating the curvature change rate and the first-order derivative change rate, specific quantitative indicators can be provided for the musical attributes, and the degree of melody transition, the speed of rhythm change, etc. can be accurately measured, making the music analysis more objective and accurate, helping users to better understand the structure and characteristics of music, and providing a basis for further analysis and creation.

[0078] Optionally, the input signal may also be user visual information, and user visual information including user hand shape information is collected. For example, a sensor (such as a depth camera, a motion capture device) may be used to capture user visual information. The hand shape information is extracted from the user visual information through a deep neural network; wherein the hand shape information may be hand overall area information, hand joint position information, hand coordinate information, etc., and the music attribute characteristics corresponding to the node information are obtained according to the hand shape information. Specifically, with respect to the music attribute characteristic of melody, the coordinate position of the hand in space may correspond to the pitch range of the melody. For example, the hand may correspond to high notes when it is in a higher position, and to low notes when it is in a lower position. The direction of change of the hand coordinates may reflect the direction of the melody. For example, the hand moving from left to right may correspond to the rise of the melody, and moving from right to left may correspond to the fall. With respect to the music attribute characteristic of accompaniment, a specific area of ​​the hand coordinates may trigger different accompaniment elements. For example, when the hand moves in a specific area, a specific instrument accompaniment may be triggered. The stability of the coordinates may affect the duration of the accompaniment. If the hand is in a certain area, the accompaniment may be triggered. If one position remains stable, the corresponding accompaniment notes may continue to play; regarding the musical attribute feature of harmony, the relative position relationship of the hand coordinates can determine the spatial distribution of the harmony. For example, different harmony effects can be produced when the hands are in different positions, creating a three-dimensional sense of musical space. The change of coordinates can trigger the change of harmony. When the hand coordinates change, the harmony will also change accordingly; regarding the musical attribute feature of rhythm, the movement speed of the hand coordinates can correspond to the speed of the rhythm. Fast hand movement corresponds to a fast rhythm, while slow movement corresponds to a slow rhythm. The regular change of coordinates can produce a regular rhythm. For example, regular movement of the hand within a certain range can produce a stable rhythm. Through this optional embodiment, the user's hand information can become an important input for controlling the music attributes, thereby creating a new interactive music experience, establishing an intuitive and expressive mapping relationship, so that the user's actions can be naturally transformed into musical expression.

[0079] Optionally, the input signal may also be the user's visual information, and the user's visual information including the user's facial information is collected, and the user's facial information is extracted from the user's visual information through a deep neural network, and the music attribute feature corresponding to the node information is obtained according to the user's facial information. Specifically, facial recognition technology is used to detect and locate the user's facial features, such as the position and shape of the eyes, nose, mouth, and eyebrows, and facial expression features are extracted, such as the dynamic changes of expressions such as smiling, frowning, and opening the mouth. The user's emotional state, such as happiness, sadness, anger, etc., is analyzed through facial expressions, and the extracted facial visual features are mapped to the music attributes. Regarding the music attribute feature of melody; facial expression: smiling may be mapped to a bright melody, while frowning may be mapped to a low or complex melody, and eye movements: the opening or squinting of the eyes can affect the dynamic range of the melody, such as opening the eyes wide may correspond to the rise of the melody, and squinting the eyes may correspond to the decrease of the melody. Regarding the musical attribute feature of accompaniment, head movement: the tilt or rotation of the head can control the rhythm and intensity of the accompaniment, such as tilting to one side may increase the accompaniment sound on that side; facial expression changes: the intensity and speed of the expression can affect the complexity and change speed of the accompaniment. Regarding the musical attribute feature of harmony, emotional state: the results of emotional analysis can be used to select the type of harmony, such as happy emotions may correspond to harmonious harmony, while sad emotions may correspond to dissonant intervals; mouth shape: the degree of opening and shape of the mouth can be mapped to the thickness and tension of the harmony. Regarding the musical attribute feature of rhythm, blinking frequency: the frequency of blinking can affect the speed of the rhythm, such as frequent blinking may correspond to a fast rhythm; head shaking: the shaking of the head can produce a sense of rhythm, and the speed and amplitude of the shaking can control the speed and strength of the rhythm. Through this optional embodiment, the user's facial visual information can become an important dimension for controlling music creation, creating an expressive and interactive music experience, and the mapping relationship is both intuitive and can accurately reflect the user's intentions, while maintaining the beauty and coherence of the music.

[0080] In this embodiment, a music score generating device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0081] This embodiment provides a music score generating device, such as Figure 3 As shown, the device comprises:

[0082] A selection module 32 is used to select node information of the music data; wherein the node information at least includes: a music segment, a phrase, a measure, and a note; a plurality of notes constitute a measure node, a plurality of measures constitute a phrase node, and a plurality of phrases constitute a music segment node;

[0083] The first acquisition module 34 is used to acquire the music attribute feature corresponding to the node information; wherein the music attribute feature includes at least: melody, accompaniment, harmony, and rhythm;

[0084] The generating module 36 is used to generate the target music based on the node information and the music attribute feature matching the node information.

[0085] Alternatively, if Figure 4 As shown, the device also includes:

[0086] A second acquisition module 42 is used to acquire an AND-OR graph; wherein the AND-OR graph includes a plurality of AND nodes, a plurality of OR nodes, and a plurality of edges; the AND nodes represent necessary elements in the music, the OR nodes represent optional elements in the music, and the edges represent the relationships between the music elements;

[0087] The representation module 44 is used to represent the music data through the AND / OR graph, and divide the music data into the music attribute features in space and the node information in time.

[0088] like Figure 5 As shown, the first acquisition module 34 includes:

[0089] The collection unit 342 is used to collect input signals; the input signals reflect the attribute characteristics of the music;

[0090] The analysis unit 344 is used to analyze the input signal to obtain the music attribute characteristics.

[0091] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0092] The exemplary embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present invention when executed by the at least one processor.

[0093] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0094] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

[0095] refer to Figure 6 , a block diagram of an electronic device that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0096] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0097] A plurality of components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 may be any type of device capable of inputting information to the electronic device 600, and the input unit 606 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 607 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 may include, but is not limited to, a disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0098] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the music data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. In some embodiments, the computing unit 601 may be configured to perform a method according to an embodiment of the present invention in any other appropriate manner (e.g., by means of firmware).

[0099] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0100] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0101] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0102] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0103] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0104] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0105] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A method for generating music scores from multimodal input based on music and / or graphs, characterized in that: The method comprises: Selecting node information of the music data; wherein the node information at least includes: a music segment, a phrase, a measure, and a note; a plurality of notes constitute a measure node, a plurality of measures constitute a phrase node, and a plurality of phrases constitute a music segment node; Acquire music attribute features corresponding to the node information; wherein the music attribute features include at least: melody, accompaniment, harmony, and rhythm; generating target music based on the node information and the music attribute features matched with the node information; Before selecting the node information of the music data, the method further includes: Obtain an AND-OR graph; wherein the AND-OR graph includes a plurality of AND nodes, a plurality of OR nodes, and a plurality of edges; the AND nodes represent necessary elements in music, the OR nodes represent optional elements in music, and the edges represent relationships between music elements; The music data is represented by the AND / OR graph, and the music data is divided into the music attribute features in space and the node information in time; Acquiring the music attribute feature corresponding to the node information includes: Collecting an input signal; the input signal reflects the music attribute characteristics; Analyzing the input signal to obtain the music attribute feature; The input signal includes drawing information, and analyzing the input signal to obtain the music attribute feature includes: Performing mathematical modeling on the drawing information to obtain a radar chart; Calculate the curvature change rate of each point of the curve in the radar chart; Calculate the rate of change of the first-order derivative of the curve in the radar chart; The music attribute feature corresponding to the node information is determined according to the curvature change rate and the change rate of the first-order derivative of the curve.

2. The method for generating a musical score according to claim 1, characterized in that: The input signal includes user visual information, and analyzing the input signal to obtain the music attribute feature includes: Collecting user visual information including user hand shape information; Extracting the hand shape information from the user visual information through a deep neural network; wherein the hand shape information includes at least one of the following: overall hand area information, hand joint position information, and hand coordinate information; The music attribute feature corresponding to the node information is obtained according to the hand shape information.

3. The method for generating a musical score according to claim 1, wherein: The input signal includes visual information of the user, and analyzing the input signal to obtain the music attribute feature includes: Collecting user visual information including user facial information; Extracting the user's facial information from the user's visual information through a deep neural network; The music attribute feature corresponding to the node information is obtained according to the user face information.

4. A multi-modal input music score generation device based on music and / or images, characterized in that: The device comprises: A selection module is used to select node information of music data; wherein the node information at least includes: a music segment, a phrase, a measure, and a note; a plurality of notes constitute a measure node, a plurality of measures constitute a phrase node, and a plurality of phrases constitute a music segment node; A first acquisition module is used to acquire music attribute features corresponding to the node information; wherein the music attribute features include at least: melody, accompaniment, harmony, and rhythm; A generating module, configured to generate target music based on the node information and the music attribute features matching the node information; the device further comprises: A second acquisition module is used to acquire an AND-OR graph; wherein the AND-OR graph includes a plurality of AND nodes, a plurality of OR nodes, and a plurality of edges; the AND nodes represent necessary elements in the music, the OR nodes represent optional elements in the music, and the edges represent the relationship between the music elements; A representation module, used for representing the music data through the AND-OR graph, dividing the music data into the music attribute features in space and the node information in time; The first acquisition module includes: A collection unit, used for collecting input signals; the input signals reflect the music attribute characteristics; An analysis unit, used for analyzing the input signal to obtain the music attribute feature; The input signal includes drawing information, and the acquisition unit and the analysis unit are further used for: Performing mathematical modeling on the drawing information to obtain a radar chart; Calculate the curvature change rate of each point of the curve in the radar chart; Calculate the rate of change of the first-order derivative of the curve in the radar chart; The music attribute feature corresponding to the node information is determined according to the curvature change rate and the change rate of the first-order derivative of the curve.

5. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-3.

7. A computer program product, characterized in that The computer program product comprises instructions which, when executed, cause a computer to perform the method according to any one of claims 1 to 3.

Images