Music analysis device and program

Abstract

This invention provides a music analysis device and program that can automatically determine the values ​​of emotional parameters based on data representing music. [Solution] The music analysis device includes an analysis processing unit. Based on music data representing the music, the analysis processing unit determines at least one of the following values: a first emotion parameter value representing the degree of emotion of "magnificent, ornate"; a second emotion parameter value representing the degree of emotion of "strange, eerie"; a third emotion parameter value representing the degree of emotion of "angry, defiant"; and a fourth emotion parameter value representing the degree of emotion of "passionate".

Description

Technical Field

[0001] The present invention relates to a music analysis apparatus and a program.

Background Art

[0002] It is expected that generating theoretically music-based visual expressions will enable production of content with rich expressiveness.

[0003] Non-Patent Document 1 describes a technique for generating a visual expression that matches the emotion expressed by music based on emotion parameters.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0005] However, with the technique described in Non-Patent Document 1, a person has to manually create the emotion parameters that serve as the basis for generating a visual expression. That is, conventionally, a person has determined the values of the emotion parameters while referring to, for example, the score of the music to be performed. Then, a person manually creates an "emotion score" that includes both information representing the score and the values of the emotion parameters. It is considered desirable to improve the efficiency of such manual work in content production.

[0006] This invention was made based on the above-mentioned problem recognition, and aims to provide a music analysis device and program that can automatically determine the value of emotional parameters based on data representing music. [Means for solving the problem]

[0007] [1] In order to solve the above problems, a music analysis device according to one aspect of the present invention comprises an analysis processing unit that determines, based on music data representing music, at least one of the following: a value of a first emotion parameter representing the degree of emotion of "magnificent, ornate"; a value of a second emotion parameter representing the degree of emotion of "strange, eerie"; a value of a third emotion parameter representing the degree of emotion of "angry, defiant"; and a value of a fourth emotion parameter representing the degree of emotion of "passionate".

[0008] [2] In addition, in one aspect of the present invention, the music analysis device described in [1] above, the music data contains information on the pitch, length, and timing of each sound that constitutes the music.

[0009] [3] Another aspect of the present invention is that, in the music analysis device described in [1] or [2] above, the analysis processing unit determines the value of the first emotion parameter to a value that indicates the emotion of "splendid, ornate" when both a slur and a staccato are indicated in the music data, or when a trill is indicated in the music data, or when the piece is in a major key and all the notes in a measure or beat correspond to notes shorter than sixteenth notes, or when the piece is in a major key and two-thirds or more of the notes in a measure or beat correspond to sixteenth notes.

[0010] [4] Another aspect of the present invention is the music analysis device described in [3] above, wherein the analysis processing unit determines the value of the first emotion parameter when a trill is indicated in the music data, according to how many notes the trill is indicated over.

[0011] [5] Another aspect of the present invention is a music analysis device according to any of the above [1] to [4], wherein the analysis processing unit determines the value of the second emotion parameter to a value that indicates the emotion of "strange, eerie" when, in the music data, there is a modulation from major to minor and the modulation is a modulation of different fundamental tones, or when, in the music data, there are two or more consecutive measures of minor and octave-structured notes in the accompaniment part, or when, in the music data, a minor and two-measure phrase is repeated an octave lower, or when, in the music data, the same note in a minor and octave-structured form is repeated for two or more measures, or when, in the music data, there is a repetition of a unit of minor and descending triplet, or when, in the music data, there is one or more instances of fundamental tone + unstable chord, and the highest note of a stable chord is raised by a semitone.

[0012] [6] Another aspect of the present invention is a music analysis device according to any of the above [1] to [5], wherein the analysis processing unit determines when the music data has an accompaniment part that rises by only one tone every two measures, or when the music data has an accompaniment part that rises by one tone every two measures in addition to the accompaniment part that rises by one tone every two measures, or when the music data has an accompaniment part that has multiple sets of notes that rise by one tone each in a minor key, or when the music data has an accompaniment part that has multiple sets of notes that rise by one tone each in a minor key The third emotion parameter is determined to represent the emotion of "angry, defiant" when, and thereafter the note descends one note at a time within a measure, or when, in the aforementioned music data, the accompaniment part is in a minor key and has an octave structure with the same note rising by a third, or when, in the aforementioned music data, the accompaniment part has a structure where the note rises one note every two measures, and there are multiple sets of notes that rise one note at a time in a minor key, and there are multiple sets of notes that rise one note at a time in a minor key.

[0013] [7] Another aspect of the present invention is a music analysis device according to any of the above [1] to [6], wherein the analysis processing unit determines the value of the fourth emotion parameter to a value that indicates the emotion of "passionate" when, in the music data, the pitch of the notes rises by three octaves or more within two measures, or when, in the music data, the pitch of the notes rises by three octaves or more within one measure, or when, in the music data, the same fundamental tone / highest note continues for three notes or more.

[0014] [8] Another aspect of the present invention is a program for causing a computer to function as a music analysis device, comprising an analysis processing unit that determines, based on music data representing music, the value of a first emotion parameter representing the degree of emotion of "magnificent, ornate," the value of a second emotion parameter representing the degree of emotion of "strange, eerie," the value of a third emotion parameter representing the degree of emotion of "angry, defiant," and the value of a fourth emotion parameter representing the degree of emotion of "passionate." [Effects of the Invention]

[0015] According to the present invention, the value of the emotional parameter can be automatically determined by analyzing music data. [Brief explanation of the drawing]

[0016] [Figure 1] A functional block diagram showing the schematic functional configuration of a system according to an embodiment of the present invention. [Figure 2] This is a schematic diagram showing an example of music data containing information from the analysis results performed by the analysis processing unit of the same embodiment. [Figure 3] This is a schematic diagram showing an example of a visualization generated by the visualization generation unit in the same embodiment, corresponding to the emotional parameters "magnificent, decorated". [Figure 4] This is a schematic diagram showing an example of a visualization generated by the visualization generation unit in the same embodiment, corresponding to the emotional parameters "strange, eerie". [Figure 5] It is a schematic diagram (Part 1) illustrating an example of a visualization expression generated by a visualization expression generation unit in response to an emotion parameter "angry and challenging" in the same embodiment. [Figure 6] It is a schematic diagram (Part 2) illustrating an example of a visualization expression generated by a visualization expression generation unit in response to an emotion parameter "angry and challenging" in the same embodiment. [Figure 7] It is a schematic diagram illustrating an example of a visualization expression generated by a visualization expression generation unit in response to an emotion parameter "" in the same embodiment. [Figure 8] It is a schematic diagram illustrating an example of a result obtained by the combining unit of the visualization expression generation device according to the same embodiment, which combines the visualization expression generated by the visualization expression generation unit (an expression of the emotion parameter "gorgeous and ornate") and a video read out from a content storage unit. [Figure 9] It is a schematic diagram illustrating an example of a result obtained by the combining unit of the visualization expression generation device according to the same embodiment, which combines the visualization expression generated by the visualization expression generation unit (an expression of the emotion parameter "strange and eerie") and a video read out from a content storage unit. [Figure 10] It is a schematic diagram illustrating an example of a result obtained by the combining unit of the visualization expression generation device according to the same embodiment, which combines the visualization expression generated by the visualization expression generation unit (an expression of the emotion parameter "angry and challenging") and a video read out from a content storage unit. [Figure 11] It is a schematic diagram illustrating an example of a result obtained by the combining unit of the visualization expression generation device according to the same embodiment, which combines the visualization expression generated by the visualization expression generation unit (an expression of the emotion parameter "passionate") and a video read out from a content storage unit. [Figure 12] It is a block diagram illustrating an example of the internal configuration of a music analysis device and a visualization expression generation device according to the same embodiment. DETAILED DESCRIPTION OF EMBODIMENTS

[0017] Next, an embodiment of the present invention will be described with reference to the drawings.

[0018] In the following embodiments, a visualization representation is generated based on emotional parameters. The emotional parameter "splendid, decorated," which represents the degree of the emotion of "magnificent, decorated," is sometimes referred to as the first emotional parameter. The emotional parameter "strange, eerie," which represents the degree of the emotion of "strange, eerie," is sometimes referred to as the second emotional parameter. Furthermore, the emotion parameter "Angry, defiant," which represents the degree of the emotion "Angry, defiant," is sometimes called the third emotion parameter. Similarly, the emotion parameter "Passionate," which represents the degree of the emotion "Passionate," is sometimes called the fourth emotion parameter.

[0019] In the embodiments described below, the value of the first emotion parameter representing the emotion of "magnificent, ornate" is a non-zero value. The value of the second parameter representing the emotion of "strange, eerie" is a non-zero value. The value of the third parameter representing the emotion of "angry, defiant" is a non-zero value. The value of the fourth parameter representing the emotion of "passionate" is a non-zero value.

[0020] The terminology used in this embodiment will now be explained. MIDI is an abbreviation for "Musical Instruments Digital Interface." MIDI is sometimes pronounced "midi." MIDI refers to an existing unified standard for connecting electronic musical instruments such as synthesizers. In other words, because MIDI is standardized, it is a data format that can be input and output to any device, regardless of the manufacturer or product of the electronic musical instrument.

[0021] Figure 1 is a functional block diagram showing the schematic functional configuration of the system according to this embodiment. As shown in the figure, system 1 consists of a music analysis device 2 and a visualization representation generation device 3. The system is configured to allow data transfer from the music analysis device 2 to the visualization representation generation device 3. In this embodiment, the data that the music analysis device 2 transfers to the visualization representation generation device 3 is the value of a parameter based on music data.

[0022] At least some of the functions of the music analysis device 2 and the visualization representation generation device 3 can be implemented, for example, by a computer and a program. Each functional unit may have storage means as needed. The storage means may be, for example, variables in the program or memory allocated by the execution of the program. Alternatively, non-volatile storage means such as a magnetic hard disk drive or a solid-state drive (SSD) may be used as needed. Furthermore, at least some of the functions of each functional unit may be implemented as a dedicated electronic circuit instead of a program.

[0023] The music analysis device 2 analyzes the music data based on the music data, determines the values ​​of predetermined parameters, and outputs them.

[0024] The visualization representation generation device 3 generates a visualization representation based on the parameter values ​​received from the music analysis device 2. In other words, the visualization representation generation device 3 generates a visualization representation corresponding to the original music based on the parameter values ​​related to music and based on predetermined design rules.

[0025] Here, a visualization is a visual representation corresponding to the values ​​of parameters. These parameters are determined based on music data. In other words, a visualization is a visual representation corresponding to the music (or each part thereof). A visualization can be represented, for example, as a collection of objects having a predetermined shape, color, and movement pattern. Displaying these objects on a computer screen or similar display is possible using existing technologies.

[0026] The music analysis device 2 comprises a music data supply unit 21, an analysis processing unit 22, and a parameter output unit 23.

[0027] The music data supply unit 21 supplies music data to the analysis processing unit 22. This music data basically contains information about the pitch, length, and timing of each sound that makes up the music. The music data may also contain information about the timbre of each sound. The notes in the musical score correspond to the above sounds. In the music data, sounds may overlap in time.

[0028] Music data is often represented and supplied as MIDI data. MIDI data contains information equivalent to musical notation. However, MIDI data may also reflect the characteristics of the performer's performance, not just provide equivalent information.

[0029] In music data, pronunciation is associated with time; that is, the timing of each pronunciation is defined within the music data. Furthermore, the progression of time is associated with beats and measures. Music data may also contain pronunciation data for each beat or measure.

[0030] In music data, the key signature is identifiable. Furthermore, it is possible to determine whether the key signature is major or minor. Modulation can be understood based on the music data. Additionally, modulation from major to minor, and from minor to major, can be understood based on the music data.

[0031] Musical scores include performance instructions. These instructions include, for example, slurs, staccatos, trills, and other symbols. Musical data can contain information corresponding to these instructions.

[0032] In music data, chords can be identified. Chords can be represented, for example, by combinations of letters, numbers, etc.

[0033] In music data, the fundamental tone of a chord can be identified. The fundamental tone is the lowest note in a chord and determines the overall key. Because the fundamental tone can be identified in music data, it is possible to determine, based on the music data, whether a modulation involves a different fundamental tone or not.

[0034] Furthermore, in music data, it is possible to determine whether a set of sounds constitutes the accompaniment or not. In other words, it is possible to distinguish whether a set of sounds is part of the accompaniment or another part (for example, the main melody).

[0035] An octave is an interval of eight degrees. In other words, an octave is the interval (one octave) from which you can reach the same note at different pitches in a seven-note scale. The frequency ratio of notes separated by one octave is 2:1.

[0036] Furthermore, music data is constructed based on music theory. The characteristics of music data described here can be used during music analysis.

[0037] The analysis processing unit 22 determines, based on the music data supplied from the music data supply unit 21, the value of a first emotion parameter representing the degree of emotion "magnificent, ornate," the value of a second emotion parameter representing the degree of emotion "strange, eerie," the value of a third emotion parameter representing the degree of emotion "angry, defiant," and the value of a fourth emotion parameter representing the degree of emotion "passionate." The analysis processing unit 22 may also determine the values ​​of other emotion parameters based on the above music data. Furthermore, the analysis processing unit 22 may determine not only emotion parameters but also music theory parameters.

[0038] In other words, the analysis processing unit 22 determines the value of at least one of the music theory parameters and the emotion parameters based on the music data. The analysis processing unit 22 may also determine both the music theory parameters and the emotion parameters based on the music data. These music theory parameters and emotion parameters are the information that the visualization representation generation device 3 uses as the basis for generating the visualization representation.

[0039] The analysis processing unit 22 reads the music data sequentially from the beginning, and calculates and outputs emotional parameters as appropriate in accordance with the progress of the music (passage of time). The analysis processing unit 22 may calculate the emotional parameters after reading all of the music data, or it may calculate and output the emotional parameters sequentially at any point in the music data where possible.

[0040] The analysis processing unit 22 may, for example, determine the emotion parameter for each individual sound. Alternatively, the analysis processing unit 22 may determine the emotion parameter for each beat. Furthermore, the analysis processing unit 22 may determine the emotion parameter for each measure.

[0041] The analysis processing unit 22 performs the following processing to determine the value of the first emotion parameter. Specifically, the analysis processing unit 22 determines the value of the first emotion parameter to a value that indicates the emotion of "magnificent, ornate" in at least one of the following cases: when both a slur and a staccato are indicated in the music data, when a trill is indicated in the music data, when the piece is in a major key and all the notes within a measure or beat correspond to notes shorter than sixteenth notes, or when the piece is in a major key and two-thirds or more of the notes within a measure or beat correspond to sixteenth notes.

[0042] Furthermore, when a trill is indicated in the music data, the analysis processing unit 22 may determine the value of the first emotion parameter according to how many notes the trill is indicated to span.

[0043] Furthermore, the analysis processing unit 22 performs the following processing to determine the value of the second emotion parameter. Specifically, the analysis processing unit 22 determines the value of the second emotion parameter to a value that represents the emotion of "strange, eerie" when, in the music data, there is a modulation from major to minor, and the modulation is a modulation with a different fundamental tone, and in the accompaniment part, there are two or more consecutive minor notes with an octave structure, and there are two or more consecutive minor notes with a descending triad unit, and there are two or more consecutive minor notes with a descending triad unit, or there are one or more instances of fundamental tone + unstable chord, and the highest note of a stable chord is raised by a semitone.

[0044] Furthermore, the analysis processing unit 22 performs the following processing to determine the value of the third emotion parameter. Specifically, the analysis processing unit 22 determines the value of the third emotion parameter to a value that indicates the emotion of "angry, defiant" when, in the music data, the accompaniment part has a structure in which the melody rises by one tone every two measures, and there is an additional structure in which the melody rises by one tone every two measures, and there are multiple sets of notes that rise one tone at a time in a minor key, and there are multiple sets of notes that rise one tone at a time in a minor key, and there are then sets of notes that fall one tone at a time within a measure, and there is an interval rise of a third of the same note in a minor key and octave structure, or when, in the music data, the accompaniment part has a structure in which the melody rises by one tone every two measures, and there is an additional structure in which the melody rises by one tone every two measures, and there are multiple sets of notes that rise one tone at a time in a minor key, and there are multiple sets of notes that rise one tone at a time in a minor key.

[0045] Furthermore, the analysis processing unit 22 performs the following processing to determine the value of the fourth emotion parameter. Specifically, the analysis processing unit 22 determines the value of the fourth emotion parameter to a value that indicates the emotion of "passionate" when, in the music data, the pitch rises by three octaves or more within two measures, or when, in the music data, the pitch rises by three octaves or more within one measure, or when, in the music data, the same fundamental note / highest note continues for three notes or more.

[0046] The method by which the analysis processing unit 22 determines the values ​​of each emotion parameter will be explained in more detail later.

[0047] The parameter output unit 23 outputs the parameters determined by the analysis processing unit 22. The parameter output unit 23 outputs the parameters determined by the analysis processing unit 22 in relation to the timing of sound production in music. The parameter output unit 23 passes the music data and the values ​​of the emotion parameters associated with that music data to the visualization expression generation unit 31 of the visualization expression generation device 3. In addition to the values ​​of the emotion parameters, the parameter output unit 23 may also pass the values ​​of the music theory parameters determined by the analysis processing unit 22 to the visualization expression generation unit 31 of the visualization expression generation device 3 in the same manner.

[0048] The visualization representation generation device 3 comprises a visualization representation generation unit 31, a content storage unit 32, and a synthesis unit 33.

[0049] The visualization generation unit 31 generates a visualization based on the value of the emotion parameter. In other words, the visualization generation unit 31 generates a visualization based on the emotion parameter associated with the music, in accordance with the progression of the music. Specifically, the visualization generation unit 31 receives the emotion parameter from the parameter output unit 23 of the music analysis device 2 and generates a visualization based on that emotion parameter. The visualization generation unit 31 then passes the generated visualization to the synthesis unit 33.

[0050] Specifically, the visualization expression generation unit 31 may generate a visualization expression based on at least one of four types of emotional parameters, such as (A) "magnificent, ornate," (B) "strange, eerie," (C) "angry, defiant," and (D) "passionate." The visualization expression generation unit 31 may also generate a visualization expression based on emotional parameters other than the four types exemplified herein. Furthermore, the visualization expression generation unit 31 may generate a visualization expression based not only on emotional parameters but also on music theory parameters.

[0051] The content storage unit 32 stores video content. The stored video content includes video and audio. For example, the content storage unit 32 stores video content that includes a music performance scene. The music included in the video content stored by the content storage unit 32 may be music that the analysis processing unit 22 is analyzing.

[0052] The synthesis unit 33 combines the visualization representation generated by the visualization representation generation unit 31 with the video content read from the content storage unit 32. For example, the synthesis unit 33 enables the visualization representation generated by the visualization representation generation unit 31 and the video content read from the content storage unit 32 to be displayed overlaid on top of each other.

[0053] Figure 2 is a schematic diagram showing an example of music data containing information from the analysis results performed by the analysis processing unit 22. The information shown in Figure 2 includes information from the musical score and information from various parameters. The example shown in Figure 2 represents the temporal progression of the values ​​(non-zero values) of the emotional parameters "magnificent, ornate" (first emotional parameter), "strange, eerie" (second emotional parameter), "angry, defiant" (third emotional parameter), and "passionate" (fourth emotional parameter). The values ​​of these emotional parameters are determined by the analysis processing unit 22. In other words, in the information shown in Figure 2, the timing of the musical notes (corresponding to sound production) and the timing of the parameter values ​​are interrelated. The visualization representation generation unit 31 receives the entire information corresponding to Figure 2 from the music analysis device 2 and generates a visualization representation.

[0054] In other words, the information shown in Figure 2 includes performance data, emotional parameters, and music theory parameters. Emotional parameters are musical information that quantifies the impression of the music based on its note structure. Performance data is musical information based on data related to the performer's characteristics. Music theory parameters are musical information written in the musical score.

[0055] The emotional parameter can be a numerical representation of the impression created by the performance instructions (slurs, staccato, trills, etc.) noted in the musical score, based on a set of rules.

[0056] Furthermore, the emotional parameter may be a numerical representation of the impression created by the note-like structure (simultaneous) noted in the musical score, based on established rules.

[0057] Furthermore, the emotional parameter may be a numerical representation of the impression created by the note-like structure (chronologically) noted in the musical score, based on established rules.

[0058] The determined parameter values ​​may be positioned on the time axis of the content data (video, audio, MIDI, etc.) to generate input data for visualization.

[0059] The 16 emotional components include, for example, bright / dark, tense / relaxed, strong / calm, attack / accent (exhilaration), magnificent / ornate, strange / eerie, angry / challenging, passionate, and so on.

[0060] Next, the process by which the analysis processing unit 22 actually determines the emotional parameters based on the music data will be described. In this embodiment, the types of emotional parameters determined by the analysis processing unit 22 are at least one of the following four types of emotional parameters: (A) "magnificent, ornate", (B) "strange, eerie", (C) "angry, defiant", and (D) "passionate". The analysis processing unit 22 may also determine values ​​for emotional parameters other than these four types.

[0061] (A) Emotional parameter: "Magnificent, ornate" The analysis processing unit 22 determines the value of the emotion parameter "magnificent, adorned" when the following conditions (A1) to (A9) are met. The value of the emotion parameter "magnificent, adorned" determined by the analysis processing unit 22 is an integer between 0 and 5.

[0062] (A1) When both slur and staccato are used ⇒ Set the value of the emotion parameter "magnificent, ornate" to "2".

[0063] (A2) When there is a trill spanning five or more notes ⇒ Set the value of the emotion parameter "magnificent, ornate" to "5".

[0064] (A3) When there is a trill spanning four notes ⇒ Set the value of the emotion parameter "magnificent, ornate" to "4".

[0065] (A4) When there is a trill spanning three notes ⇒ Set the value of the emotion parameter "magnificent, ornate" to "3".

[0066] (A5) When there is a trill spanning two notes ⇒ Set the value of the emotion parameter "magnificent, ornate" to "2".

[0067] (A6) When there is a trill that spans a single note ⇒ Set the value of the emotion parameter "magnificent, ornate" to "1".

[0068] (A7) If the song is in a major key and all notes within a measure or beat are sixteenth notes or longer (sixteenth notes, thirty-second notes, sixty-fourth notes, etc., notes of the same length as or shorter than a sixteenth note) ⇒ Set the value of the emotion parameter "magnificent, ornate" to "3".

[0069] (A8) If the song is in a major key and more than two-thirds of the notes within a measure or beat are sixteenth notes, then set the emotional parameter "magnificent, ornate" to "2".

[0070] (A9) If none of the above conditions A1 through A8 are met, then the value of the emotion parameter "magnificent, decorated" is set to "0".

[0071] (B) Emotional parameter: "Strange, eerie" The analysis processing unit 22 determines the value of the emotion parameter "strange, eerie" when the following conditions (B1) to (B4) are met. The value of the emotion parameter "strange, eerie" determined by the analysis processing unit 22 is an integer between 0 and 3.

[0072] (B1) There is a modulation from major to minor, and the modulation is one in which the root note is different. Alternatively, in the accompaniment section, there are two or more consecutive measures of notes in a minor key and in an octave configuration. Alternatively, a minor key and a two-bar phrase are repeated an octave lower. In either of the following cases ⇒ Set the value of the emotion parameter "strange, eerie" to "3". An example of a modulation with a different fundamental tone is the modulation from A major to C-sharp minor.

[0073] (B2) When there are two or more consecutive measures of the same note in a minor key and an octave structure, or when there is a repetition of a descending trinote unit in a minor key, or a repetition of a ascending trinote unit in a minor key ⇒ Set the emotional parameter "strange, eerie" to "2".

[0074] (B3) If there is one or more instances of the root note + unstable chord, or if the highest note of a stable chord is raised by a semitone, then the emotional parameter "strange, eerie" is set to "1". An example of a root note + unstable chord is the A+F#7(omit3) chord.

[0075] (B4) If none of the above conditions from B1 to B3 are met ⇒ Set the value of the emotion parameter "strange, creepy" to "0".

[0076] (C) Emotional parameter: "Angry, defiant" The analysis processing unit 22 determines the value of the emotion parameter "angry, defiant" when the following conditions (C1) to (C4) are met. The value of the emotion parameter "angry, defiant" determined by the analysis processing unit 22 is an integer between 0 and 3, inclusive.

[0077] (C1) When the accompaniment consists of only one note rising every two measures (this is called "composition 1"), set the value of the emotion parameter "angry, defiant" to "1".

[0078] (C2) If the accompaniment section has a structure where it rises by one tone every two measures on top of the above structure 1, or if there are multiple sets of notes that rise by one tone each in a minor key (this is called "structure 2"), or if there are multiple sets of notes that rise by one tone each in a minor key in addition to the above structure 2 (this is called "structure 3"), or if it descends by one tone each within a measure after the above structure 3, or if there is a minor key and an octave structure with the same note rising by a third interval ⇒ set the value of the emotion parameter "angry, defiant" to "2".

[0079] (C3) If there is an unstable chord immediately preceding a major third chord, or if there are multiple occurrences of the above composition 2 and a minor key with a set of notes rising one tone at a time (i.e., "composition 3"), then set the emotion parameter "angry, defiant" to "3". An example of an "unstable chord immediately preceding a major third chord" is a chord transition such as E / G# → A5 / E.

[0080] (C4) If none of the above conditions C1 through C3 are met, then the value of the emotion parameter "angry, defiant" is set to "0".

[0081] (D) Emotional parameter: "Passionate" The analysis processing unit 22 determines the value of the emotion parameter "passionate" when the following conditions (D1) to (D4) are met. The value of the emotion parameter "passionate" determined by the analysis processing unit 22 is an integer between 0 and 3, inclusive.

[0082] (D1) When the key is major and the pitch rises by three or more octaves within two measures, set the emotional parameter "passionate" to "1".

[0083] (D2) When the key is major and the pitch rises by three or more octaves within one measure, set the emotional parameter "passionate" to "2".

[0084] (D3) When a fanfare is played, that is, when the same fundamental note / highest note continues for three or more notes, set the emotional parameter "passionate" to "3".

[0085] (D4) If none of the above conditions D1 through D3 are met ⇒ Set the value of the emotion parameter "passionate" to "0".

[0086] [Example of generating a visualization representation] Next, an example of how the visualization generation unit 31 generates visualizations will be described. As already explained, the visualization generation unit 31 generates visualizations based on the values ​​of either or both of the music theory parameters and the emotion parameters. Below, as an example, visualizations based on four types of emotion parameters will be described: (A) "magnificent, ornate," (B) "strange, eerie," (C) "angry, challenging," and (D) "passionate."

[0087] Furthermore, drawing various objects on a display device such as a computer (for example, a liquid crystal display) and moving those objects over time is feasible using existing technologies. Here, an object may be any shape such as a circle, triangle, or square, or it may be a more complex shape, or it may represent a pattern, etc. In addition, the outer perimeter and interior of each object may each have a predetermined color.

[0088] Emotional parameter: (A) "Splendid, ornate" Figure 3 is a schematic diagram showing an example of a visualization generated by the visualization generation unit 31 in response to the emotional parameter "magnificent, decorated". As mentioned above, the emotional parameter "magnificent, decorated" can take integer values ​​from 0 to 5, for example. In other words, the emotional parameter "magnificent, decorated" can take six values. The figure shows a set of objects generated by the visualization generation unit 31 and displayed on the screen. As shown in the figure, the visualization generation unit 31 generates a visualization including objects 701 and 702 based on the value of the emotional parameter "magnificent, decorated".

[0089] Object 701 is the main object. Object 701 itself, being the main object, may be generated based on other music theory parameters or emotion parameters. Object 702 is an object generated according to the value of the emotion parameter "magnificent, decorated". When the value of the emotion parameter "magnificent, decorated" is any value from 1 to 5, the visualization representation generation unit 31 generates a number of small circles (objects 702) equal to that value. For example, if the value of the emotion parameter "magnificent, decorated" is 3, the visualization representation generation unit 31 will manifest 3 small circles (objects 702) as a visualization representation. Each small circle (object 702) manifests in the vicinity of the main object (object 701) (for example, above it), and then, as shown by the arrow lines in Figure 3, it falls down and then moves to the upper right. The size (diameter) of the small circles (objects 702) may be, for example, 4% or more and 5% or less of the size (diameter) of the main object (object 701). Each small circle (object 702) moves as described above after manifestation and disappears between 0.2 seconds and 0.3 seconds after manifestation.

[0090] Furthermore, when the value of the emotion parameter "magnificent, decorated" is 0, the visualization expression generation unit 31 does not generate a small circle (object 702) to represent the emotion "magnificent, decorated".

[0091] This type of visual representation (visual design) corresponding to the emotional parameter "magnificent, ornate" focuses on how musical splendor is expressed through the ornamentation of musical notes, and expresses it through decorative depictions.

[0092] Note that the visualization shown with reference to Figure 3 is just one example, and the visualization generation unit 31 may generate a different visualization based on the emotional parameter "magnificent, decorated".

[0093] Emotional parameter: (B) "Strange, eerie" Figure 4 is a schematic diagram showing an example of a visualization generated by the visualization generation unit 31 in response to the emotional parameter "strange, eerie". As mentioned above, the emotional parameter "strange, eerie" can take integer values ​​from 0 to 3, for example. In other words, the emotional parameter "strange, eerie" can take four values. The figure shows a set of objects generated by the visualization generation unit 31 and displayed on the screen. As shown in the figure, the visualization generation unit 31 generates a visualization including objects 711 and 712 based on the value of the emotional parameter "magnificent, decorated".

[0094] Figure 4 shows the change in the pattern of displayed objects. Specifically, the visualization generated in response to the emotion parameter "strange, eerie" initially includes objects 711A and 712 on the left side of Figure 4. In subsequent stages, the generated visualization includes object 711B on the left side of Figure 4.

[0095] Objects 711A and 711B are the main objects. Object 711A (the first stage), which is the main object, may be generated based on other music theory parameters or emotion parameters. Object 712 is an object generated according to the value of the emotion parameter "strange, eerie".

[0096] In the visualization shown in Figure 4, the shape of the main object 711A (the larger circle) collapses, and it crumbles into granular (or sandy) pieces that fall. These falling granular (or sandy) pieces are object 712. Multiple objects 712 are generated, and these objects 712 move to the right while falling downwards. As a result, the visualization changes to the right side of Figure 4. In other words, object 711B is the result of a part (the lower part) of object 711 collapsing.

[0097] For example, the visualization generation unit 31 disintegrates object 711A as follows, depending on the value of the emotion parameter "strange, eerie". When the value of the emotion parameter "strange, eerie" is 1, the visualization generation unit 31 disintegrates the lower 1 / 5 (20%) portion of the original object 711A. When the value of the emotion parameter "strange, eerie" is 2, the visualization generation unit 31 disintegrates the lower 2 / 5 (40%) portion of the original object 711A. When the value of the emotion parameter "strange, eerie" is 3, the visualization generation unit 31 disintegrates the lower 3 / 5 (60%) portion of the original object 711A. The result of disintegrating object 711A at each of these ratios is object 711B.

[0098] Furthermore, when the value of the emotional parameter "strange, eerie" is 0, the visualization generation unit 31 does not cause object 711A to collapse.

[0099] This visualization (visual design) that corresponds to the emotional parameter "strange, eerie" is an expression using an eerie motif, where the circle representing the notebook has an unpredictable collapse.

[0100] Emotional parameter: (C) "Angry, defiant" Figures 5 and 6 are schematic diagrams showing examples of visualizations generated by the visualization generation unit 31 in response to the emotion parameter "angry, defiant". As mentioned above, the emotion parameter "angry, defiant" can take integer values ​​from 0 to 3, for example. In other words, the emotion parameter "angry, defiant" can take four values. Figures 5 and 6 each show an object generated by the visualization generation unit 31 and displayed on the screen. As shown in the figures, the visualization generation unit 31 generates a visualization that changes the brightness of a certain area of ​​object 721A based on the value of the emotion parameter "angry, defiant".

[0101] Object 721A is the main object. Object 721A (the initial stage), being the main object, may be generated based on other music theory parameters or emotion parameters.

[0102] Figure 5 shows the change in brightness of the displayed object. Specifically, the visualization generated in response to the emotion parameter "angry, defiant" initially includes object 721A on the left side of Figure 5. The brightness of object 721A changes in response to the emotion parameter "angry, defiant," resulting in object 721B on the right. In the example shown in Figure 5, a brightness gradient occurs in object 721B. That is, the brightness of object 721B decreases near the center. In other words, the visualization generation unit 31 generates a visualization that changes from the state of object 721A to the state of object 721B according to the value of the emotion parameter "angry, defiant."

[0103] Figure 6 shows another aspect of the change in brightness in the displayed object. Specifically, the visualization generated in response to the emotion parameter "angry, defiant" initially includes object 721A on the left side of Figure 6. The brightness of object 721A changes in response to the emotion parameter "angry, defiant," resulting in object 721C on the right. In the example shown in Figure 6, a brightness gradient occurs in object 721C. That is, in object 721B, the brightness decreases not only near the center but also in the periphery. In other words, the visualization generation unit 31 generates a visualization that changes from the state of object 721A to the state of object 721C according to the value of the emotion parameter "angry, defiant."

[0104] Furthermore, when comparing object 721B shown in Figure 5 with object 721C shown in Figure 6, the brightness of object 721C is lower even at the periphery of the object (circle).

[0105] For example, the visualization generation unit 31 changes the degree of brightness change according to the value of the emotion parameter "angry, defiant". For example, when the value of the emotion parameter "angry, defiant" is 1, the brightness is reduced only in the area near the center of the object (circle), creating a gradient with the other areas. Also, for example, when the value of the emotion parameter "angry, defiant" is 3, the brightness is reduced in the area including not only the area near the center of the object (circle) but also the surrounding area. When the value of the emotion parameter "angry, defiant" is 2, it is set to an intermediate value between the cases of 1 and 3.

[0106] Furthermore, when the value of the emotion parameter "angry, defiant" is 0, the visualization generation unit 31 does not change the brightness of object 721A.

[0107] This visualization (visual design) corresponding to the emotional parameter "angry, defiant" expresses the unsettling atmosphere immediately preceding a chord resolution with a low-luminosity gradient.

[0108] Emotional parameter: (D) "Passionate" Figure 7 is a schematic diagram showing an example of a visualization generated by the visualization generation unit 31 in response to the emotion parameter "passionate". As mentioned above, the emotion parameter "passionate" can take integer values ​​from 0 to 3, for example. In other words, the emotion parameter "passionate" can take four values. Figure 7 shows a set of objects generated by the visualization generation unit 31 and displayed on the screen. As shown in the figure, the visualization generation unit 31 generates a visualization including objects 731 and 732 based on the value of the emotion parameter "passionate".

[0109] Figure 7 shows the movement of the displayed objects. In Figure 7, object 731 is the main object. Object 731 itself, being the main object, may be generated based on other music theory parameters or emotion parameters. Object 732 is an object generated around object 731 according to the value of the emotion parameter "passionate".

[0110] In the visualization shown in Figure 7, depending on the value of the emotion parameter "passionate," objects 732 with different patterns appear around the main object 731 (circle), and multiple objects 732 rotate around the main object 731. The direction of rotation may be, for example, the direction of the arrow line shown in Figure 7, or the opposite direction. In the example shown, object 732 is an object with a pattern based on Art Nouveau depictions that appeared during the Romantic era. However, object 732 may be made up of other patterns.

[0111] When the value of the emotion parameter "passionate" is 1, 2, or 3, the visualization expression generation unit 31 ensures that the larger the value of the emotion parameter "passionate," the more objects 732 are expressed. In other words, for example, when the value of the emotion parameter "passionate" is 2, more objects 732 are generated than when the value of the emotion parameter "passionate" is 1.

[0112] Furthermore, when the value of the emotional parameter "passionate" is 0, the visualization expression generation unit 31 does not generate object 732.

[0113] This type of visual representation (visual design) corresponding to the emotional parameter "passionate" draws on Symbolism and expresses the passionate nature that is prominent in Romanticism through clear motifs.

[0114] Figures 8, 9, 10, and 11 are schematic diagrams showing examples of the results obtained when the synthesis unit 33 of the visualization representation generation device 3 synthesizes the visualization representation generated by the visualization representation generation unit 31 with the video read from the content storage unit 32.

[0115] Figure 8 shows an example of the result of combining a visual representation generated based on the emotional parameter "magnificent, ornate" (A) with the video read from the content storage unit 32. The video shown in Figure 8 includes the visualization representation described with reference to Figure 3.

[0116] Figure 9 shows an example of the result of combining a visual representation generated based on the emotional parameter "strange, eerie" (B) with video footage read from the content storage unit 32. The video footage shown in Figure 9 includes the visualization representation described with reference to Figure 4.

[0117] Figure 10 shows an example of the result of combining a visual representation generated based on the emotion parameter "angry, defiant" (C) with video footage read from the content storage unit 32. The video footage shown in Figure 10 includes the visualization representations explained with reference to Figures 5 and 6.

[0118] Figure 11 shows an example of the result of combining a visual representation generated based on the emotional parameter "passionate" (D) with video footage read from the content storage unit 32. The video footage shown in Figure 11 includes the visualization representation described with reference to Figure 7.

[0119] Figure 12 is a block diagram showing an example of the internal configuration of the music analysis device 2 and the visualization representation generation device 3 that constitute System 1. At least some of the functions of each device can be realized using a computer. As shown in the figure, the computer consists of a central processing unit 901, RAM 902, input / output ports 903, input / output devices 904 and 905, etc., and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions contained in programs read from RAM 902, etc. The central processing unit 901 writes data to RAM 902, reads data from RAM 902, and performs arithmetic and logical operations according to each instruction. RAM 902 stores data and programs. Each element contained in RAM 902 has an address and can be accessed using that address. RAM stands for "Random Access Memory". Input / output ports 903 are ports for the central processing unit 901 to exchange data with external input / output devices, etc. Input / output devices 904 and 905 exchange data with the central processing unit 901 via input / output port 903. Bus 906 is a common communication channel used within the computer. For example, the central processing unit 901 reads and writes data to RAM 902 via bus 906. Also, for example, the central processing unit 901 accesses input / output port 903 via bus 906.

[0120] Furthermore, at least some of the functions of the music analysis device 2 and the visualization expression generation device 3 in the above-described embodiment can be realized using a computer and a program. In that case, the program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be loaded into a computer system and executed. Here, "computer system" includes hardware such as the OS and peripheral devices. Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, DVD-ROMs, USB memory, and storage devices such as hard disks built into a computer system. In other words, "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Moreover, "computer-readable recording medium" may also include those that temporarily and dynamically hold programs, such as communication lines when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and those that hold programs for a certain period of time, such as volatile memory inside a computer system that acts as a server or client in that case. Furthermore, the above-mentioned program may be for realizing some of the functions described above, and may also be able to realize the above-mentioned functions in combination with a program already recorded in the computer system.

[0121] [Differentiation] In the embodiment described above, the analysis processing unit 22 of the music analysis device 2 determined the value of a first emotion parameter representing the degree of emotion of "magnificent, ornate," the value of a second emotion parameter representing the degree of emotion of "strange, eerie," the value of a third emotion parameter representing the degree of emotion of "angry, defiant," and the value of a fourth emotion parameter representing the degree of emotion of "passionate," all based on the music data representing the music. As a modification, the analysis processing unit 22 may determine at least one of the values ​​of the first emotion parameter, the second emotion parameter, the third emotion parameter, and the fourth emotion parameter.

[0122] [Note] As described above, this embodiment is a method for generating input data for visualizing music. This embodiment relates to media technology, content expression methods, music analysis, and video generation. This embodiment is part of a comprehensive approach to music data generation based on music analysis, as well as music visualization design and system construction. This embodiment contributes to a method for visualizing music in an essential way based on music theory. This embodiment contributes to a method for expressing "the content and impression of music" in accordance with the progression of the music, with the aim of improving the entertainment value of music, music education, and music education and musical understanding for the hearing impaired.

[0123] While music visualizers have existed in the past that visualized the acoustic physical quantities of music (sound pressure and tempo) in sync with the music, these did not visualize the content or impression of the music. They merely captured and visualized the physical quantities of sound. Although the physical quantities of sound are part of music theory, sound pressure and tempo are extremely insufficient for people to perceive music; the chronological note structure and music theory information of each piece are essential for feeling and understanding music. In contrast to conventional music visualizer technologies, this embodiment aims to reflect the chronological note structure and music theory information of each song in order to feel and understand the music.

[0124] This embodiment enables the simultaneous or progressive quantification of the impression of music based on its note-like structure, and the use of this quantified data as input for visualization generation. In other words, this embodiment is part of an effort to analyze or learn the causal relationship between the note-like structure of music that changes over time and the impression that humans receive from it, and to use the quantified data as input for music visualization generation.

[0125] This embodiment allows the content and impression of music, based on the simultaneous and chronological note structure that forms the core of music, to be used as input data for music visualization. In other words, it contributes to truly visualizing musical information and the impressions that humans feel from music.

[0126] While embodiments and modifications of this invention have been described in detail above with reference to the drawings, the specific configuration is not limited to these embodiments and includes designs and the like that do not depart from the spirit of this invention. [Industrial applicability]

[0127] This invention can be used, for example, in music analysis and content creation. However, the scope of use of this invention is not limited to those exemplified herein. [Explanation of symbols]

[0128] 1 System 2. Music analysis device 3 Visual representation generation device 21 Music Data Supply Department 22 Analysis Processing Unit 22 23 Parameter output unit 23 31 Visualization expression generation unit 32 Content storage unit 33 Synthesis section 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. An analysis processing unit determines, based on music data representing music, at least one of the following values: a first emotion parameter representing the degree of emotion of "magnificent, ornate"; a second emotion parameter representing the degree of emotion of "strange, eerie"; a third emotion parameter representing the degree of emotion of "angry, defiant"; and a fourth emotion parameter representing the degree of emotion of "passionate". A music analysis device equipped with the following features.

2. The aforementioned music data contains information on the pitch, length, and timing of each sound that constitutes the music. The music analysis apparatus according to claim 1.

3. The aforementioned analysis processing unit When both slurs and staccatos are indicated in the aforementioned musical data, or, When a trill is indicated in the aforementioned music data, or, When the piece is in a major key, and all the notes within a measure or beat are shorter than sixteenth notes, or When a piece is in a major key, and more than two-thirds of the notes within a measure or beat are played as sixteenth notes, The value of the first emotion parameter is determined to represent the emotion of "magnificent, ornate." The music analysis apparatus according to claim 1.

4. The aforementioned analysis processing unit When a trill is indicated in the aforementioned music data, the value of the first emotion parameter is determined according to the number of notes over which the trill is indicated. The music analysis apparatus according to claim 3.

5. The aforementioned analysis processing unit In the aforementioned music data, when there is a modulation from major to minor, and that modulation is one in which the fundamental note is different, or In the aforementioned music data, when the accompaniment section contains notes in a minor key and in an octave configuration for two measures or more consecutively, In the aforementioned musical data, when a minor key and two-bar phrase are repeated an octave lower, or In the aforementioned music data, when the same note in a minor key and an octave configuration is repeated for two or more measures, In the aforementioned music data, the unit is a minor key and descending trinotes, or In the aforementioned musical data, when there is a repetition of a unit that is in a minor key and ascends by a triplet, or In the aforementioned music data, when there is one or more instances of the fundamental tone plus an unstable chord, and the highest note of the stable chord is raised by a semitone, The value of the second emotion parameter is determined to represent the emotion of "strange, eerie". The music analysis device according to claim 1.

6. The aforementioned analysis processing unit In the aforementioned music data, when the accompaniment consists only of a single note rising every two measures, or In the aforementioned music data, when the accompaniment section has a structure where the pitch rises by one note every two measures, and then there is another structure where the pitch rises by one note every two measures, or In the aforementioned music data, when the accompaniment part contains multiple sets of notes that ascend one tone at a time in a minor key, or, In the aforementioned music data, when the accompaniment section contains multiple sets of notes that ascend one tone at a time in a minor key, and then descend one tone at a time within the measure, or In the aforementioned music data, when the accompaniment part is in a minor key and has an octave structure with the same note raised by a third, or In the aforementioned music data, when the accompaniment section has a structure where the pitch rises by one tone every two measures, and there is another structure where the pitch rises by one tone every two measures, and there are multiple sets of notes that rise by one tone each in a minor key, and there are multiple sets of notes that rise by one tone each in a minor key, The value of the third emotion parameter is determined to represent the emotion of "angry, defiant". The music analysis device according to claim 1.

7. The aforementioned analysis processing unit In the aforementioned musical data, when the key is major and the pitch rises by three octaves or more within two measures, or In the aforementioned musical data, when the key is major and the pitch rises by three octaves or more within one measure, or, In the aforementioned music data, when the same fundamental tone / highest note continues for three or more notes, The value of the fourth emotion parameter is determined to be a value that indicates the presence of the emotion "passionate". The music analysis device according to claim 1.

8. An analysis processing unit determines, based on music data representing music, at least one of the following values: a first emotion parameter representing the degree of emotion of "magnificent, ornate"; a second emotion parameter representing the degree of emotion of "strange, eerie"; a third emotion parameter representing the degree of emotion of "angry, defiant"; and a fourth emotion parameter representing the degree of emotion of "passionate". A program to make a computer function as a music analysis device equipped with [specific features / functions].

Images