A music color visualization method based on music semantics

Abstract

The present invention relates to a music color visualization method based on music semantics. The method comprises: obtaining content information containing music semantics; initializing a color palette containing multiple colors used to represent music colors; encoding the content information and the color information in the color palette separately and fusing them to generate a fused code; generating a color palette corresponding to the music semantics using a pre-trained color palette generation model based on the fused code; and visualizing the music based on the colors in the color palette. Compared with existing technologies, the present invention can enrich the visual color of music and effectively display the music semantics through color.

Description

Technical Field

[0001] The present invention relates to the technical field of music color visualization, and in particular to a music color visualization method based on music semantics. Background Art

[0002] Currently, there are many mature music visualization tools available on the market, and most music players have built-in visualization features. These tools bring songs to life by animating shapes and colors within images, enriching the visual experience of songs. The changing shapes and colors can reflect audio characteristics such as loudness, frequency, and rhythm. Numerous studies in color design theory have explored the connection between color and semantics. For example, happy colors are typically bright and warm, while sad colors are typically dark and soft. Music also has inherent semantic meaning, which can be reflected in related textual descriptions (such as lyrics and song descriptions), musical genres (such as rock, punk, and electronic music), and visual design (such as album covers and live performance posters). Combining the semantics of music and color for music visualization is a natural idea. However, the colors used in most existing music visualizations are typically randomly generated or selected from preset palettes, without considering the relevant musical semantics and visual design. Summary of the Invention

[0003] The purpose of the present invention is to provide a music color visualization method based on music semantics in order to overcome the defects of the above-mentioned prior art.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] A music color visualization method based on music semantics, the method comprising:

[0006] Obtaining content information containing music semantics;

[0007] Initialize a color palette, which includes multiple colors used to represent musical colors;

[0008] The content information and the color information in the color palette are encoded separately and fused to generate a fused code. Based on the fused code, a pre-trained color palette generation model is used to generate a color palette corresponding to the music semantics.

[0009] Visualize music based on the colors in the palette.

[0010] Preferably, the content information includes category information representing music genres, image information representing music semantics in image form, and text information representing music speech in text form.

[0011] Preferably, the category information, image information and text information are encoded respectively by a category encoder, an image encoder and a text encoder.

[0012] Preferably, the category encoder is a one-hot encoder, the image encoder is an encoder based on the VGG16 model, and the text encoder is an encoder based on the BERT model.

[0013] Preferably, the color information in the color palette is encoded by combining the RGB values ​​of each color in the color palette into a vector.

[0014] Preferably, the content information and the color information in the color palette are encoded separately and then fused through a multi-layer perceptron to generate a fused code.

[0015] Preferably, when generating a color palette corresponding to musical semantics, each color in the color palette is generated one by one in a recursive form, specifically: one color is generated at a time, the color palette and the corresponding fusion code are updated after each color is generated, and the next color is generated based on the new fusion code until the generation of all colors in the color palette is completed.

[0016] Preferably, the color palette generation model is a conditional adversarial generative network, including a generator and an adversary. When generating a color palette, the fusion code and noise are input into the generator, and the color code corresponding to the generated color is obtained through the fully connected layer. The color code and the fusion code are input into the adversary to determine whether the output meets expectations.

[0017] Preferably, the method of visualizing the music based on the colors in the color palette is: using a visual graphic to display the music playing progress, and randomly using the colors in the color palette to color the visual graphic.

[0018] Preferably, the visualization of music also includes a synchronous visualization display of the song frequency, specifically: extracting the song frequency, changing the size of the visualization graphic to perform a synchronous visualization display of the song frequency.

[0019] Compared with the prior art, the present invention has the following advantages:

[0020] (1) Based on multimodal fusion coding and generative adversarial networks, this paper implements a music visualization method based on music semantic information. By integrating the semantics and genre information in song lyrics with the graphic features in visual design, the method can enrich the visualization of music and present music works more vividly.

[0021] (2) The present invention uses a deep learning model to automatically associate the semantic information and visualization effects of music, eliminating the manual matching process and improving the efficiency of the entire process. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a framework diagram of the color palette generation model training process of the present invention;

[0023] Figure 2 A framework diagram of a color palette generated by a color palette generation model according to the present invention;

[0024] Figure 3 This is a framework diagram of the music color visualization method based on music semantics of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Note that the following embodiments are merely illustrative in nature and the present invention is not intended to limit its applicable objects or uses, and the present invention is not limited to the following embodiments.

[0026] Example

[0027] This embodiment provides a music color visualization method based on music semantics, the method comprising:

[0028] First, content information containing musical semantics is obtained. The content information includes category information representing the musical genre, image information representing the musical semantics in image form, and text information representing the musical voice in text form. In this embodiment, the category information uses the musical genre, the image information uses the song cover, and the text information uses the lyrics.

[0029] Then, the color palette is initialized. The color palette includes a plurality of colors for representing the color of music. In this embodiment, five colors are set in the color palette. When initializing the color palette, the RGB values ​​of all colors in the color palette are set to 0.

[0030] Secondly, the content information and the color information in the color palette are encoded separately and fused to generate a fused code. Based on the fused code, a pre-trained color palette generation model is used to generate a color palette corresponding to the music semantics.

[0031] Specifically, the category information, image information, and text information are encoded respectively through the category encoder, image encoder, and text encoder. The category encoder is a one-hot encoder, the image encoder is an encoder based on the VGG16 model, and the text encoder is an encoder based on the BERT model. The color information in the color palette is encoded by combining the RGB values ​​of each color in the palette into a vector. After the content information and the color information in the palette are encoded separately, they are fused through a multi-layer perceptron to generate a fusion code. When generating a palette corresponding to the musical semantics, each color in the palette is generated one by one in a recursive form, specifically: one color is generated at a time, and after each color is generated, the palette and the corresponding fusion code are updated, and the next color is generated based on the new fusion code, until all colors in the palette are generated.

[0032] The color palette generation model is a conditional adversarial generative network, consisting of a generator and an adversary. When generating a color palette, the fusion code and noise are input into the generator. The color code corresponding to the generated color is obtained through the fully connected layer. The color code and the fusion code are input into the adversary to determine whether the output meets expectations.

[0033] Finally, the music is visualized based on the colors in the palette. Specifically, a visual graphic displays the music playback progress, randomly coloring the graphic with the colors in the palette. The music visualization also includes a synchronized visualization of the song frequencies. Specifically, the song frequencies are extracted and the visual graphic is resized to synchronize the display.

[0034] The specific process of visualizing music colors in this embodiment is described in detail below.

[0035] 1. Training of the Color Plate Generation Model

[0036] Figure 1 The architecture of the model based on the Conditional Generative Adversarial Network (CGAN) in this project is shown. During training, a multimodal music semantic color dataset is given as training samples. The content information containing music semantics (song cover, text description, and genre) and the previous color in the palette are input as multimodal content information to the generator, causing the generator to generate the next color in the palette. The discriminator determines whether the generated color conforms to the input multimodal content information. To train the model, a unique music color dataset was constructed to train the model, which contains 948 graphic design images from 189 Chinese bands, as well as corresponding song text descriptions, genre categories, and 5-color palette data.

[0037] 2. Generate color palette using color palette generation model

[0038] The specific framework for generating color palettes using the color palette generation model is as follows: Figure 2 As shown, the song cover, lyrics, and genre are used as input to generate a series of color palettes corresponding to the lyrics. The frequency information of the song is used as input for music visualization to visualize the song. In this embodiment, a series of color palettes is output for each line of lyrics, allowing for a visual display of each line of lyrics.

[0039] The music-related visual design process can be summarized as four steps: category selection, text theme determination, color design, and image design. The three aspects of information, category, text, and image, interact with color, interacting, coordinating, and constraining each other, ultimately resulting in a captivating visual design effect. Therefore, the influencing factors of the music-related visual design process are abstracted into input information in three dimensions: category, text, and image. Existing encoders are used to encode different types of datasets. The encoded vectors are then passed through a multi-layer perceptron (MLP) to obtain a fused encoding. During model training, this fused encoding incorporates the visual, textual, and genre semantics of the song, which is then fed into the subsequent CGAN as the content vector.

[0040] During model implementation, the lyrics are divided into sentences, with each sentence serving as a separate input text vector. This means each song will consist of anywhere from a few to dozens of text vectors. Each text vector, along with the song cover and song category, is then encoded using an MLP multimodal fusion algorithm and fed into the CGAN.

[0041] like Figure 2 As shown in the figure, each line of a song's lyrics corresponds to a text vector, which, along with the song's cover and category, forms a fused encoding. In other words, a song is converted into a fused encoding equal to the number of lines in the lyrics and fed into the CGAN. Correspondingly, the CGAN outputs the same number of color palette results, each reflecting the musical meaning of the corresponding line of lyrics.

[0042] In addition, the relationship between the colors in each palette is crucial for color design in music visualization. Therefore, this model generates palettes by recursively predicting palette colors. Another benefit of this approach is that each of the five-color palettes in the original data can be expanded into five input data points, effectively performing data augmentation. This reduces the original 948 data sets to 4,740 data sets, significantly increasing the data volume and improving model training. CGAN was used to build the palette generation network. The fused encoding containing the musical semantic information was given as conditional input to the generator and discriminator in CGAN, which generated color results one by one and determined whether the results met the fused encoding requirements.

[0043] This process involves two main steps, which can be summarized as follows:

[0044] 1. Multimodal Data Encoding

[0045] This part uses a combination of pre-trained models and multi-layer perceptrons (MLP) to complete the feature extraction and fusion of multimodal content.

[0046] Multimodal content includes text, images, categories, and color palettes. The first three are user input, while the color palettes are generated by the model. Text is encoded into fixed-length vectors using the pre-trained BERT model; images are encoded into fixed-length vectors using the pre-trained VGG16 model; categories are encoded into fixed-length vectors using one-hot encoding; and color palettes are directly combined into a 15-dimensional vector by combining the five RGB values.

[0047] After obtaining the encoding result, the four fixed-length vectors are concatenated into one vector, and then the combined vector is input into a multi-layer perceptron (MLP) to obtain the fusion code.

[0048] 2. Generate a corresponding color palette for each line of lyrics

[0049] The song lyrics are split into sentences, and each sentence forms a text vector. The following recursive color palette generation process is performed on these text vectors to finally obtain a corresponding series of color palette results.

[0050] The recursive color palette generation process is based on a conditional generative adversarial network, which uses a recursive approach to generate the colors in the palette.

[0051] Among them, the condition of the conditional generative adversarial network is the fusion code of the previous step, and the model can generate a color (that is, a set of R, G, B values) based on this fusion code.

[0052] The entire recursive generation process is as follows:

[0053] 1) Initialize all 5 colors in the color palette to 0;

[0054] 2) Generate fusion code based on the current color palette and user input;

[0055] 3) Encode the fusion as a condition, input it into CGAN, and generate a color;

[0056] 4) Update the color palette according to the color;

[0057] 5) Repeat steps 2-4 until all five color palettes are generated.

[0058] 3. Visualization Process

[0059] The visualization process is performed by Figure 3 To demonstrate this, the song's image, lyrics, and genre information are processed through the color palette generation model described above to generate a color palette corresponding to the lyrics. Simultaneously, the song's music file is processed through the sound data processing module, the Minim library, to obtain the song's frequency data. The color palette data and song frequency data are then used to visualize the song in the graphics programming software Processing, resulting in a dynamic visualization.

[0060] The frequency information visualization method specifically involves extracting song frequencies using the frequency module in the audio visualization component library minim. The first 50 frequencies are used as the x-axis, and the size of the graphic represents the amplitude (for example, if a circle is selected as the visualization graphic, the diameter can be used to represent the amplitude of the song frequency). The color visualization is divided into two parts: the lyrics part and the instrumental part. The graphics for the lyrics part are colored according to the color palette data generated in the previous step, and their distribution is random. The graphics for the instrumental part (no lyrics) are manually defined by the designer based on the meaning of the song. According to the song's lyric timeline, the color and graphic changes are synchronized with the corresponding lyrics, producing a dynamic visual effect as the music plays, forming a dynamic music video.

[0061] Based on the above-mentioned music color visualization method based on music semantics, a corresponding music color visualization system based on music semantics was designed, including the user side and the model side. Its specific business process is as follows:

[0062] User side: User uploads song cover -> uploads song lyrics with timeline -> selects music genre -> obtains dynamic visualization video results that follow the changes in music semantics and frequency

[0063] On the model side: The user uploads an image, a string with a timeline, and selects a song genre. The string with the timeline is processed to obtain an array of time points and lyrics. The multimodal content encoding model generates a fused code corresponding to each line of lyrics. The color palette generation model recursively generates a color palette corresponding to each line of lyrics based on the fused code. Visual programming is performed using the song frequency amplitude and color palette results. The output is a dynamic visualization video that changes with the musical semantics and frequency.

[0064] The above embodiments are merely examples and do not limit the scope of the present invention. These embodiments can be implemented in various other ways, and various omissions, replacements, and changes can be made without departing from the technical concept of the present invention.

Claims

1. A music color visualization method based on music semantics, characterized in that: The method includes: Obtaining content information containing music semantics; Initialize a color palette, which includes multiple colors used to represent musical colors; The content information and the color information in the color palette are encoded separately and fused to generate a fused code. Based on the fused code, a pre-trained color palette generation model is used to generate a color palette corresponding to the music semantics. Visualize music based on colors in a palette; When generating a color palette corresponding to musical semantics, a recursive method is used to generate each color in the palette one by one. Specifically, one color is generated at a time. After each color is generated, the palette and the corresponding fusion code are updated. The next color is generated based on the new fusion code until all colors in the palette are generated. The color palette is generated by recursively predicting the color of the palette. Each 5-color palette in the original data can be expanded into 5 input data, which is equivalent to data enhancement. The original 948 data groups are turned into 4740 data groups, which greatly increases the data volume and improves the model training effect. CGAN is used to build a color palette generation network, and the fusion code containing music semantic information is given as conditional input to the generator and discriminator in CGAN respectively, generating color results one by one and judging whether the results conform to the fusion code.

2. The music color visualization method based on music semantics according to claim 1, characterized in that: The content information includes category information representing music genres, image information representing music semantics in image form, and text information representing music speech in text form.

3. The music color visualization method based on music semantics according to claim 2, characterized in that: The category information, image information and text information are encoded respectively through a category encoder, an image encoder and a text encoder.

4. The music color visualization method based on music semantics according to claim 3, characterized in that: The category encoder is a one-hot encoder, the image encoder is an encoder based on the VGG16 model, and the text encoder is an encoder based on the BERT model.

5. The music color visualization method based on music semantics according to claim 1, characterized in that: The color information in the color palette is encoded by combining the RGB values ​​of each color in the color palette into a vector.

6. The music color visualization method based on music semantics according to claim 1, characterized in that: The content information and the color information in the palette are encoded separately and then fused through a multi-layer perceptron to generate a fused code.

7. The music color visualization method based on music semantics according to claim 1, characterized in that: The color palette generation model is a conditional adversarial generative network, including a generator and an adversary. When generating a color palette, the fusion code and noise are input into the generator, and the color code corresponding to the generated color is obtained through the fully connected layer. The color code and the fusion code are input into the adversary to determine whether the output meets expectations.

8. The music color visualization method based on music semantics according to claim 1, characterized in that: The method of visualizing music based on the colors in the color palette is: using a visual graphic to display the progress of music playback, and randomly using the colors in the color palette to color the visual graphic.

9. The music color visualization method based on music semantics according to claim 8, characterized in that: The visualization of music also includes the synchronous visualization display of the song frequency, specifically: extracting the song frequency, changing the size of the visualization graphic to perform the synchronous visualization display of the song frequency.

Images