Method and apparatus for classifying music emotions by using spectrogram

Abstract

A method by which a processor uses a spectrogram to classify music emotions is disclosed. The music emotion classification method comprises the steps of: converting an audio signal into a Mel spectrogram; extracting an arbitrary first region from the Mel spectrogram; and predicting the audio signal to be any one from among a plurality of emotions by applying the arbitrary first region extracted from the Mel spectrogram to a first neural network.

Description

Method and device for classifying music emotions using spectrograms

[0001] The present invention relates to a method and device for classifying music emotions, and more particularly, to a method and device for classifying music into emotions using a spectrogram.

[0002] There is a need to classify music, such as songs, into one of several emotions to recommend them to users or use them as background music in videos. Existing methods for classifying music by emotion have been proposed, but they are inefficient because they utilize the entire music.

[0003] The technical task to be achieved by the present invention is to provide a method and device for classifying music emotions using a spectrogram for efficiently classifying music into emotions.

[0004] A method for classifying music emotions using a spectrogram, performed by a processor according to an embodiment of the present invention, includes the steps of converting an audio signal into a Mel spectrogram, extracting an arbitrary first region from the Mel spectrogram, and applying the arbitrary first region extracted from the Mel spectrogram to a first neural network to predict the audio signal as one of a plurality of emotions.

[0005] The step of extracting an arbitrary first region from the above Mel spectrogram includes a step of extracting the arbitrary first region by applying the Mel spectrogram to a second neural network.

[0006] The above first neural network or the above second neural network is a CNN (Convolutional Neural network) algorithm.

[0007] The method for classifying music emotions using the above spectrogram further includes the step of extracting an arbitrary second region different from the arbitrary first region from the Mel spectrogram when any one of the predicted emotions among the plurality of emotions is below a threshold value, and the step of applying the arbitrary first region and the arbitrary second region to the first neural network to re-predict the audio signal as any one of the plurality of emotions.

[0008] A computing device according to an embodiment of the present invention includes a processor that executes commands for classifying music emotions using a spectrogram, and a memory that stores the commands. The commands are implemented to convert an audio signal into a Mel spectrogram, extract an arbitrary first region from the Mel spectrogram, and apply the arbitrary first region extracted from the Mel spectrogram to a first neural network to predict the audio signal as one of a plurality of emotions.

[0009] The commands for extracting an arbitrary first region from the above Mel spectrogram are implemented to extract the arbitrary first region by applying the Mel spectrogram to a second neural network.

[0010] The above first neural network or the above second neural network is a CNN (Convolutional Neural network) algorithm.

[0011] The above commands are further implemented to extract an arbitrary second region different from the arbitrary first region from the Mel spectrogram when any one of the predicted emotions among the plurality of emotions is below a threshold value, and to apply the arbitrary first region and the arbitrary second region to the first neural network to re-predict the audio signal as any one of the plurality of emotions.

[0012] A method and device for classifying music emotions using a spectrogram according to an embodiment of the present invention converts an audio signal into a Mel spectrogram, and instead of directly applying the converted Mel spectrogram to a neural network to predict the audio signal as one of a plurality of emotions, only a portion of the converted Mel spectrogram is applied to the neural network to predict the audio signal as one of a plurality of emotions, thereby reducing the computational load and time required to classify music emotions.

[0013] In order to more fully understand the drawings cited in the detailed description of the present invention, a detailed description of each drawing is provided.

[0014] FIG. 1 illustrates a block diagram of a computing device according to an embodiment of the present invention.

[0015] FIG. 2 illustrates a conceptual diagram for explaining a music emotion classification method according to an embodiment of the present invention.

[0016] Figure 3 shows various Mel spectrograms according to various emotions.

[0017] FIG. 4 is a conceptual diagram illustrating an operation of extracting an arbitrary region from the Mel spectrogram illustrated in FIG. 2 according to an embodiment of the present invention.

[0018] FIG. 5 shows a flowchart for explaining a music emotion classification method according to an embodiment of the present invention.

[0019] Specific structural or functional descriptions of embodiments according to the concept of the present invention disclosed in this specification are merely illustrative for the purpose of explaining embodiments according to the concept of the present invention, and embodiments according to the concept of the present invention may be implemented in various forms and are not limited to the embodiments described in this specification.

[0020] Embodiments according to the concept of the present invention may have various modifications and take various forms, and thus, embodiments are illustrated in the drawings and described in detail in this specification. However, this is not intended to limit embodiments according to the concept of the present invention to specific disclosed forms, but rather includes all modifications, equivalents, or alternatives falling within the spirit and technical scope of the present invention.

[0021] While terms such as "first" or "second" may be used to describe various components, these components should not be limited by these terms. These terms are intended solely to distinguish one component from another. For example, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component, without departing from the scope of the invention.

[0022] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

[0023] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the present invention. The singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, it should be understood that the terms "comprises" or "has" indicate the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0024] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0025] Hereinafter, the present invention will be described in detail by describing a preferred embodiment of the present invention with reference to the attached drawings.

[0026] FIG. 1 illustrates a block diagram of a computing device according to an embodiment of the present invention.

[0027] Referring to FIG. 1, a computing device (10) refers to an electronic device for executing a music emotion classification method using a spectrogram. The computing device (10) refers to an electronic device such as a smartphone, tablet PC, personal PC, server, desktop, or laptop.

[0028] A computing device (10) includes a processor (11) that executes music emotion classification commands using a spectrogram, and a memory (13) that stores the commands.

[0029] Below, the music emotion classification operations are described as being performed by the processor (11).

[0030] FIG. 2 illustrates a conceptual diagram for explaining a music emotion classification method according to an embodiment of the present invention.

[0031] Referring to FIGS. 1 and 2, the processor (11) can receive an audio signal (20) via a network or a storage medium such as a flash memory. The audio signal (20) refers to a file representing music or a song. For example, the file format of the audio signal (20) can be various, such as WAV or MP3. The audio signal (20) can be expressed as a 2D graph illustrated in FIG. 2. In the 2D graph illustrated in FIG. 2, the X-axis represents time, and the Y-axis represents amplitude.

[0032] The processor (11) converts the audio signal (20) into a Mel spectrogram (Spectrogram; 30). The Mel spectrogram (30) refers to a visual representation of the frequency spectrum of the audio signal (20) over time. Hereinafter, the Mel spectrogram (30) is referred to as a Mel spectrogram or Mel spectrogram. The Mel spectrogram (30) is a 2D representation of the audio signal (20), in which the X-axis represents time and the Y-axis represents frequency. The Y-axis is expressed as a Mel scale. The Mel spectrogram (30) is expressed in color, and the color represents the amplitude or energy of each frequency component over time.

[0033] The conversion of an audio signal (20) into a Mel Spectrogram (30) is based on the Fast Fourier Transform (FFT) conversion. The conversion of an audio signal (20) into a Mel Spectrogram (30) is widely known, and a detailed description thereof is omitted.

[0034] The processor (11) can apply a Mel Spectrogram (30) corresponding to the audio signal (20) to a first neural network (40) to predict the audio signal (20) as one of a plurality of emotions (50) (e.g., Sad). Mel-Frequency Cepstral Coefficients (MFCC) representing spectral characteristics of the audio signal (20) can be derived from the Mel Spectrogram (30). The MFCC is used as features in the first neural network (40). The first neural network (40) may be a CNN (Convolutional Neural network) algorithm. That is, the first neural network (40) is a widely known neural network algorithm.

[0035] In Fig. 2, multiple emotions (50) are displayed as nine categories, including Aggressive, Calm, Cheerful, Depressive, Energetic, Exciting, Relaxing, Sad, and Scary. However, depending on the embodiment, the multiple emotions (50) may be classified in various ways. For example, the multiple emotions (50) may be classified into four categories, including exuberance, anxious, contentment, and depression.

[0036] Figure 3 shows various Mel spectrograms according to various emotions. Figures 3 (a) to 3 (g) show different Mel spectrograms according to various emotions.

[0037] For example, (a) of Fig. 3 may represent a Mel spectrogram that can be classified as Aggressive. (b) of Fig. 3 may represent a Mel spectrogram that can be classified as Calm. (c) of Fig. 3 may represent a Mel spectrogram that can be classified as Cheerful. (d) of Fig. 3 may represent a Mel spectrogram that can be classified as Depressive. (e) of Fig. 3 may represent a Mel spectrogram that can be classified as Energetic. (f) of Fig. 3 may represent a Mel spectrogram that can be classified as Exciting. (g) of Fig. 3 may represent a Mel spectrogram that can be classified as Relaxing.

[0038] That is, when any one of the various Mel spectrograms illustrated in FIG. 3 is applied to the first neural network (40) illustrated in FIG. 2, any one of a plurality of emotions (50) can be predicted.

[0039] Referring to the multiple mel spectrograms illustrated in FIG. 3, it can be seen that the intensity distributions of the color pixels included in the mel spectrograms differ depending on various emotions. For example, it can be seen that the intensity distributions of the color pixels included in the mel spectrogram of FIG. 3 (b) and the intensity distributions of the color pixels included in the mel spectrogram of FIG. 3 (c) differ from each other.

[0040] FIG. 4 is a conceptual diagram illustrating an operation of extracting an arbitrary region from the Mel spectrogram illustrated in FIG. 2 according to an embodiment of the present invention. FIG. 4 (a) is a conceptual diagram illustrating an operation of extracting an arbitrary first region (31) from the Mel spectrogram illustrated in FIG. 2, and FIG. 4 (b) is a conceptual diagram illustrating an operation of extracting a second arbitrary region (33) from the Mel spectrogram illustrated in FIG. 2.

[0041] Referring to (a) of FIGS. 1 to 4, the mel spectrogram (30) refers to a visual representation of the frequency spectrum of an audio signal (20) over time. Applying the entire mel spectrogram (30) to the first neural network (40) to predict the audio signal (20) as one of a plurality of emotions (50) (e.g., Sad) requires a lot of neural network computation load and time. Therefore, a new method for reducing the load and time of the neural network computation is required. That is, rather than applying the entire mel spectrogram (30) to the first neural network (40) to predict the audio signal (20) as one of a plurality of emotions (50), a part of the mel spectrogram (30) is applied to the first neural network (40) to reduce the load and time of the neural network computation.

[0042] A conceptual diagram such as (a) of Fig. 4 is shown to determine a part of the Mel spectrogram (30).

[0043] The processor (11) applies a Mel Spectrogram (30) corresponding to the audio signal (20) to the second neural network (60) to extract an arbitrary first region (31).

[0044] The second neural network (60) may be a CNN (Convolutional Neural Network) algorithm. That is, the second neural network (60) is a widely known neural network algorithm. The second neural network (60) is trained to extract an arbitrary first region (31) by applying a Mel Spectrogram (30) to the second neural network (60). The region in the Mel Spectrogram that best expresses the most distinct emotions is used as training data.

[0045] Any first region (31) refers to a part of a region included in a Mel spectrogram (30), and the arbitrary first region (31) may have a rectangular shape, and the size of the arbitrary first region (31) is smaller than the size of the Mel spectrogram (30). Any first region (31) is a region that can best express a specific emotion among multiple emotions (50) in the Mel spectrogram (30).

[0046] Referring to Figure 3, it can be seen that even within a single spectrogram, the pixel intensity distributions of each color are different. Therefore, extracting only the areas that best express a specific emotion from a single spectrogram and applying only the extracted areas to the first neural network (40) to predict one of multiple emotions (50) would be much more efficient than applying the entire spectrogram to the first neural network (40).

[0047] In the present invention, by applying an arbitrary first region (31) extracted from the Mel spectrogram (30) rather than the entire Mel spectrogram (30) to the first neural network (40) to predict the audio signal (20) as one of a plurality of emotions (50), the computational load and time of the first neural network (40) can be reduced.

[0048] The time taken to apply the entire Mel spectrogram (30) to the first neural network (40) and predict one of multiple emotions (50) is longer than the time taken to apply the Mel spectrogram (30) to the second neural network (60) to extract an arbitrary first region (31) and apply the extracted arbitrary first region (31) to the first neural network (40) to predict one of multiple emotions (50).

[0049] When an arbitrary first region (31) extracted from a Mel spectrogram (30) is applied to a first neural network (40) to predict an audio signal (20) as one of multiple emotions (50), the output of the first neural network (40) is displayed as a value between 0 and 1.

[0050] When one of the multiple emotions (50) predicted, that is, the largest output value (e.g., 0.7) of the outputs of the first neural network (40) is greater than the threshold value (e.g., 0.5), the emotion (e.g., Sad) corresponding to the largest output value (e.g., 0.7) of the outputs of the first neural network (40) is predicted as the final classified emotion. The output values ​​of the first neural network (40) may be 0.7, 0.2, and 0.1. 0.7 may represent Sad, 0.2 may represent calm, and 0.1 may represent depressive.

[0051] When any one of the predicted emotions (50) (e.g., 0.4) is less than the threshold (e.g., 0.5), the outputs of the first neural network (40) are not reliable. Therefore, when any one of the predicted emotions (50) (e.g., 0.4) is less than the threshold (e.g., 0.5), the processor (11) extracts an arbitrary second region (33) different from an arbitrary first region (31) in the Mel spectrogram (30).

[0052] Referring to (b) of FIG. 4, the processor (11) can extract an arbitrary second region (33) by applying the Mel spectrogram (30) to the second neural network (60). In order to extract an arbitrary second region (33) without overlapping with an arbitrary first region (31), a portion corresponding to an arbitrary first region (31) in the Mel spectrogram (30) can be masked.

[0053] The size of any second region (33) is smaller than the size of any first region (31). The arbitrary second region (33) is applied to the first neural network (40) like the arbitrary first region (31) to improve the reliability of the prediction of the first neural network (40). In some embodiments, a third neural network (not shown) instead of the second neural network (60) may be used to extract the arbitrary second region (33). The third neural network may also be a CNN algorithm.

[0054] According to an embodiment, the processor (11) can extract an arbitrary second region (33) by adding an arbitrary region that is in contact with an arbitrary first region (31) in the Mel spectrogram (30) without using the second neural network (60).

[0055] The smaller the difference between the value (e.g., 0.4) predicted as one of the multiple emotions (50) for the audio signal (20) applied to the first neural network (40) and the threshold value (e.g., 0.5), the smaller the size of the arbitrary second region (33) is set. The value predicted as one of the multiple emotions (50) means the largest value among the output values ​​of the first neural network (40).

[0056] Conversely, the larger the difference between the value (e.g., 0.35) predicted to be one of the multiple emotions (50) for the audio signal (20) applied to the first neural network (40) and the threshold value (e.g., 0.5), the larger the size of the arbitrary second region (33) is set. The size of the second region (33) can be set with or without using the second neural network (60).

[0057] If the difference between the predicted value (e.g., 0.35) of one of the multiple emotions (50) and the threshold value (e.g., 0.5) is large, the output values ​​of the first neural network (40) are all similar, so the first neural network (40) needs more input information for more precise prediction. Therefore, a second arbitrary region (33) with a large size is input to the first neural network (40).

[0058] When the difference between the value (e.g., 0.3) predicted as one of the plurality of emotions (50) of the audio signal (20) and the threshold value (e.g., 0.5) is greater than an arbitrary difference (e.g., 0.1) by applying the first neural network (40), the size of any second region (33) is extracted to be larger than the size of the second region (33) when the difference between the value (e.g., 0.45) predicted as one of the plurality of emotions (50) of the audio signal (20) and the threshold value (e.g., 0.5) is less than an arbitrary difference (e.g., 0.1) by applying the first neural network (40). Therefore, the first neural network (40) can more precisely predict the audio signal (20) as one of the plurality of emotions (50).

[0059] The processor (11) can re-predict the audio signal (20) as one of a plurality of emotions (50) by applying an arbitrary first region (31) and an arbitrary second region (33) to the first neural network (40).

[0060] According to an embodiment, the processor (11) can extract an arbitrary second region (33) without using a second neural network (60). Hereinafter, a specific method for extracting an arbitrary second region (33) without using a second neural network (60) will be described.

[0061] When any one of the predicted emotions (e.g., 0.4) among the multiple emotions (50) is less than a threshold (e.g., 0.5), the processor (11) divides the audio signal (20) into multiple time intervals (e.g., 5 seconds). The processor (11) calculates the Root Mean Square Energy (RMSE) for the audio signal (20) at the multiple time intervals. The RMSE is calculated as shown in the following mathematical expression 1.

[0062] [Mathematical Formula 1]

[0063]

[0064] The above n means an arbitrary time and is an integer, the above N means an integer greater than the above n, and the above x(n) means the amplitude of the audio signal (20).

[0065] The processor (11) selects the time interval corresponding to the largest RMSE value among the RMSEs calculated for each of the multiple time intervals. A large RMSE indicates a loud portion of the audio signal (20), and the loud portion can best reflect the characteristics of the audio signal (20).

[0066] The processor (11) determines whether the time interval selected from the Mel spectrogram (30) and the time interval of the arbitrary first region (31) overlap. Even if the time interval selected from the Mel spectrogram (30) and the time interval of the arbitrary first region (31) partially overlap, they are determined to overlap.

[0067] When it is determined that the selected time interval and the time interval of an arbitrary first region (31) do not overlap, the processor (11) determines an arbitrary region corresponding to the selected time interval as an arbitrary second region (33). The arbitrary region corresponding to the selected time interval means that in the Mel spectrogram (30), the X-axis includes the selected time interval, and the Y-axis includes from 0 to the highest frequency value.

[0068] When it is determined that the selected time interval and the time interval of an arbitrary first region (31) overlap, the processor (11) selects the time interval corresponding to the second largest RMSE value among the RMSEs calculated in each of the plurality of time intervals.

[0069] The processor (11) determines whether the time interval corresponding to the second largest RMSE value in the Mel spectrogram (30) overlaps with the time interval of an arbitrary first region (31).

[0070] When it is determined that the time interval corresponding to the second largest RMSE value and the time interval of the arbitrary first region (31) do not overlap, the processor (11) determines the arbitrary region corresponding to the time interval corresponding to the second largest RMSE value as the arbitrary second region (33). The arbitrary region corresponding to the time interval corresponding to the second largest RMSE value means that in the Mel spectrogram (30), the X-axis includes the time interval corresponding to the second largest RMSE value, and the Y-axis includes from 0 to the highest frequency value.

[0071] FIG. 5 shows a flowchart for explaining a music emotion classification method according to an embodiment of the present invention.

[0072] Referring to FIGS. 1 to 5, the processor (11) converts an audio signal (20) into a Mel spectrogram (30) (S10).

[0073] The processor (11) extracts an arbitrary first region (31) from the Mel spectrogram (30) (S20). The arbitrary first region (31) can be used by the second neural network (60).

[0074] The processor (11) applies a random first region (31) extracted from the Mel spectrogram (30) to the first neural network (40) to predict the audio signal (20) as one of a plurality of emotions (50) (S30).

[0075] While the present invention has been described with reference to the embodiments illustrated in the drawings, these are merely exemplary, and those skilled in the art will appreciate that various modifications and equivalent alternative embodiments are possible. Therefore, the true scope of technical protection of the present invention should be determined by the technical spirit of the appended claims.

Claims

1. A method for classifying music emotions using a spectrogram performed by a processor, A step of converting an audio signal into a Mel spectrogram; A step of extracting an arbitrary first region from the above Mel spectrogram; and A music emotion classification method comprising a step of applying an arbitrary first region extracted from the Mel spectrogram to a first neural network to predict the audio signal as one of a plurality of emotions.

2. In the first paragraph, the step of extracting an arbitrary first region from the Mel spectrogram comprises: A method for classifying music emotions, comprising a step of applying the above Mel spectrogram to a second neural network to extract the first arbitrary region.

3. A method for classifying music emotions in the first paragraph, wherein the first neural network or the second neural network is a CNN (Convolutional Neural network) algorithm.

4. In the first paragraph, the method for classifying music emotions using the spectrogram is as follows: When any one of the above-mentioned multiple emotions is predicted to be below a threshold, a step of extracting an arbitrary second region different from the arbitrary first region from the Mel spectrogram; and A music emotion classification method further comprising a step of re-predicting the audio signal as one of a plurality of emotions by applying the arbitrary first region and the arbitrary second region to the first neural network.

5. A processor that executes music emotion classification commands using spectrograms; and Contains memory for storing the above commands, The above commands are, Converts audio signals into Mel spectrograms, Extract an arbitrary first region from the above Mel spectrogram, A computing device implemented to predict the audio signal as one of a plurality of emotions by applying an arbitrary first region extracted from the above Mel spectrogram to a first neural network.

6. In the fifth paragraph, the commands for extracting an arbitrary first region from the Mel spectrogram are: A computing device implemented to extract the arbitrary first region by applying the above Mel spectrogram to a second neural network.

7. A computing device in accordance with paragraph 5, wherein the first neural network or the second neural network is a CNN (Convolutional Neural network) algorithm.

8. In paragraph 5, the above commands, When any one of the above-mentioned multiple emotions is predicted to be below the threshold, an arbitrary second region different from the arbitrary first region is extracted from the Mel spectrogram. A computing device further configured to re-predict the audio signal as one of a plurality of emotions by applying the first arbitrary region and the second arbitrary region to the first neural network.

Images