Operation Method for Music Recommendation and device supporting the same

Abstract

The present invention discloses a music recommendation providing device and a method of operation thereof, comprising a memory for storing a sound source and a processor functionally connected to the memory, wherein the processor extracts a singing section from the stored sound source to generate segment-unit input data, obtains a song-unit weight corresponding to the segment-unit and the sound source based on a hierarchical attention model, extracts and stores a voice marker capable of distinguishing an artist based on the segment-unit weight and the song-unit weight, and is configured to recommend other sound sources other than the sound source based on the voice marker.

Description

Technology Field

[0001] The present invention relates to music recommendation, and more specifically, to a method for operating music content and a device supporting the same that enables the recommendation of related music based on the audio characteristics of a singer. Background Technology

[0002] Recently, due to technological advancements, various forms of media content are being used in daily life. For instance, music content is increasingly establishing itself as a form of content consumed daily by many users. In this regard, technologies for automatically generating playlists for music playback based on a user's listening history are being proposed. These technologies generate playlists by utilizing models that analyze user history or its similarity, by using meta-information as a filter, or by employing feature discovery or feature learning on undivided audio signals.

[0003] Existing playlist generation methods utilizing models that analyze user history or similarity have limitations because they do not use the audio signals themselves; consequently, they fail to generate playlists for songs without user history and merely produce playlists skewed toward popular tracks. Similarity-based playlist generation methods that rely on unseparated audio signals and metadata suffer from the problem that it is difficult to identify the reasons for similarities between songs included in a playlist unless interpreted retrospectively after creation. Furthermore, since the reasons for song similarities can be influenced by a wide variety of factors, there is the issue of having to utilize text information related to the songs during the playlist generation process. Prior art literature

[0004] Korean Published Patent No. 10-2017-0136200, registered December 11, 2017 (Title: Method and system for automatic playlist generation using audio content and meta information) The problem to be solved

[0005] To solve the aforementioned problems, the present invention provides a method for operating music content and a device supporting the same, which extracts factors reflecting unique artist characteristics such as timbre using source-separated signals, and thereby limits the reason for similarity to unique artist characteristics that are robust to changes in song style and accompaniment, thereby enabling the automatic generation of artist-unit recommendation lists and playlists based on similar timbres.

[0006] The present invention provides a method for operating music content and an apparatus supporting the same, which enables recommending songs with similar vocal tones and related artists in providing a recommendation service based on separated sound source signals and neural network processing.

[0007] The present invention provides a method for operating music content capable of providing music and singer recommendation services to a user based on unique characteristic factors of a sound source, and an apparatus supporting the same.

[0008] The present invention provides a method for operating music content and a device supporting the same, which enables a user to more easily select a song more suitable to their taste in an environment where a recommendation service based on user selection, such as a karaoke room, is provided.

[0009] However, the present invention is not limited to the purposes mentioned above, and other unmentioned purposes will be clearly understood from the description below. means of solving the problem

[0010] The present invention, for achieving the purpose described above, may include a memory for storing a plurality of sound sources and a processor functionally connected to said memory. The processor may be configured to generate segment-unit input data by extracting a singing section from a specific sound source among the stored plurality of sound sources, obtain song-unit weights corresponding to said segment-units and said sound sources based on a hierarchical attention model, extract and store a voice marker capable of distinguishing an artist based on said segment-unit weights and said song-unit weights, and recommend music based on said voice marker.

[0011] Here, the processor may be configured to extract a vocal signal from which the accompaniment is separated from the specific sound source, and to extract a section having a time-frequency characteristic greater than or equal to a predefined first threshold value in the extracted vocal signal as the singing section.

[0012] Additionally, the processor may be configured to provide the time-frequency characteristic as an input to an encoder and to convert the output value of the encoder into a characteristic value in the segment unit of the singing section.

[0013] In particular, the processor may be configured to apply the segment-unit characteristic value to the hierarchical attention model to calculate the segment-unit attention weight and the song-unit attention weight.

[0014] Additionally, the processor may be configured to extract a plurality of songs having a song-unit weight greater than or equal to a preset second threshold as representative candidate songs of an artist, extract a plurality of song-specific segment indices having a segment-unit weight greater than or equal to a preset third threshold among the representative candidate songs, and generate the voice marker based on the song-specific segment indices.

[0015] Additionally, the processor may be configured to select a specific artist based on the user's audio listening history, compare the song-unit weights of the selected artist with a predefined fourth threshold, extract artist voice markers of songs above the predefined threshold, and search for and recommend other artists having a similarity of at least a specified size of the extracted voice markers.

[0016] The method for operating music content according to the present invention may include a music recommendation providing device, the steps of: generating input data in segment units by extracting a singing section from a specific sound source among a plurality of previously stored sound sources; obtaining song unit weights corresponding to the segment units and the sound source based on a hierarchical attention model; and extracting and storing voice markers capable of distinguishing artists based on the segment unit weights and the song unit weights.

[0017] Here, the step of generating the input data may include the step of extracting a vocal signal from which the accompaniment is separated from the specific sound source, and the step of extracting a section having a time-frequency characteristic greater than or equal to a predefined first threshold value in the extracted vocal signal as the singing section.

[0018] And, the step of obtaining the weight may include the step of providing the time-frequency characteristic as an input to an encoder to convert the output value of the encoder into a characteristic value at the segment level of the singing section, and the step of applying the characteristic value at the segment level to the hierarchical attention model to calculate the attention weight at the segment level and the attention weight at the song level.

[0019] The above-mentioned saving step may include the step of extracting a plurality of songs having a song unit weight greater than or equal to a preset second threshold as representative candidate songs of the artist, the step of extracting a plurality of song-specific segment indices having a segment unit weight greater than or equal to a preset third threshold among the representative candidate songs, and the step of generating the voice marker based on the song-specific segment indices.

[0020] Meanwhile, the above method may further include the steps of selecting a specific artist based on the user's audio listening history, comparing the song-unit weights of the selected artist with a predefined fourth threshold, extracting artist voice markers of songs above the predefined threshold, and searching for and recommending other artists having a similarity of at least a specified size of the extracted voice markers. Effects of the invention

[0021] According to the present invention, by recommending songs and related artists with similar timbres based on separated sound source signals and neural network processing, the content production and consumption environment is improved, and by facilitating the supply of more music content, the opportunity to easily search for and listen to a wider variety of artists that suit consumers' tastes is provided.

[0022] In addition, the present invention can provide clear reasons for music recommendations and, at the same time, contribute to enhancing user experience and improving the reliability of the service.

[0023] In addition, the present invention can provide song selection recommendations with high user satisfaction in a karaoke environment that reflect the singer's unique style, which cannot be reflected through song-unit analysis and recommendation. Brief explanation of the drawing

[0024] FIG. 1a is a diagram showing an example of a music recommendation provision system according to an embodiment of the present invention. FIG. 1b is a drawing showing an example of a music recommendation providing device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of a processor configuration of a music recommendation providing device according to an embodiment of the present invention. FIG. 3 is a diagram showing an example of an operation method of an input data generation unit related to the generation of a music recommendation list according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of an artist voice marker extraction method among the music recommendation list generation methods according to an embodiment of the present invention. FIG. 5 is a diagram showing the structure of a singer recognition model utilizing hierarchical attention according to an embodiment of the present invention. FIG. 6a is a diagram illustrating a method for extracting artist voice markers according to an embodiment of the present invention. FIG. 6b is a flowchart of an artist voice marker extraction method illustrating the flow of characteristic and weight extraction and utilization according to one embodiment of the present invention. FIG. 7 is a flowchart of a personalized artist recommendation method using an artist voice marker according to an embodiment of the present invention. Specific details for implementing the invention

[0025] In order to clarify the features and advantages of the means for solving the problem of the present invention, the present invention will be described in more detail with reference to specific embodiments of the present invention illustrated in the attached drawings.

[0026] However, detailed descriptions of known functions or configurations that may obscure the essence of the invention are omitted in the following description and the attached drawings. Additionally, it should be noted that identical components throughout the drawings are indicated by the same reference numerals whenever possible.

[0027] Terms and words used in the following description and drawings should not be interpreted as being limited to their ordinary or dictionary meanings, but should be interpreted in a meaning and concept consistent with the technical spirit of the invention, based on the principle that the inventor can appropriately define the concept of terms to best describe his invention. Accordingly, the embodiments described in this specification and the configurations illustrated in the drawings are merely the most preferred embodiments of the invention and do not represent all aspects of the technical spirit of the invention; therefore, it should be understood that various equivalents and modifications capable of replacing them may exist at the time of filing this application.

[0028] Furthermore, terms including ordinal numbers, such as first, second, etc., are used to describe various components and are used solely for the purpose of distinguishing one component from another, and are not used to limit said components. For example, without departing from the scope of the present invention, the second component may be named the first component, and similarly, the first component may be named the second component.

[0029] Furthermore, the terms used in this specification are used merely to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. Additionally, terms such as “comprising” or “having” described in this specification are intended to indicate the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0030] Additionally, terms such as "part," "unit," and "module" as described in the specification refer to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software. Furthermore, "one (a or an)," "one," "the," and similar related terms may be used in the context describing the invention (particularly in the context of the following claims) in a sense that includes both singular and plural forms, unless otherwise indicated in the specification or clearly contradicted by the context.

[0031] In addition to the terms described above, specific terms used in the following description are provided to aid in understanding the present invention, and the use of such specific terms may be modified in other forms without departing from the technical spirit of the present invention.

[0032] In addition, embodiments within the scope of the present invention include a computer-readable medium having or transmitting computer-executable instructions or data structures stored on a computer-readable medium. Such a computer-readable medium may be any available medium accessible by a general-purpose or special-purpose computer system. For example, such a computer-readable medium may include, but is not limited to, physical storage media such as RAM, ROM, EPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium accessible by a general-purpose or special-purpose computer system that can be used to store or transmit certain program code means in the form of computer-executable instructions, computer-readable instructions or data structures.

[0033] The present invention, described below, can provide reasons for similarity in the recommendation of similar content in the field of music. That is, conventionally, because various characteristics of a song are reflected simultaneously, it is difficult to identify the reasons for similarity. Furthermore, since the singer's unique timbre or singing style cannot be specifically reflected, recommendations at the singer level were impossible. Additionally, although meta-information describing the song, consisting of text describing the song, can be used as a post-processing step, there was a problem in that the task of collecting meta-information indicating timbre and singing style had to be performed beforehand. In this regard, the present invention provides a method for recommending songs and artists with similar timbres—which is one of the factors considered by users when providing recommendation services—based on an audio signal from which accompaniment has been removed and neural network processing.

[0035] FIG. 1a is a diagram showing an example of a music recommendation provision system according to an embodiment of the present invention.

[0036] Referring to FIG. 1a, a music recommendation providing system (10) according to an embodiment of the present invention may include an information input device (300), a user terminal (200), a communication network (500), and a music recommendation providing device (100). This music recommendation providing system (10) may extract a singer's voice characteristics (e.g., artist voice marker) and provide a music recommendation function based thereon.

[0037] The information input device (300) may be a device that provides music content to the music recommendation providing device (100). For example, the information input device (300) may generate new music and provide the generated new music content to the music recommendation providing device (100). In this regard, the information input device (300) may grant accounts to various artists, receive and store music content created by the artists, and provide the music content to the music recommendation providing device (100) at designated intervals. Meanwhile, in the music recommendation providing system (10), the information input device (300) may be integrated with the music recommendation providing device (100) and operated as a single configuration.

[0038] The user terminal (200) can access the music recommendation providing device (100) through the communication network (500) and search for at least one music content provided by the music recommendation providing device (100). The user terminal (200) can receive and play at least one music content among the various music contents provided by the music recommendation providing device (100). In this process, the search history of the music content searched by the user terminal (200) or the download history of the music content downloaded can be stored and managed in the music recommendation providing device (100).

[0039] The music recommendation providing device (100) can receive and store music content from an information input device (300) through the communication network (500). The music recommendation providing device (100) can classify and store the stored music content based on specified conditions, such as the audio characteristics of an artist. The music recommendation providing device (100) can form a communication channel with a user terminal (200) upon request and provide a music playlist related to the user terminal (200). In this regard, the music recommendation providing device (100) may include a music content database classified by the audio characteristics of an artist. The music recommendation providing device (100) can provide a song by another artist similar to the audio characteristics of a specific artist searched by the user terminal (200) or a specific song downloaded by the user terminal (200) as a recommended song.

[0040] The communication network (500) can form a communication channel between the information input device (300), the user terminal (200), and the music recommendation providing device (100). At least one communication method of the communication network (500) can be supported. For example, the communication channel between the information input device (300) and the music recommendation providing device (100) and the communication channel between the music recommendation providing device (100) and the user terminal (200) within the communication network (500) can be implemented as the same type or a different type. The communication network (500) can be implemented as, for example, at least one of a wireless communication network or a wired communication network.

[0041] Meanwhile, in the above description, the music recommendation function of the present invention is exemplified as being provided to a user terminal (200) through a music recommendation providing device (100), but the present invention is not limited thereto. For example, the music recommendation function of the present invention may receive music recommendations from the music recommendation providing device (100) while the user terminal (200) is connected to the music recommendation providing device (100), but it may also be configured to receive music recommendations by a processor within the user terminal (200). Accordingly, the configuration of the music recommendation providing device (100) described below may also be the configuration of the user terminal (200).

[0042] FIG. 1b is a diagram illustrating an example of a music recommendation providing device according to an embodiment of the present invention. Among the characteristics utilized as sound source signal-based music recommendation characteristics, segment-unit characteristics may be included. For example, even for the same singer, the musical style may differ from song to song. For instance, when songs by a singer who performs both rap music and ballad music are displayed in a space where differences can be expressed, the rap song and the ballad song, despite being the same singer, have a long distance corresponding to a relatively large difference. Even within the same song, the characteristics of the intro section and the chorus section may have a long distance representing a relatively large difference. When representing such different segment-unit characteristics at a higher level, such as a song, simple methods such as averaging may be utilized. In this case, when extracting song-unit characteristics, since the style or atmosphere of the song is similar within the same song, methods such as averaging and valuing result in relatively less loss of information, but they are not suitable for extracting singer-unit characteristics. Accordingly, the music recommendation providing device (100) of the present invention can provide a singing timbre-based artist voice marker extraction and a music recommendation function based thereon.

[0043] Referring to FIG. 1b, the music recommendation providing device (100) recommends a singer whose singing tone is similar to the music the user has listened to among a plurality of items. At this time, the music recommendation providing device (100) may include a memory (110), a database (120), and a processor (130).

[0044] The memory (110) can store information used by the music recommendation providing device (100). The memory (110) may be configured as a single device with the processor (130) or connected via wired or wireless connection to temporarily or permanently store instructions and data processed by the processor (130). For example, in the present invention, it can store instructions for generating a music recommendation list and data to be processed, including separated sound sources and singer information. The music recommendation list may include items classified based on the segment-unit singing or timbre characteristics of a specific song.

[0045] The database (120) can store sound sources. Alternatively, the database (120) can store sound sources with accompaniment removed, vocal appearance location data, whether it is a solo singer, singer's gender and country, song style and mood, album information, user listening history, etc.

[0046] The processor (130) may include any device capable of processing any sequence and may be connected to memory (110) and database (120). The processor (130) may process data in the database (120) to generate eigenvectors for generating a music recommendation list. In this regard, the processor (130) may include a configuration as illustrated in FIG. 3.

[0047] Additionally or generally, if the entity performing the music recommendation function exists as a server via a communication network (500) and the system is one in which a user terminal (200) receives and plays music from the server via the communication network (500), the music recommendation providing device (100) may further include a communication circuit. The processor (130) of the music recommendation providing device (100) forms a communication channel with the user terminal (200) through the communication circuit, receives a music search request, a music download request, etc. from the user terminal (200), and can store and manage the user's listening history based on this. In response to the user search request or download request, the processor (130) can generate a recommendation list while providing related data or music to the user terminal (200) and provide it to the user terminal (200). In relation to the generation of the recommendation list above, the processor (130) can process the separation of vocals and background audio, detection of artist voice markers using a hierarchical attention method, comparison between detected artist voice markers, search for artists and songs having similarity greater than a specified threshold value, and generation of a recommendation list based on the search results.

[0048] FIG. 2 is a diagram showing an example of a processor configuration of a music recommendation providing device according to an embodiment of the present invention.

[0049] Referring to FIG. 2, the processor (130) of the music recommendation providing device (100) of the present invention may include an input data generation unit (210), an artist voice marker extraction unit (220), and an artist recommendation unit (230).

[0050] The above input data generation unit (210) can process the generation of input data for the generation of artist voice markers. In this regard, the input data generation unit (210) can perform sound source acquisition, sound source separation, and vocal filtering. In relation to sound source acquisition, the input data generation unit (210) can acquire meta information and learning and actual use sound sources from the database (120). In relation to sound source separation, the input data generation unit (210) can extract a vocal signal from which the accompaniment has been separated. In relation to vocal filtering, the input data generation unit (210) can detect vocal sections in a song.

[0051] The artist voice marker extraction unit (220) can generate artist voice markers based on sound source and singing filtered data obtained by the input data generation unit (210). In relation to the generation of artist voice markers, the artist voice marker extraction unit (220) can perform artist recognition and artist voice marker extraction based on a hierarchical attention model.

[0052] The artist recommendation unit (230) can generate recommended artists by utilizing artist voice markers. For example, the artist recommendation unit (230) can select music that the user has listened to, select at least one artist of the music, and then detect other artists that have virtual timbre characteristics similar to the artist above a threshold value using the artist voice markers. In this process, the artist recommendation unit (230) can classify the similarity of the voice markers for the songs by magnitude (e.g., 90%, 80%, 70%...), assign a similarity confidence level to the voice markers for the songs of the classified artists, place songs with high similarity artist voice markers at the front of the list, or compose a list consisting only of songs with artist voice markers that have similarity above a certain threshold value.

[0053] FIG. 3 is a diagram showing an example of an operation method of an input data generation unit related to the generation of a music recommendation list according to an embodiment of the present invention.

[0054] Referring to FIG. 3, in the input data processing method related to the generation of a music recommendation list, each step of the input data processing method described below can be performed in order. For example, among the steps described below, steps that have a preceding step can receive the output of the preceding step as input to perform model training. That is, the sound source obtained in the sound source acquisition step is used as input for sound source separation training, and the vocal signal obtained in the sound source separation step can be used as input for singing filtering training.

[0055] In step S310, the processor (130) (e.g., input data generation unit (210)) obtains meta-information and sound sources for learning and actual use from the database (120). Here, the input subjects of the sound sources and meta-information required for learning are not limited. The meta-information required for model learning may include text information corresponding to the sound source. For example, the meta-information required for model learning may include one or more of singer information, gender information, country information, album information, singing location information, and information on whether the singer is a solo artist regarding the sound source.

[0056] In step S320, the processor (130) can separate multiple signals contained in the sound source. For example, the processor (130) can extract a vocal signal from which the accompaniment has been separated. In this process, the processor (130) utilizes signal processing techniques to convert the signal in the time domain into characteristics in the time-frequency unit. For example, the sound source can be represented in the time-frequency domain through a Short-Time Fourier Transform (STFT), and both magnitude and phase can be utilized. Next, the processor (130) performs neural network learning between the magnitude and phase of the learning sound source and the magnitude and phase of the corresponding recorded vocal, and can obtain a signal separated from the actual sound source by utilizing the included weights. At this time, the artificial neural network learning can be performed based on one or more neural network algorithms. The neural network algorithm may, for example, use a Convolutional Neural Network (CNN) based model with skip-connections. At this time, the input time unit of the artificial neural network is not limited and is typically used as a signal unit corresponding to 1 to 10 seconds.

[0057] In step S330, the processor (130) can extract a singing segment. For example, the processor (130) can extract a singing segment from a vocal signal from which the accompaniment has been separated. In this regard, the processor (130) utilizes signal processing techniques to convert a signal in the time domain into characteristics in the time-frequency unit. For example, the sound source can be represented in the time-frequency domain through a Short-Time Fourier Transform (STFT), and the converted sound source in the time-frequency domain can be converted into time-frequency domain characteristics in the Mel scale through a Mel-frequency filter bank transformation. Additionally, the converted characteristics in the magnitude unit for the sound source can be expressed in decibel (dB) units through a log transformation. Next, the processor (130) performs neural network learning between the time-frequency representation of the sound source from which the accompaniment has been removed and the singing position information. The processor (130) can use the learned model to generate probability values ​​between 0 and 1 based on an arbitrary time unit. This probability value can be filtered by a threshold value and used as a criterion for extracting singing segments. The threshold value can be arbitrarily selected considering the distribution and can typically be set within the range of 0.6 to 0.8. For example, the processor (130) can set an arbitrary time unit to 3 seconds and a threshold value to 0.7 in a sound source of 30 seconds in total length. If the probability value predicted by the artificial neural network is [0.1, 0.1, 0.2, 0.75, 0.8, 0.9, 0.9, 0.9, 0.9, 0.9], the processor (130) can filter out the preceding 9 seconds that are not 0.7 and extract a total of 21 seconds of characteristics as singing segment characteristics. At this time, the input time unit of the artificial neural network is not limited and is typically used as a time unit corresponding to 1 to 10 seconds. One or more neural network algorithms may be used for artificial neural network learning. For example, artificial neural network algorithms can utilize artificial neural network models based on CNN (Convolutional Neural Network).A learning model for predicting singing segments includes at least one convolutional layer, a pooling layer, and a fully connected layer, and a processor (130) can receive a time-frequency representation with the accompaniment removed as input and repeatedly perform convolutional and pooling operations to output whether singing is present. Various functions can be used as functions for each layer, for example, ReLU function, Leaky ReLU function, sigmoid function, tanh function, etc. Various pooling techniques such as maximum pooling, average pooling, and L2 pooling can be used as pooling techniques.

[0058] In step S340, the processor (130) may generate input data in segment units for extracting artist voice markers. For example, the processor (130) may extract singing section time-frequency characteristics based on the extracted singing information. In this step, the processor (130) utilizes the extracted time-frequency characteristics, and if a specific sound source section corresponds to a singing section, the singing section characteristics may be extracted in such a way that the time-frequency characteristics corresponding to the input time unit set in the previous step are selected. At this time, the number of segment-unit time-frequency characteristics extracted per song is not limited, and typically, time units corresponding to 90 seconds or less per song can be used.

[0059] FIG. 4 is a diagram showing an example of an artist voice marker extraction method among the music recommendation list generation methods according to an embodiment of the present invention.

[0060] Referring to FIG. 4, in relation to the artist voice marker extraction method of the present invention, a processor (130) (e.g., artist voice marker extraction unit (220)) can extract an artist voice marker by utilizing the input data extracted in the step described in FIG. 3. In this step, the processor (130) can process the result of the previous step according to the order of the steps so that it can be utilized in the next step as a learned model or weight, etc.

[0061] In step S410, the processor (130) can learn weights at the song and segment levels through a singer recognition task based on a hierarchical attention model. In relation to the singer recognition task, the processor (130) can perform the singer recognition task by matching which singer the time-frequency characteristics of a specific segment among several singers correspond to. In the present invention, the singer recognition model hierarchically structures the input x into singer-song-segment to predict the output y, while simultaneously learning hierarchical weights corresponding to the song-segment. At this time, one or more neural network algorithms may be used to construct the encoder. For example, a timbral CNN block in which the filter includes all frequency axes, an RNN that considers both forward and backward directions, etc., may be used. For example, the processor (130) can learn and store a neural network between singing characteristics and singer information consisting only of solo singers, based on a hierarchical attention model that hierarchically calculates the sum of weights at the song and segment levels of the input data using the input data and singer information extracted in the previous step. In relation to the hierarchical attention model, the processor (130) trains the model to assign high weights to segments that have a significant influence on singer recognition within a specific song through segment-unit weight learning, and at the same time, assigns high weights to songs that have a significant influence on singer recognition within a specific song through song-unit weight learning, and stores the weights of the trained model. In this regard, the artificial neural network model used may include an encoder that represents time-frequency characteristics in a lower dimension and a hierarchical attention block that calculates segment and song-unit weights.

[0062] In step S420, the processor (130) can obtain segment-unit embedding feature vectors and curve-unit and segment-unit attention weights among the weights of the stored model.

[0063] In step S430, the processor (130) can extract artist voice markers by processing the acquired embedding feature vector based on the acquired weights (e.g., hierarchical attention weights).

[0064] FIG. 5 is a diagram showing the structure of a singer recognition model utilizing hierarchical attention according to an embodiment of the present invention.

[0065] Referring to FIG. 5, as previously described in FIG. 3, when the processor (130) acquires a sound source (e.g., input data of song2), it divides the acquired sound source into designated segment units (segment N of song 2) and provides the divided segments as inputs to the encoder (51). In this process, the processor (130) can extract time-frequency characteristics of the sound source in segment units and provide them as inputs to the encoder (51).

[0066] The time-frequency characteristics extracted in the previous step can be passed through an encoder (51) and converted into segment-unit characteristic values ​​(52) that maintain the time order. At this time, CNN and RNN may be applied to calculate the characteristic values ​​(52) of the encoder (51). However, the present invention is not limited thereto, and for example, the encoder (51) may utilize a fully connected neural network to extract the characteristic values ​​(52).

[0067] The processor (130) can process the output characteristic value (52) of the encoder (51) as an input to a hierarchical attention block (53) that calculates segment-unit weights. At this time, the dimension of the characteristic value (52) is not fixed, and typically the dimension of the characteristic value (52) is used from 64 to 256. In addition, the processor (130) performs data processing based on the hierarchical attention block (53) to calculate segment-unit weights for multiple songs, and can perform artist voice marker extraction for multiple songs using a hierarchical attention block (54) that calculates weights for each song.

[0068] In this regard, the processor (130) can calculate the segment-unit attention weight (532) and the song-unit attention weight (542) based on mathematical formula 1.

[0069] [Mathematical Formula 1]

[0070]

[0071] In the above mathematical formula 1, i is the index for songs by the same singer (or artist), and t is the index for segments of the same song. i is sorted based on the release year and album order to reflect the influence of music changing over time, while t is sorted according to the flow of the songs. W11 and W12 are weight vectors for calculating weights at the segment level, and H it represents the characteristic value of a specific song (i) at a specific point in time (t). A it is a segment-unit attention weight (532). W21 and W22 are weight vectors for calculating weights per song, and song i is a characteristic value at a specific song (i), and A irepresents the song-unit attention weight (542) of a specific song. When a bidirectional RNN is used in the encoder (51), the feature value (52) in Equation 1 may be a value that combines the feature values ​​of the unidirectional and inverse directions at a specific time point (t).

[0072] The segment-unit attention weight value (532) can be expressed as a probability value between 0 and 1 and indicates how much to reflect for each segment. The segment-unit attention weight (532) is operated with the feature value (52) and used as an input (541) to calculate the singer-unit weight (542). The singer-unit attention weight value (542) can be expressed as a probability value between 0 and 1 and indicates how much to reflect for each song.

[0073] The artist voice marker extraction unit (220) can estimate singer recognition information for a song group by singer by utilizing the fully connected layer (55) in the neural network structure between the value corresponding to the artist in mathematical formula 1 and the singer voice information through the singer recognition operation.

[0074] FIG. 6a is a diagram illustrating a method for extracting artist voice markers according to an embodiment of the present invention. In particular, FIG. 6a is a diagram illustrating step S430 described above in FIG. 4.

[0075] Referring to FIGS. 5 and 6a, the processor (130) may obtain, in relation to artist voice marker extraction, the output characteristic value (52) of the encoder (51), the attention weight per segment (532), and the attention weight per song (542) as inputs. However, the method of reflecting weights according to the present invention is not limited to one method. The method of reflecting weights may utilize at least one of the following methods: a method of filtering using a threshold value (621), a method of hierarchically multiplying weights by characteristic values ​​(52) (622), and a method of calculating based on meta-information but combining the two using style-specific weights (623) (e.g., one of the three methods may be selected, or a combination that can be made may be created and the results combined). For example, the artist voice marker extraction unit (220) can extract multiple songs with high song-unit weights based on a threshold value as candidate songs representing the artist (611), and extract multiple song-specific segment indices with high segment-unit attention weights based on a threshold value (621). Alternatively, the artist voice marker extraction unit (220) can calculate the artist voice marker (63) by multiplying the segment-unit attention weights of all songs by the segment embedding values ​​and adding them by song (612), and then multiplying the song-unit attention weights based on the added values ​​and adding them by singer (622). Alternatively, the artist voice marker extraction unit (220) can calculate the artist voice marker (63) by extracting candidate segments representing multiple songs with high segment-unit attention weights based on a threshold value for all songs (613), calculating style-specific weights for the extracted values, multiplying them by the songs having the corresponding style, and adding them by singer (623). For all song segments and song-unit attention weight values, the threshold value is not limited to a single value and can be set to a weight value corresponding to the top 20% to 50% depending on the distribution of weights.

[0076] FIG. 6b is a flowchart of an artist voice marker extraction method illustrating the flow of characteristic and weight extraction and utilization according to one embodiment of the present invention.

[0077] Referring to FIG. 6b, the processor (130) divides the sound sources of a specific singer into multiple segments, wherein the number of time-frequency characteristics can be determined in time units smaller than the total playback time of the sound source. The processor (130) performs encoding of the encoder (51) on the multiple segments to extract characteristic values ​​(52), and for the extracted characteristic values ​​(52), segment-unit attention weight values ​​(532) and song-unit attention weight values ​​(542) can be obtained using the previously described mathematical formula 1.

[0078] The above processor (130) can extract an artist voice marker (63) by operating at least one of the following methods based on the segment unit attention weight value (532) and the song unit attention weight value (542): a first method of extracting representative candidate songs based on the threshold value (611) and extracting segment indices based on the threshold value (621) described in FIG. 6a; a second method of applying the segment unit weight value of all songs to the songs (612) and then applying the song unit weight value to the singer unit (22); and a third method of applying the threshold value to all songs to extract candidate segments (613) and applying the style-specific weight to the songs (623).

[0079] FIG. 7 is a flowchart of a personalized artist recommendation method using an artist voice marker according to an embodiment of the present invention.

[0080] Referring to FIG. 7, in step S701, the processor (130), such as the artist recommendation unit (230), checks the user's listening history for the audio source, sorts the artists in the user's history in order of listening frequency, and then compares the song-unit weights with the threshold value of the corresponding artist to select a seed artist based on the history frequency. For example, the artist recommendation unit (230) sorts the artists in the user's history in descending order of listening frequency and then compares the song-unit weights of the corresponding artist with a preset threshold value. At this time, the threshold value is determined by considering the distribution of threshold values ​​for all of the artist's songs and can typically be set to a weight value corresponding to the top 20% to 50%. The processor (130) can exclude songs that were not important for recognizing the artist's voice (e.g., songs with a listening history of less than a certain number of times or a certain amount of time) by using the threshold setting.

[0081] In step S702, the artist recommendation unit (230) checks whether there is a song that is above a threshold value. If there is a song that is above the threshold value, in step S703, the similarity is calculated using artist voice markers, and a list of candidate artists that are similar by a certain percentage or more can be recommended through similarity comparison. In this process, the artist recommendation unit (230) can select the artist of the song that is above the threshold value as a seed artist and calculate the similarity between the seed artist and other artists. At this time, the voice marker of the seed artist can be used to calculate the similarity between artists. The distance used for similarity calculation is not limited to one type, and various distances can be used. For example, cosine distance, Manhattan distance, Euclidean distance, etc., can be used.

[0082] If there are no songs with a value higher than the threshold value in step S702, the artist recommendation unit (230) may repeat the above process in step S704 by excluding the corresponding artist from the user history and selecting the artist with the next highest history frequency as the seed artist. In addition to the method described above, similar artists can be extracted using artist voice markers, and for example, it is also possible to extract similar artists for all artists by setting all artists as seed artists.

[0083] The artist recommendation system having a similar timbre utilizing the artist voice marker of the present invention described above can achieve karaoke song selection recommendations, karaoke score calculation, vocal synthesis evaluation, and improved singer recognition accuracy.

[0084] As explained above, this specification includes details of a number of specific embodiments, but these should not be understood as limiting the scope of any invention or claimables, but rather as descriptions of features that may be characteristic of a specific embodiment of a specific invention.

[0085] Furthermore, although operations are depicted in the drawings in a specific order, this should not be understood as requiring that such operations be performed in that specific or sequential order or that all depicted operations must be performed in order to obtain a desirable result. In certain cases, multitasking and parallel processing may be advantageous. Additionally, the separation of the various system components of the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0086] The description provided herein presents the best mode of the invention and offers examples to explain the invention and to enable a person skilled in the art to manufacture and use the invention. The specification thus written is not intended to limit the invention to the specific terms presented. Accordingly, although the invention has been described in detail with reference to the examples above, a person skilled in the art can make modifications, changes, and variations to these examples without departing from the scope of the invention.

[0087] Therefore, the scope of the present invention should not be determined by the described embodiments but by the claims. Industrial applicability

[0088] According to the present invention, the present invention can promote the consumption of sound sources by making it easier to search for artists or songs that suit user preferences. Explanation of the symbols

[0089] 10: Music recommendation system 100: Music recommendation device 200: User terminal 300: Information input device 500: Communication network

Claims

Claim 1 A music recommendation providing device comprising: a memory for storing a sound source; and a processor functionally connected to the memory; wherein the processor extracts a singing section from the sound source to generate segment-unit input data, obtains song-unit weights corresponding to the segment-units and the sound source based on a hierarchical attention model, extracts a plurality of songs having song-unit weights greater than or equal to a preset first threshold value as representative candidate songs for an artist, extracts a plurality of song-specific segment indices among the representative candidate songs having segment-unit weights greater than or equal to a preset second threshold value, generates a voice marker capable of distinguishing an artist based on the song-specific segment indices, and is configured to recommend other sound sources other than the sound source based on the voice marker. Claim 2 In claim 1, the processor extracts a vocal signal from which the accompaniment is separated from the sound source, and the predefined in the extracted vocal signal 3 A music recommendation providing device characterized by being configured to extract a section having time-frequency characteristics greater than or equal to a threshold value as the singing section. Claim 3 A music recommendation providing device according to claim 2, characterized in that the processor provides the time-frequency characteristic as an input to an encoder and is configured to convert the output value of the encoder into a characteristic value in segment units of the singing section. Claim 4 A music recommendation providing device according to claim 3, characterized in that the processor is configured to apply the segment-unit characteristic value to the hierarchical attention model to calculate the segment-unit attention weight and the song-unit attention weight. Claim 5 delete Claim 6 A music recommendation providing device according to claim 1, characterized in that the processor selects a specific artist based on the user's audio listening history, compares the song unit weights of the selected artist with a predefined fourth threshold value, extracts artist voice markers of songs above the predefined threshold value, and searches for and recommends other artists having a similarity of a specified size or greater than the extracted voice markers. Claim 7 A method for operating music content characterized by comprising: a step of generating segment-unit input data by extracting a singing section from a previously stored sound source; a step of obtaining song-unit weights corresponding to the segment-units and the sound source based on a hierarchical attention model; a step of extracting a plurality of songs having song-unit weights greater than or equal to a previously set first threshold as representative candidate songs for an artist; a step of extracting a plurality of song-specific segment indices among the representative candidate songs having segment-unit weights greater than or equal to a previously set second threshold; and a step of extracting and storing voice markers capable of distinguishing an artist based on the song-specific segment indices. Claim 8 In claim 7, the step of generating the input data comprises: a step of extracting a vocal signal from which the accompaniment is separated from the sound source; and a predefined [type] in the extracted vocal signal 3 A method for operating music content characterized by including the step of extracting a section having time-frequency characteristics greater than or equal to a threshold value as the singing section. Claim 9 A method for operating music content according to claim 8, wherein the step of obtaining the weight comprises: providing the time-frequency characteristic as an input to an encoder to convert the output value of the encoder into a characteristic value in the segment unit of the singing section; and applying the characteristic value in the segment unit to the hierarchical attention model to calculate the attention weight in the segment unit and the attention weight in the song unit. Claim 10 delete Claim 11 A method for operating music content according to claim 7, further comprising: a step of selecting a specific artist based on a user's audio listening history; a step of comparing the song unit weights of the selected artist with a predefined fourth threshold value; a step of extracting artist voice markers of songs above the predefined threshold value; and a step of searching for and recommending other artists having a similarity of a specified size or greater than that of the extracted voice markers.

Images