Music recommendation method, model training method, device, and storage medium

Abstract

This application discloses a method for recommending background music, a training method for a model, an apparatus, and a storage medium. The method includes: acquiring a first attribute feature of a reading audio, the first attribute feature including one or more of duration, pitch, and intensity features; processing the first attribute feature with second attribute features of each piece of background music using a pre-trained matching degree prediction model to obtain the matching degree between the reading audio and each piece of background music; wherein the second attribute feature includes one or more of rhythm, mood, range, and pitch features of the background music; and recommending background music based on the matching degree between the reading audio and each piece of background music. This application can make the recommended background music more personalized and make the superposition effect of the recommended background music more harmonious when superimposed with the reading audio.

Description

Technical Field

[0001] This invention relates to the field of audio processing technology, and in particular to a method for recommending background music, a training method for a matching degree prediction model, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Currently, in reading aloud scenarios, the reader can add suitable background music to their recorded audio to enhance its aesthetic appeal. This background music can be found by the reader themselves or selected from recommendations provided by the system.

[0003] Currently, recommendation systems often recommend background music by finding music that matches the text content of the audio recording; alternatively, they recommend music by analyzing the background music chosen by most readers when reading the same text. However, this approach may result in the system recommending the same background music to different readers for the same text, leading to a limited range of recommendations that fail to meet the individual needs of each reader. Summary of the Invention

[0004] This application provides a method for recommending background music, a method for training a model, an apparatus, and a storage medium. These methods not only make the recommended background music more personalized, but also make the superposition effect of the recommended background music more harmonious when superimposed with the audio of the reading.

[0005] In a first aspect, embodiments of this application provide a method for recommending background music. The method includes: obtaining a first attribute feature of a reading audio, wherein the reading audio is based on a voice recording of a target object, and the first attribute feature includes one or more of a duration feature, a pitch feature, and a sound intensity feature; wherein the duration feature, the pitch feature, and the sound intensity feature are respectively used to indicate the duration, pitch, and sound intensity of the target object.

[0006] The matching degree prediction model, which has been pre-trained, processes the first attribute feature and the second attribute feature of each background music to obtain the matching degree between the reading audio and each background music; wherein, the second attribute feature includes one or more of the rhythm feature, emotion feature, vocal range feature and pitch feature of the background music;

[0007] Based on the matching degree between the reading audio and each of the background music tracks, background music recommendations are made.

[0008] In the method described in the first aspect, the features included in the first attribute feature can reflect the target object's vocal attributes such as duration, pitch, and intensity when recording the reading audio. These vocal attributes are usually related to the target object's emotions and rhythm. The features included in the second attribute feature can reflect the background music's emotions, rhythm, range, and pitch. Therefore, by determining the matching degree between the reading audio and background music and making background music recommendations based on the first and second attribute features, the recommended background music will be more suitable for the target object's recorded reading audio, thus making the recommended background music more personalized. Furthermore, when the recommended background music is superimposed on the reading audio, the superposition effect is more harmonious.

[0009] In one possible implementation, obtaining the first attribute feature of the audio reading includes:

[0010] Based on the reading text corresponding to the reading audio, multiple audio segments are determined to include the reading audio, wherein the reading text is a pronunciation reference text used to record the reading audio, and one audio segment corresponds to one word segment in the reading text;

[0011] Based on the multiple audio segments, one or more of the duration, pitch, and intensity features of the reading audio are obtained.

[0012] This method uses word segmentation as the smallest unit and obtains the first attribute feature of the reading audio through the audio segment corresponding to each word. This approach can capture as many attribute features as possible from the reading audio, thus achieving a more accurate match later based on the first and second attribute features.

[0013] In one possible implementation, determining the multiple audio segments included in the audio reading based on the text corresponding to the audio reading includes:

[0014] The audio recording is subjected to speech recognition to obtain the recognized text;

[0015] Align each word in the identified text with the corresponding word in the read-aloud audio text, using word segmentation as the unit.

[0016] Based on the alignment result between the identified text and the read-aloud text, the multiple audio segments included in the read-aloud audio are determined.

[0017] This method allows you to align the recognized text with the text being read aloud to obtain the audio segments included in the audio reading.

[0018] In one possible implementation, obtaining one or more of the duration, pitch, and intensity features of the read-out audio based on the plurality of audio segments includes:

[0019] Obtain one or more of the duration, pitch, and intensity features of each audio segment;

[0020] Based on one or more of the duration, pitch, and intensity features of each audio segment, determine one or more of the duration, pitch, and intensity features corresponding to each sentence in the read-aloud text;

[0021] One or more of the average values ​​of the duration features, pitch features, and intensity features corresponding to each sentence are determined as one or more of the duration features, pitch features, and intensity features of the reading audio.

[0022] This method first works on a sentence-by-sentence basis, obtaining features for each sentence by analyzing the audio segments corresponding to one or more word segments within each sentence. These features are then combined to obtain the first attribute feature. This approach compresses the features corresponding to all sentences, reducing the amount of data input to the matching prediction model and improving its processing efficiency, as well as the efficiency of background music recommendation.

[0023] In one possible implementation, determining the length features of each sentence in the read-aloud text based on the length features of each audio segment includes:

[0024] The length features of each audio segment are determined as the length features of each word in the read-aloud text;

[0025] Based on the phonetic length characteristics of each word segment, the sentence length, average word segment duration, and word segment duration variance of each sentence are determined.

[0026] Based on the sentence length, average word segmentation duration, and word segmentation duration variance of each sentence, the corresponding phonetic length features of each sentence are obtained.

[0027] This method allows us to obtain the sentence length, average word segmentation duration, and word segmentation duration variance of each sentence, and further obtain the corresponding phonetic length features of each sentence.

[0028] In one possible implementation, determining the pitch features of each sentence in the read-aloud text based on the pitch features of each of the audio segments includes:

[0029] The pitch features of each audio segment are determined as the pitch features of each word in the read-aloud text;

[0030] Based on the pitch characteristics of each word segment, determine the average pitch of each word segment and the pitch variation between word segments in each sentence;

[0031] Based on the average pitch of each word segment and the pitch variation between word segments, the pitch features corresponding to each sentence are obtained.

[0032] This method allows us to obtain the average pitch of each sentence segment and the pitch variation between segments, and further obtain the pitch features corresponding to each sentence.

[0033] In one possible implementation, determining the intensity features of each sentence in the read-aloud text based on the intensity features of each audio segment includes:

[0034] The difference between the intensity features of each audio segment and the intensity features of noise in the audio being read is determined as the intensity features of each word segment in the text being read.

[0035] Based on the intensity characteristics of each word segment, determine the intensity changes between word segments in each sentence;

[0036] Based on the intensity changes between word segments of each sentence, the intensity features corresponding to each sentence are obtained.

[0037] This method allows us to obtain the intensity changes between word segments in each sentence, and further obtain the intensity features corresponding to each sentence.

[0038] In one possible implementation, the first attribute feature further includes the text feature of the text corresponding to the audio being read aloud;

[0039] The pre-trained matching prediction model processes the first attribute feature and the second attribute features of each background music piece to obtain the matching degree between the reading audio and each background music piece, including:

[0040] The matching degree prediction model, which has been pre-trained, obtains a first matching degree based on one or more of the duration feature, the pitch feature, and the intensity feature, and the second attribute feature;

[0041] The matching degree prediction model obtains the second matching degree based on the text features and the second attribute features;

[0042] The matching degree between the reading audio and each of the background music pieces is obtained based on the first matching degree and the second matching degree.

[0043] This method allows for two approaches: firstly, a first matching degree can be obtained based on the duration, pitch, intensity, and second attribute features, indicating the degree of matching between the target object's sound attributes and the background music attributes; secondly, a second matching degree can be obtained based on the text features and second attribute features of the read-aloud text, indicating the degree of matching between the read-aloud text's attributes and the background music attributes; combining the first and second matching degrees yields a more accurate matching degree between the read-aloud audio and the background music.

[0044] Secondly, embodiments of this application provide a training method for a matching degree prediction model, the method comprising: acquiring audio samples of reading aloud, background music samples, and the labeled matching degree of the audio samples of reading aloud and the background music samples;

[0045] Obtain the attribute features of the audio sample and the attribute features of the background music sample;

[0046] The attribute features of the audio sample and the background music sample are input into the initial matching degree prediction model to obtain the predicted matching degree of the audio sample and the background music sample.

[0047] With the goal of narrowing the difference between the labeled matching degree and the predicted matching degree, the initial matching degree prediction model is trained to obtain the trained matching degree prediction model; wherein, the matching degree prediction model is used to predict the matching degree between the input reading audio and each input background music.

[0048] By implementing the method described in the second aspect, a matching degree prediction model can be trained based on the audio samples of reading aloud, the background music samples, and the labeled matching degree of the audio samples of reading aloud and the background music samples.

[0049] In one possible implementation, the method further includes:

[0050] Obtain historical feedback behavior data for the training works, wherein the training works are obtained by superimposing the reading audio samples and the background music samples, and the historical feedback behavior data includes one or more of the following: satisfaction with the background music samples in the training works, the probability of publishing the training works, and the number of interactions with the published training works.

[0051] The historical feedback behavior data is standardized to obtain the labeled matching degree.

[0052] This method allows for the determination of label matching based on historical feedback behavior data of the training samples. Compared to methods that rely on empirical values, using historical feedback behavior data results in more accurate label matching, which in turn allows for the training of a more accurate matching prediction model.

[0053] In one possible implementation, the method further includes:

[0054] Acquire target feedback behavior data for the target work, wherein the target work is obtained by superimposing the reading audio and the target background music, and the target background music is selected from the background music recommendations, which are generated based on the matching degree between each background music and the reading audio;

[0055] Based on the target feedback behavior data, the labeled matching degree between the reading audio and the target background music is obtained;

[0056] Based on the labeled matching degree between the reading audio and the target background music, the matching degree prediction model is optimized.

[0057] This method allows for the optimization of the already trained matching prediction model based on the target feedback behavior data collected after the music recommendation.

[0058] Thirdly, embodiments of this application provide a background music recommendation device, which includes an acquisition module and a processing module;

[0059] The acquisition module is used to: acquire a first attribute feature of the audio recording, wherein the audio recording is based on the speech of the target object, and the first attribute feature includes one or more of the following: duration feature, pitch feature, and intensity feature; wherein the duration feature, the pitch feature, and the intensity feature are used to indicate the duration, pitch, and intensity of the target object, respectively.

[0060] The processing module is used to: process the first attribute feature and the second attribute feature of each background music by a pre-trained matching degree prediction model to obtain the matching degree between the reading audio and each background music; wherein, the second attribute feature includes one or more of the rhythm feature, emotion feature, vocal range feature and pitch feature of the background music;

[0061] Based on the matching degree between the reading audio and each of the background music tracks, background music recommendations are made.

[0062] Fourthly, embodiments of this application provide a training method for a matching degree prediction model, the apparatus including an acquisition module and a processing module;

[0063] The acquisition module is used to: acquire audio samples for reading aloud, background music samples, and the labeled matching degree of the audio samples for reading aloud and the background music samples; acquire the attribute features of the audio samples for reading aloud and the attribute features of the background music samples;

[0064] The processing module is used to: input the attribute features of the reading audio sample and the attribute features of the background music sample into an initial matching degree prediction model to obtain the predicted matching degree of the reading audio sample and the background music sample; train the initial matching degree prediction model with the goal of reducing the difference between the labeled matching degree and the predicted matching degree to obtain a trained matching degree prediction model; wherein, the matching degree prediction model is used to predict the matching degree between the input reading audio and each input background music.

[0065] Fifthly, embodiments of this application provide an electronic device, the electronic device including a memory and a processor; the memory is used to store a computer program, the computer program including program instructions; the processor is used to call the program instructions from the memory, causing the electronic device to perform the method described in any one of the first or second aspects above.

[0066] Sixthly, embodiments of this application provide a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform any one of the methods described in the first or second aspect.

[0067] The beneficial effects of each possible implementation in the third to sixth aspects can be found in the corresponding descriptions in the first or second aspects, and will not be repeated here. Attached Figure Description

[0068] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0069] Figure 1 This is a schematic diagram of the architecture of a background music recommendation system provided in an embodiment of this application;

[0070] Figure 2 This is a flowchart illustrating a method for recommending background music provided in an embodiment of this application;

[0071] Figure 3 This is a schematic diagram of a display interface in a client provided in an embodiment of this application;

[0072] Figure 4 This is a schematic diagram of the display interface in another client provided in an embodiment of this application;

[0073] Figure 5 This is a schematic diagram of the training process of a matching degree prediction model provided in an embodiment of this application;

[0074] Figure 6 This is a schematic diagram of the structure of a communication device provided in an embodiment of this application;

[0075] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0076] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0077] The terms "first" and "second," etc., used in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0078] This application provides a method for recommending background music, which can be applied to... Figure 1 The music recommendation system shown. Figure 1 As shown, the music recommendation system includes at least one server, such as server 10, and at least one client, such as client 11. The server and client can establish a communication connection through a network, which can be a wired network or a wireless network, etc.

[0079] The server 10 can store pre-trained matching degree prediction models and execute the background music recommendation method proposed in this application embodiment. The matching degree prediction model is used to output the matching degree between the reading audio and each background music. For example, when the reading audio contains the voice (or speech) of a target object, the server 10 can obtain the first attribute features of the reading audio. These first attribute features can indicate various sound attributes of the target object, such as the duration, pitch, and intensity features of the reading audio. The duration, pitch, and intensity features can respectively indicate the target object's attributes related to duration, pitch, and intensity. Then, the server 10 can also obtain the second attribute features of each background music. These second attribute features can indicate the attribute category of each background music. For example, the second attribute features of the background music can include rhythm, mood, range, and pitch features. The rhythm, mood, range, and pitch features can respectively indicate the rhythm category, mood category, range category, and pitch category to which the background music belongs. Next, server 10 inputs both the first and second attribute features into the matching prediction model for processing. The matching prediction model outputs the matching degree between the audio reading and each piece of background music based on the correlation between the first and second attribute features. Finally, server 10 recommends background music based on the matching degree between the audio reading and each piece of background music.

[0080] Optionally, server 10 may pre-store the second attribute features of each piece of background music, or server 10 may first determine the rhythm category, mood category, range category, and pitch category of each piece of background music, and then obtain the second attribute features of each piece of background music based on the rhythm category, mood category, range category, and pitch category of each piece of background music.

[0081] Optionally, server 10 can also obtain the text features corresponding to the audio reading text, and then input the text features, duration features, pitch features, intensity features, and the secondary attribute features of the background music into the matching degree prediction model for processing. This method can make the matching degree prediction model obtain a more accurate matching degree.

[0082] Optionally, client 11 may store a pre-trained matching degree prediction model and execute the aforementioned music recommendation method. Alternatively, the aforementioned music recommendation method may be jointly executed by client 11 and server 10, i.e., client 11 executes some steps of the aforementioned music recommendation method, and server 10 executes other steps of the aforementioned music recommendation method. For example, client 11 extracts the first attribute features of the reading audio and sends the extracted first attribute features to server 10; server 10 inputs the stored second attribute features of each piece of music and the received first attribute features into the matching degree prediction model to obtain the matching degree between the reading audio and each piece of music; server 10 obtains a list of recommended music for music recommendation based on the matching degree between the reading audio and each piece of music and sends it to client 11; client 11 displays the list of recommended music to achieve music recommendation for the target object.

[0083] It should be noted that the aforementioned server 10 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The aforementioned client 11 can be a terminal device, which can be a smartphone, tablet, laptop, desktop computer, intelligent voice interaction device, smart home appliance, in-vehicle terminal, etc., but is not limited to these.

[0084] The above provides a brief overview of the music recommendation system provided in the embodiments of this application. The following will combine... Figures 2 to 7 The music recommendation method, matching degree prediction model training method, music recommendation device, electronic device, and computer-readable storage medium provided in the embodiments of this application will be described in detail.

[0085] Please see Figure 2 , Figure 2 This is a flowchart illustrating a music recommendation method provided in an embodiment of this application. The method includes steps S201 to S203, and its execution entity can be a server, a client, or both. The following description uses a server as the execution entity as an example. The server can be one of the aforementioned... Figure 1 Server 10 is described in the text. Among them:

[0086] S201. The server obtains the first attribute feature of the audio recording. The audio recording is based on the speech of the target object. The first attribute feature includes one or more of the following: duration feature, pitch feature, and intensity feature.

[0087] In this embodiment, the audio recording is based on the speech of the target object. For example, the target object can enable the recording function on the client. After enabling the recording function, the acquisition device on the client captures the speech corresponding to the target to obtain the audio recording.

[0088] Optionally, the reading audio can be the raw audio captured by the client, or the audio obtained after preprocessing the raw audio captured by the client. For example, the raw audio can be a combination of the target's speech, ambient noise, and reading prompts (used to prompt the target to start reading). The reading audio can then contain these types of audio, or the combined audio can be filtered to obtain audio containing only the target's speech, and then this audio containing only the target's speech can be used as the reading audio.

[0089] The primary attribute features of a read-aloud audio recording can indicate the vocal attributes of the target audio. For example, the duration feature in the primary attribute features can be used to indicate the duration of the target audio, the pitch feature in the primary attribute features can be used to indicate the pitch of the target audio, and the intensity feature in the primary attribute features can be used to indicate the intensity of the target audio. These vocal attributes are usually related to the target audio's mood and rhythm at the time of recording.

[0090] It should be noted that when the embodiments of this application are applied to specific products or technologies, the audio used in the readings must be licensed or authorized by the target object, and the collection, use and processing of the audio readings must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0091] Optionally, the server can process the audio sent by the client after receiving it, thereby extracting the first attribute feature; or, the server can obtain the first attribute feature extracted and sent by the client.

[0092] The following example, using server-side extraction of the first attribute feature, illustrates the method for extracting the first attribute feature:

[0093] In one possible implementation, the server obtains the first attribute features of the audio to be read aloud, including: determining multiple audio segments included in the audio to be read aloud based on the text to be read aloud corresponding to the audio to be read aloud, wherein one audio segment corresponds to a word segment in the text to be read aloud; and obtaining one or more of the length features, pitch features, and intensity features of the audio to be read aloud based on the multiple audio segments.

[0094] The reading text is a pronunciation reference text used for recording audio readings. For an example, please see... Figure 3 , Figure 3This is a schematic diagram of a client display interface provided in an embodiment of this application. In the display interface, text 301 is the content of poem A (for example, “XXX…”), and this text 301 is also the text read aloud by the target object during recording.

[0095] It should be noted that when sending audio to the server, the client can also send the corresponding audio text, or an identifier corresponding to the audio text, etc., which is not limited in this application. For example, if the server does not store the audio text, the client can send the audio text to the server. Alternatively, if the server pre-stores the correspondence between audio text and identifiers, the client can send the identifier corresponding to the audio text to the server, and the server determines the audio text based on the received identifier.

[0096] In one possible implementation, the server determines multiple audio segments included in the audio reading based on the corresponding reading text, including: performing speech recognition on the audio reading to obtain the recognized text of the audio reading; aligning each word in the recognized text with the corresponding word in the reading text of the audio reading, using word segmentation as the unit; and determining multiple audio segments included in the audio reading based on the alignment result of the recognized text and the reading text.

[0097] Optionally, the recognized text can be obtained by recognizing the audio reading using automatic speech recognition (ASR) technology. For example, the recognized text can be obtained by the server recognizing the audio reading after receiving it, or the recognized text can be obtained by the client recognizing the audio reading and sending it to the server. For instance, the target object can click... Figure 3 The client uses control 302 to start recording the audio reading, so that the client can capture the audio reading. Then the client sends the audio reading to the server, and the server uses ASR technology to perform speech recognition on the audio reading to obtain the recognized text.

[0098] In this embodiment, the server first determines the audio segments corresponding to each segment in the read-aloud text based on the alignment results between each segment in the recognized text and each segment in the read-aloud text. Optionally, a segment can be a character, a word, a phrase, etc.

[0099] For example, suppose a word segment is a character, for Figure 3For the first character "suo" in the read-aloud text, the server first aligns "suo" with the first character "suo" in the recognition text, and then determines the segment corresponding to the first character "suo" in the recognition text in the read-aloud audio as the audio segment corresponding to the first character "suo" in the read-aloud text. For example, when the server performs speech recognition on the read-aloud audio, if the segment from 5000ms to 5500ms in the read-aloud audio is recognized as the first character "suo", then the audio segment corresponding to the first character "suo" in the read-aloud text is the segment from 5000ms to 5500ms in the read-aloud audio. In a similar manner, the server can obtain the corresponding relationship between each audio segment in the read-aloud audio and each word segmentation in the read-aloud text; then, the server can obtain the attribute features of the read-aloud audio according to multiple audio segments, such as the duration feature, pitch feature and intensity feature of the read-aloud audio. To distinguish from other attribute features below, the attribute feature of the read-aloud audio herein can be referred to as the first attribute feature.

[0100] Since the audio segments are in one-to-one correspondence with the word segments in the read-aloud text, more attribute features of the read-aloud audio can be captured as much as possible with word segmentation as the minimum unit, so that a more accurate matching degree can be obtained when the matching degree is predicted according to the first attribute feature and the second attribute feature in the subsequent process.

[0101] Next, how the server obtains the first attribute feature of the read-aloud audio based on the audio segments will be described:

[0102] In a possible implementation, the read-aloud text includes at least one sentence; the server obtaining one or more of the duration feature, pitch feature and intensity feature of the read-aloud audio based on a plurality of audio segments includes: acquiring one or more of the duration feature, pitch feature and intensity feature of each audio segment; determining one or more of the duration feature, pitch feature and intensity feature corresponding to each sentence in the read-aloud text according to one or more of the duration feature, pitch feature and intensity feature of each audio segment; determining one or more of the average duration feature, average pitch feature and average intensity feature corresponding to each sentence as one or more of the duration feature, pitch feature and intensity feature of the read-aloud audio.

[0103] Because in most vocalization scenarios, the changes of speech rhythm, emotion, etc. corresponding to the same sentence are small, while the changes of speech rhythm, emotion, etc. corresponding to different sentences are large. Therefore, in the embodiment of the present application, the server may first obtain features corresponding to each sentence (including, for example, duration feature, pitch feature and intensity feature) in units of sentences, then fuse the features corresponding to all sentences (for example, take the average value of the features corresponding to each sentence) to obtain the first attribute feature of the read-aloud audio, and the first attribute feature is used to characterize the emotion and / or rhythm of the target object when reading the sentences.

[0104] For example, if the text to be read contains N sentences, where N is a positive integer, and the server extracts the duration, pitch, and intensity features for each sentence, forming an M-dimensional vector, then the server can obtain N M-dimensional vectors. By averaging the duration, pitch, and intensity features of all sentences—for example, by summing the vector elements corresponding to the duration features of all sentences and taking the average—the server finally obtains an M-dimensional vector for the first attribute feature. Based on this method, the server can compress the features of all sentences to reduce the amount of data input to the matching degree prediction model, thereby improving the processing efficiency of the matching degree model and the efficiency of background music recommendation.

[0105] Optionally, after obtaining the features corresponding to each sentence, the server can also use other processing methods to obtain the first attribute features, which is not limited in this application. For example, the server can concatenate the features of N sentences to obtain an N*M dimensional vector, or the server can form an N*M dimensional vector matrix from the features of N sentences, and so on.

[0106] In one possible implementation, the length features corresponding to each sentence include vectors corresponding to the sentence length, average word segment duration, and word segment duration variance of each sentence. The server determines the length features corresponding to each sentence in the reading text based on the length features of each audio segment, including: determining the length features of each audio segment as the length features of each word segment in the reading text; determining the sentence length, average word segment duration, and word segment duration variance of each sentence based on the length features of each word segment; and obtaining the length features corresponding to each sentence based on the sentence length, average word segment duration, and word segment duration variance of each sentence.

[0107] Among them, the duration of each audio segment can indicate the duration of each audio segment, the duration of each word segment can indicate the duration of the target object when uttering each word, and the sentence length of each sentence is the duration of the target object when uttering each sentence.

[0108] In this embodiment of the application, the server can first determine one or more segments included in each sentence based on the text being read aloud. Then, the server determines the sentence length of each sentence by summing the duration of the phonetic length features of all segments included in each sentence. The average duration of each segment is determined by the ratio of the sentence length of each sentence to the number of segments included in each sentence. Based on the average duration of each segment and the phonetic length features of each segment included in each sentence, the average variance of each sentence is determined.

[0109] Specifically, the sentence length, average word duration, and word duration variance of each sentence are related to the target audience's emotions and / or rhythm when reading each sentence. For example, if the target audience reads a sentence with deep and enthusiastic emotion, their speaking speed will be slower, resulting in a larger average word duration and a longer sentence length. If the target audience reads a sentence with tension and urgency, their speaking speed will be faster, resulting in a smaller average word duration and a shorter sentence length. If the target audience reads a sentence with a pronounced rhythm or many emotional changes, the word duration variance of that sentence will be larger.

[0110] In one possible implementation, the pitch features corresponding to each sentence include the average pitch of each sentence segment and the vector corresponding to the pitch changes between segments. The server determines the pitch features corresponding to each sentence in the reading text based on the pitch features of each audio segment, including: determining the pitch features of each audio segment as the pitch features of each segment in the reading text; determining the average pitch of each sentence segment and the pitch changes between segments based on the pitch features of each segment; and obtaining the pitch features corresponding to each sentence based on the average pitch of each sentence segment and the pitch changes between segments.

[0111] In this embodiment, the server can extract the pitch features of each audio segment by frame using a pitch extraction algorithm. Since there is a one-to-one correspondence between audio segments and word segments, the determined pitch features of each audio segment are also the pitch features of each word segment in the read-aloud text.

[0112] For example, an audio segment may include one or more audio frames. The server first uses the YIN algorithm or PYIN algorithm to extract the pitch features corresponding to each audio frame. Then, the server determines the average pitch features of each audio segment as the average pitch features of all audio frames included in each audio segment.

[0113] Optionally, the server can first filter out invalid fundamental frequencies in each audio frame, and then use a pitch extraction algorithm to extract the pitch features corresponding to the filtered audio frames. For example, the invalid fundamental frequency is 0, or the invalid fundamental frequency is a frequency value exceeding a preset range. Based on this method, the obtained pitch features can be more accurate.

[0114] Then, the server determines the average pitch of each word in each sentence based on the pitch features of each word in the text and the words included in each sentence, and determines the pitch variation between words in each sentence based on the changes in pitch features between adjacent words in each sentence.

[0115] Specifically, the average pitch of each word segment and the pitch variation between word segments in each sentence are related to the target audience's emotions and / or rhythm when reading each sentence. For example, if the target audience reads a sentence with passionate emotion, their tone will be high and have obvious intonation, resulting in a larger average pitch of each word segment and a larger pitch variation between word segments. Conversely, if the target audience reads a sentence with a somber and low mood, their tone will be low and have less obvious intonation, resulting in a smaller average pitch of each word segment and a smaller pitch variation between word segments.

[0116] In one possible implementation, the intensity features corresponding to each sentence include vectors corresponding to the intensity changes between word segments of each sentence; the server determines the intensity features corresponding to each sentence in the reading text based on the intensity features of each audio segment, including: determining the difference between the intensity features of each audio segment and the noise intensity features in the reading audio as the intensity features of each word segment in the reading text; determining the intensity changes between word segments of each sentence based on the intensity features of each word segment; and obtaining the intensity features corresponding to each sentence based on the intensity changes between word segments of each sentence.

[0117] The noise intensity feature can indicate whether it is the intensity of steady-state background noise or non-steady-state background noise. For example, when the noise is steady-state background noise, the server can determine the noise intensity feature based on one or more audio segments corresponding to the text without a fundamental frequency in the text being read (for example, multiple audio segments with a total duration of 5 seconds).

[0118] In this embodiment, the server can first use a sound intensity extraction algorithm to determine the sound intensity features and noise sound intensity features of each audio segment. For example, this sound intensity extraction algorithm could be the root mean square error (RMSE) algorithm, the European Broadcasting Union Recommendation 128 (EBU R 128) algorithm, etc. Then, the server uses the difference between the sound intensity features of each audio segment and the noise sound intensity features as the sound intensity features of each word in the read-aloud text, and determines the changes in sound intensity features between adjacent words in each sentence as the inter-word sound intensity changes in each sentence.

[0119] Specifically, the intensity variations between word segments in each sentence are related to the target audience's emotions and / or rhythm when reading each sentence. For example, if the target audience reads a sentence with great passion and significant rhythmic variations, the intensity variations between word segments in that sentence will be relatively large.

[0120] It should be noted that the above-mentioned representations and contents of duration, pitch, and intensity features are merely examples. In specific implementations, there may be other representations, such as tables or sets. Furthermore, multiple pronunciation units can be considered to obtain the above features. For example, when extracting duration features, the duration features corresponding to multiple units such as characters, words, and sentences can be considered separately.

[0121] Furthermore, the aforementioned first attribute feature may include one or more of the duration, pitch, and intensity features of the audio being read aloud, or the first attribute feature may also include other sound attribute features of the audio being read aloud, such as features used to characterize timbre (e.g., mel-scale frequency cepstral coefficients (MFCC), spectral roll-off points, etc.). If each feature is represented by a vector, different vector processing methods can be used to process each feature to obtain the first attribute feature, such as concatenating the beginning and end of each feature, forming a vector matrix, or superimposing corresponding vector elements, etc.

[0122] S202. The server processes the first attribute features and the second attribute features of each background music by a pre-trained matching degree prediction model to obtain the matching degree between the reading audio and each background music; wherein, the second attribute features include one or more of the rhythm features, emotion features, vocal range features and pitch features of the background music.

[0123] The rhythmic, emotional, range, and pitch characteristics of the background music are used to indicate the rhythmic, emotional, range, and pitch categories to which the background music belongs, respectively. These rhythmic, emotional, range, and pitch characteristics can be represented as vectors, tables, or sets, etc., and this application does not limit their representation.

[0124] For example, rhythm features, mood features, range features, and pitch features can be one-hot vectors corresponding to rhythm categories, mood categories, range categories, and pitch categories. If a rhythm category, mood category, range category, or pitch category includes I subcategories, then the rhythm features, mood features, range features, or pitch features can be represented as an I-dimensional vector. When the rhythm category, mood category, range category, or pitch category to which the background music belongs is the target subcategory among the I subcategories, the vector elements corresponding to the target subcategory in the I-dimensional vector can take the value 1, and the vector elements corresponding to the other subcategories in the I-dimensional vector can take the value 0.

[0125] For example, the emotional category of background music is divided into four subcategories: "high-spirited," "low-spirited," "joyful," and "sad." The rhythm category is divided into two subcategories: "medium tempo" and "fast tempo." The vocal range category is divided into three subcategories: "high," "low," and "middle." The pitch category is divided into two subcategories: "C major" and "G major." When a piece of background music belongs to the emotional category of "joyful," the rhythm category of "fast," the vocal range category of "middle," and the pitch category of "C major," its rhythmic characteristics, emotional characteristics, vocal range characteristics, and pitch characteristics are "0010," "01," "001," and "10," respectively.

[0126] Optionally, the server can first determine the rhythm category, mood category, range category, and pitch category of the background music, and then determine the attribute features of the background music based on these categories. To distinguish it from the first attribute feature mentioned above, the attribute features of the background music here can be called the second attribute feature (such as a one-hot vector). Alternatively, the server can pre-store the correspondence between the second attribute features of the background music and the identifier of the background music, and the server can directly determine the second attribute feature of a certain background music based on its identifier.

[0127] In this embodiment, since the first attribute feature is related to the target object's emotions, rhythm, etc., when recording the audio, and the second attribute features of each background music can reflect the emotions, rhythm, range, pitch, etc., of each background music, when the first and second attribute features are input into the pre-trained matching prediction model, the matching prediction model can obtain the matching degree between the audio and each background music based on the degree of correlation between the first attribute feature and the second attribute features of each background music in terms of emotions, rhythm, etc. For example, the more similar the target object's emotions, rhythm, etc., are to the emotions, rhythm, range, pitch, etc., of the background music, the stronger the correlation between the first and second attribute features, and the higher the matching degree between the audio and the background music; conversely, the more different the target object's emotions, rhythm, range, pitch, etc., are from the emotions, rhythm, range, pitch, etc., of the background music, the weaker the correlation between the first and second attribute features, and the lower the matching degree between the audio and the background music.

[0128] For example, the matching degree prediction model described above can be a supervised regression model, such as a support vector regression (SVR) model, a decision tree, or a deep neural network (DNN) model. This application does not limit this. Furthermore, the first attribute feature and the second attribute feature can be input into the matching degree prediction model in any representation. For example, when both the first attribute feature and the second attribute feature are vector representations, they can be concatenated and then input into the matching degree prediction model. This application does not limit this.

[0129] Optionally, the first attribute feature mentioned above may also include the text features of the text corresponding to the audio recording. These text features may be pre-stored in the server or obtained by the server through real-time processing of the text.

[0130] In one possible implementation, the server processes the first attribute features and the second attribute features of each background music based on a matching degree prediction model to obtain the matching degree between the reading audio and each background music. Specifically, this includes: obtaining a first matching degree based on one or more of the duration, pitch and intensity features of the reading audio and the second attribute features; obtaining a second matching degree based on the text features and the second attribute features; and obtaining the matching degree between the reading audio and each background music based on the first matching degree and the second matching degree.

[0131] Among them, the text features of the text to be read can indicate the attribute information of the text to be read. For example, the attribute information of the text to be read can include the author, category, content ideas and other attributes of the text to be read. The attribute information of the text to be read is related to the emotion, scene and other aspects of the text to be read.

[0132] In this implementation, the first matching degree indicates the degree of matching between the attribute information of the audio being read (reflecting attributes such as the sound performance of the target object) and the attribute information indicated by the second attribute feature; the second matching degree indicates the degree of matching between the attribute information of the text being read and the attribute information indicated by the second attribute feature. Therefore, by combining the first and second matching degrees, the matching degree between the audio being read and each piece of background music is obtained. This matching degree reflects whether the audio being read matches each piece of background music from both sound and text perspectives, resulting in higher accuracy.

[0133] It should be noted that the matching degree between the reading audio and each background music can be the average of the first matching degree and the second matching degree, the weighted average, etc., and this application does not limit it in this way.

[0134] S203. The server recommends background music based on the matching degree between the reading audio and each background music.

[0135] In one possible implementation, the server can select a preset number of background music tracks from each track in descending order of matching degree to obtain a recommended track list. The server then sends the recommended track list to the client, so that the client can display each track in the recommended track list in descending order of matching degree.

[0136] For example, this preset number can be the maximum number displayed on a single screen or the number exceeding a preset threshold. For instance, if the maximum number displayed on a single screen is 7, and the top seven most matched background music tracks are songs A through G, then the client can display, as shown below. Figure 4 The interface shown displays the names of songs A through G, along with download controls, arranged from highest to lowest matching accuracy. When the target user clicks the download control 401 for song A, the client can download song A and generate a composite image of song A and its audio recording for the target user to listen to.

[0137] Optionally, the server can also send the matching degree of the reading audio and each background music to the client, so that the client can automatically generate and display a list of recommended background music based on the matching degree of the reading audio and each background music.

[0138] based on Figure 2 In the described embodiment, the features included in the first attribute feature can reflect the target object's vocal attributes such as duration, pitch, and intensity when recording the audio reading. These vocal attributes are usually related to the target object's emotions and rhythm. The features included in the second attribute feature can reflect the background music's emotions, rhythm, range, and pitch. Therefore, by determining the matching degree between the audio reading and the background music and recommending background music based on the first and second attribute features, the recommended background music will be more suitable for the target object's recorded audio reading, making the recommended background music more personalized. Furthermore, when the recommended background music is superimposed on the audio reading, the superposition effect is more harmonious, effectively improving the listening experience of the superimposed background music and audio reading.

[0139] The above embodiments describe the music recommendation method in this application. The training and optimization methods for the matching degree prediction model in the music recommendation method are explained below:

[0140] In one possible implementation, the server obtains audio samples for reading aloud, background music samples, and the labeled matching degree between the audio samples and background music samples; it obtains the attribute features of the audio samples for reading aloud and the attribute features of the background music samples; it inputs the attribute features of the audio samples for reading aloud and the attribute features of the background music samples into an initial matching degree prediction model to obtain the predicted matching degree between the audio samples for reading aloud and the background music samples; it trains the initial matching degree prediction model with the goal of reducing the difference between the labeled matching degree and the predicted matching degree to obtain a trained matching degree prediction model; the matching degree prediction model is used to predict the matching degree between the input audio samples for reading aloud and each input background music sample.

[0141] The audio samples can be audio recordings of different objects reading aloud within a historical period, while the background music samples are the music chosen by these objects before they release the audio samples. The background music samples can be selected manually by these objects from a music library, or they can be selected based on a set of recommended music displayed by the client.

[0142] The label matching degree is used to indicate the degree of matching between the audio sample and the background music sample. For example, the label matching degree can be any value in the range [0, 1], and the larger the label matching degree value, the more matched the audio sample and the background music sample are.

[0143] Optionally, the label matching degree can be obtained based on empirical values, transformed from relevant behavioral data, or based on the matching degree between the pronunciation reference text of the audio sample and the background music sample. For example, for a training work composed of an overlay of audio samples and background music samples, the server can obtain historical feedback behavior data for the training work, standardize the historical feedback behavior data, and obtain the label matching degree. This historical feedback behavior data includes one or more of the following: satisfaction with the background music sample in the training work (for example, through a pop-up window, the recipient's rating of the background music sample's satisfaction), the probability of publishing the training work, and the number of interactions with the published training work (for example, the number of interactions can include the number of likes, listens, shares, etc.). Optionally, the higher the satisfaction with the background music sample in the training work, the higher the probability of publishing the training work, or the more interactions with the published training work, the larger the label matching degree value.

[0144] For example, for a reading audio sample, the server can obtain the pronunciation reference text corresponding to the reading audio sample, then analyze the pronunciation reference text to obtain the matching degree between the pronunciation reference text and the background music sample, and use the matching degree between the pronunciation reference text and the background music sample as the labeled matching degree between the reading audio sample and the background music sample.

[0145] Optionally, the number of audio samples and background music samples in this application can be multiple, and the multiple audio samples can cover different voice attributes, different reading styles, etc., while the multiple background music samples can cover different background music attributes. In this way, the samples can be more diverse, which helps to avoid overfitting during the training process of the matching degree prediction model, resulting in better training performance of the matching degree prediction model and enabling the trained matching degree prediction model to output more accurate matching degrees.

[0146] by Figure 5 For example, the process of obtaining a matching degree prediction model based on audio samples and background music samples will be explained: Figure 5 As shown, the server first acquires the attribute features of the audio sample being read aloud, such as the duration, pitch, and intensity features; and it also acquires the attribute features of the background music sample, such as rhythm, mood, range, and pitch features. If both the attribute features of the audio sample and the background music sample are vector representations, the server can concatenate them to obtain an audio-background music representation vector. Next, the server inputs this representation vector into the initial matching degree prediction model to obtain the predicted matching degree. Then, by fitting the predicted matching degree to the labeled matching degree, the server obtains the trained initial matching degree prediction model, which is the matching degree prediction model. This model can be used to obtain the matching degree between the audio sample and each piece of background music. For example, the server uses the mean squared error (MSE) to... The loss function (MSE) is constructed to characterize the difference between the predicted matching degree and the labeled matching degree. The parameters in the initial matching degree prediction model are updated through the loss function and the backpropagation algorithm. When the value of the loss function is less than the preset threshold, the initial matching degree prediction model converges (equivalent to the initial matching degree prediction model being trained and the parameters in the initial matching degree prediction model being determined). The converged initial matching degree prediction model is the matching degree prediction model used in the music recommendation method of this application.

[0147] In one possible implementation, the server can also optimize the matching degree prediction model during the process of using the matching degree prediction model: obtain target feedback behavior data for the target work, which is obtained by superimposing the reading audio and the target background music; obtain the labeled matching degree between the reading audio and the target background music based on the target feedback behavior data; and optimize the matching degree prediction model based on the labeled matching degree between the reading audio and the target background music.

[0148] In this embodiment, the target background music is the music selected by the target object based on background music recommendations. For example, the target background music can be the music with the highest matching degree selected by the target object from the recommended background music list. Since the target object may publish a target work obtained by superimposing the target background music and the reading audio after selecting the target background music, or if the target object publishes the target work, other objects can like, share, comment, etc. on the target work, this application can collect this data to obtain the labeled matching degree between the target background music and the reading audio. The labeled matching degree between the target background music and the reading audio can be used to further optimize the matching degree prediction model, and the optimized matching degree prediction model can be used in subsequent background music recommendation scenarios to improve the accuracy of background music recommendation.

[0149] Please see Figure 6 , Figure 6 This is a schematic diagram of a communication device provided in an embodiment of this application. The communication device includes an acquisition module 601 and a processing module 602. Wherein:

[0150] In one embodiment, the communication device can be used to execute a music recommendation method, and the communication state used to execute the music recommendation method can be referred to as the music recommendation device, wherein:

[0151] The acquisition module 601 is used to acquire a first attribute feature of the reading audio, the reading audio being recorded based on the speech of the target object, the first attribute feature including one or more of the following: duration feature, pitch feature, and intensity feature; the duration feature, the pitch feature, and the intensity feature are respectively used to indicate the duration, pitch, and intensity of the target object;

[0152] The processing module 602 is used to process the first attribute feature and the second attribute feature of each background music by a pre-trained matching degree prediction model to obtain the matching degree between the reading audio and each background music; wherein, the second attribute feature includes one or more of the rhythm feature, emotion feature, vocal range feature and pitch feature of the background music.

[0153] The processing module 602 is used to recommend background music based on the matching degree between the reading audio and each background music.

[0154] In one possible implementation, when acquiring the first attribute feature of the audio reading, the acquisition module 601 is specifically used for:

[0155] The processing module 602 determines multiple audio segments included in the reading audio based on the reading text corresponding to the reading audio, wherein the reading text is a pronunciation reference text used to record the reading audio, and one audio segment corresponds to one word segment in the reading text;

[0156] The processing module 602 obtains one or more of the duration, pitch, and intensity features of the reading audio based on the multiple audio segments.

[0157] In one possible implementation, when the acquisition module 601 determines the multiple audio segments included in the audio reading based on the text corresponding to the audio reading, it is specifically used for:

[0158] The processing module 602 performs speech recognition on the audio reading to obtain the recognized text.

[0159] The processing module 602 aligns each word in the identified text with the corresponding word in the read-aloud audio text, using word segmentation as the unit.

[0160] The processing module 602 determines the multiple audio segments included in the reading audio based on the alignment result of the recognized text and the reading text.

[0161] In one possible implementation, when the processing module 602 obtains one or more of the duration, pitch, and intensity features of the reading audio based on the plurality of audio segments, it is specifically used for:

[0162] Obtain one or more of the duration, pitch, and intensity features of each audio segment;

[0163] Based on one or more of the duration, pitch, and intensity features of each audio segment, determine one or more of the duration, pitch, and intensity features corresponding to each sentence in the read-aloud text;

[0164] One or more of the average values ​​of the duration features, pitch features, and intensity features corresponding to each sentence are determined as one or more of the duration features, pitch features, and intensity features of the reading audio.

[0165] In one possible implementation, the processing module 602, when determining the length features of each sentence in the read-aloud text based on the length features of each audio segment, is specifically used for:

[0166] The length features of each audio segment are determined as the length features of each word in the read-aloud text;

[0167] Based on the phonetic length characteristics of each word segment, the sentence length, average word segment duration, and word segment duration variance of each sentence are determined.

[0168] Based on the sentence length, average word segmentation duration, and word segmentation duration variance of each sentence, the corresponding phonetic length features of each sentence are obtained.

[0169] In one possible implementation, the processing module 602, when determining the pitch features corresponding to each sentence in the read-aloud text based on the pitch features of each audio segment, is specifically used for:

[0170] The pitch features of each audio segment are determined as the pitch features of each word in the read-aloud text;

[0171] Based on the pitch characteristics of each word segment, determine the average pitch of each word segment and the pitch variation between word segments in each sentence;

[0172] Based on the average pitch of each word segment and the pitch variation between word segments, the pitch features corresponding to each sentence are obtained.

[0173] In one possible implementation, the processing module 602, when determining the intensity features of each sentence in the read-aloud text based on the intensity features of each audio segment, is specifically used for:

[0174] The difference between the intensity features of each audio segment and the intensity features of noise in the audio being read is determined as the intensity features of each word segment in the text being read.

[0175] Based on the intensity characteristics of each word segment, determine the intensity changes between word segments in each sentence;

[0176] Based on the intensity changes between word segments of each sentence, the intensity features corresponding to each sentence are obtained.

[0177] In one possible implementation, the first attribute feature further includes the text feature of the text corresponding to the audio recording; when the processing module 602 processes the first attribute feature and the second attribute feature of each background music by the pre-trained matching degree prediction model to obtain the matching degree between the audio recording and each background music, it is specifically used for:

[0178] The matching degree prediction model, which has been pre-trained, obtains a first matching degree based on one or more of the duration feature, the pitch feature, and the intensity feature, and the second attribute feature;

[0179] The matching degree prediction model obtains the second matching degree based on the text features and the second attribute features;

[0180] The matching degree between the reading audio and each of the background music pieces is obtained based on the first matching degree and the second matching degree.

[0181] In one embodiment, the communication device can be used to execute a training method for a matching degree prediction model, and the device for executing the training method can be referred to as a training device for the matching degree prediction model, wherein:

[0182] The acquisition module 601 is used to: acquire audio samples for reading aloud, background music samples, and the label matching degree of the audio samples for reading aloud and the background music samples; acquire the attribute features of the audio samples for reading aloud and the attribute features of the background music samples.

[0183] The processing module 602 is used to input the attribute features of the reading audio sample and the attribute features of the background music sample into an initial matching degree prediction model to obtain the predicted matching degree of the reading audio sample and the background music sample; with the goal of reducing the difference between the labeled matching degree and the predicted matching degree, the initial matching degree prediction model is trained to obtain a trained matching degree prediction model; wherein, the matching degree prediction model is used to predict the matching degree between the input reading audio and each input background music.

[0184] In one possible implementation, the acquisition module 601 is further configured to acquire historical feedback behavior data for the training work, wherein the training work is obtained by superimposing the reading audio sample and the background music sample, and the historical feedback behavior data includes one or more of the following: satisfaction with the background music sample in the training work, the probability of publishing the training work, and the number of interactions with the published training work.

[0185] The processing module 602 is further configured to standardize the historical feedback behavior data to obtain the label matching degree.

[0186] In one possible implementation, the acquisition module 601 is further configured to acquire target feedback behavior data for the target work, wherein the target work is obtained by superimposing the reading audio and the target background music, and the target background music is selected from the background music recommendation, which is generated based on the matching degree between each background music and the reading audio.

[0187] The processing module 602 is further configured to: obtain the labeled matching degree of the reading audio and the target background music based on the target feedback behavior data; and optimize the matching degree prediction model based on the labeled matching degree of the reading audio and the target background music.

[0188] It should be noted that the functions of each module of the communication device in the embodiments of this application can be specifically implemented according to the methods in the above method embodiments. The specific implementation process and beneficial effects can be referred to the relevant descriptions in the above method embodiments, and will not be repeated here.

[0189] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device can be a server or client as described in the above method embodiments, and it may include one or more processors 701 and a memory 702. Optionally, the electronic device may also include a transceiver 703. The processor 701, memory 702, and transceiver 703 can be connected via a bus 704. The memory 702 is used to store a computer program, which includes program instructions.

[0190] In one embodiment, processor 701 performs the following operations by running program instructions stored in memory 702:

[0191] A first attribute feature is obtained from the audio recording of the reading aloud, which is based on the speech of the target object. The first attribute feature includes one or more of the following: duration feature, pitch feature, and intensity feature. The duration feature, pitch feature, and intensity feature are used to indicate the duration, pitch, and intensity of the target object, respectively.

[0192] The matching degree prediction model, which has been pre-trained, processes the first attribute feature and the second attribute feature of each background music to obtain the matching degree between the reading audio and each background music; wherein, the second attribute feature includes one or more of the rhythm feature, emotion feature, vocal range feature and pitch feature of the background music;

[0193] Based on the matching degree between the reading audio and each of the background music tracks, background music recommendations are made.

[0194] In one possible implementation, when the processor 701 acquires the first attribute feature of the audio being read aloud, it specifically performs the following operations:

[0195] Based on the reading text corresponding to the reading audio, multiple audio segments are determined to include the reading audio, wherein the reading text is a pronunciation reference text used to record the reading audio, and one audio segment corresponds to one word segment in the reading text;

[0196] Based on the multiple audio segments, one or more of the duration, pitch, and intensity features of the reading audio are obtained.

[0197] In one possible implementation, when the processor 701 determines the multiple audio segments included in the audio reading based on the text corresponding to the audio reading, it specifically performs the following operations:

[0198] The audio recording is subjected to speech recognition to obtain the recognized text;

[0199] Align each word in the identified text with the corresponding word in the read-aloud audio text, using word segmentation as the unit.

[0200] Based on the alignment result between the identified text and the read-aloud text, the multiple audio segments included in the read-aloud audio are determined.

[0201] In one possible implementation, when the processor 701 obtains one or more of the duration, pitch, and intensity features of the reading audio based on the plurality of audio segments, it specifically performs the following operations:

[0202] Obtain one or more of the duration, pitch, and intensity features of each audio segment;

[0203] Based on one or more of the duration, pitch, and intensity features of each audio segment, determine one or more of the duration, pitch, and intensity features corresponding to each sentence in the read-aloud text;

[0204] One or more of the average values ​​of the duration features, pitch features, and intensity features corresponding to each sentence are determined as one or more of the duration features, pitch features, and intensity features of the reading audio.

[0205] In one possible implementation, when the processor 701 determines the length features of each sentence in the read-aloud text based on the length features of each audio segment, it specifically performs the following operations:

[0206] The length features of each audio segment are determined as the length features of each word in the read-aloud text;

[0207] Based on the phonetic length characteristics of each word segment, the sentence length, average word segment duration, and word segment duration variance of each sentence are determined.

[0208] Based on the sentence length, average word segmentation duration, and word segmentation duration variance of each sentence, the corresponding phonetic length features of each sentence are obtained.

[0209] In one possible implementation, when the processor 701 determines the pitch features corresponding to each sentence in the read-aloud text based on the pitch features of each audio segment, it specifically performs the following operations:

[0210] The pitch features of each audio segment are determined as the pitch features of each word in the read-aloud text;

[0211] Based on the pitch characteristics of each word segment, determine the average pitch of each word segment and the pitch variation between word segments in each sentence;

[0212] Based on the average pitch of each word segment and the pitch variation between word segments, the pitch features corresponding to each sentence are obtained.

[0213] In one possible implementation, when the processor 701 determines the intensity features corresponding to each sentence in the read-aloud text based on the intensity features of each audio segment, it specifically performs the following operations:

[0214] The difference between the intensity features of each audio segment and the intensity features of noise in the audio being read is determined as the intensity features of each word segment in the text being read.

[0215] Based on the intensity characteristics of each word segment, determine the intensity changes between word segments in each sentence;

[0216] Based on the intensity changes between word segments of each sentence, the intensity features corresponding to each sentence are obtained.

[0217] In one possible implementation, the first attribute feature further includes the text feature of the text corresponding to the audio being read aloud;

[0218] When the processor 701 processes the first attribute feature and the second attribute features of each background music by the pre-trained matching degree prediction model to obtain the matching degree between the reading audio and each background music, it specifically performs the following operations:

[0219] The matching degree prediction model, which has been pre-trained, obtains a first matching degree based on one or more of the duration feature, the pitch feature, and the intensity feature, and the second attribute feature;

[0220] The matching degree prediction model obtains the second matching degree based on the text features and the second attribute features;

[0221] The matching degree between the reading audio and each of the background music pieces is obtained based on the first matching degree and the second matching degree.

[0222] In another embodiment, processor 701 performs the following operations by executing program instructions stored in memory 702:

[0223] Obtain audio samples for reading aloud, background music samples, and the labeled matching degree of the audio samples for reading aloud and the background music samples;

[0224] Obtain the attribute features of the audio sample and the attribute features of the background music sample;

[0225] The attribute features of the audio sample and the background music sample are input into the initial matching degree prediction model to obtain the predicted matching degree of the audio sample and the background music sample.

[0226] With the goal of narrowing the difference between the labeled matching degree and the predicted matching degree, the initial matching degree prediction model is trained to obtain the trained matching degree prediction model; wherein, the matching degree prediction model is used to predict the matching degree between the input reading audio and each input background music.

[0227] In one possible implementation, the processor 701 further performs the following operations:

[0228] Obtain historical feedback behavior data for the training works, wherein the training works are obtained by superimposing the reading audio samples and the background music samples, and the historical feedback behavior data includes one or more of the following: satisfaction with the background music samples in the training works, the probability of publishing the training works, and the number of interactions with the published training works.

[0229] The historical feedback behavior data is standardized to obtain the labeled matching degree.

[0230] In one possible implementation, the processor 701 further performs the following operations:

[0231] Acquire target feedback behavior data for the target work, wherein the target work is obtained by superimposing the reading audio and the target background music, and the target background music is selected from the background music recommendations, which are generated based on the matching degree between each background music and the reading audio;

[0232] Based on the target feedback behavior data, the labeled matching degree between the reading audio and the target background music is obtained;

[0233] Based on the labeled matching degree between the reading audio and the target background music, the matching degree prediction model is optimized.

[0234] It should be understood that in some feasible implementations, the processor 701 described above may be a central processing unit (CPU), which may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor. The memory 702 may include read-only memory and random access memory, and provides instructions and data to the processor 501. A portion of the memory 702 may also include non-volatile random access memory. For example, the memory 702 may also store device type information.

[0235] In specific implementation, the implementation process and beneficial effects of the above-mentioned electronic device can be found in the specific content of the above method embodiments, and will not be repeated here.

[0236] This application also provides a computer-readable storage medium storing a computer program executed by the aforementioned music recommendation device. This computer program includes program instructions, which, when executed by a processor, enable the execution of the content described in the method embodiments above. Therefore, further details will not be repeated here. Additionally, the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments of this application, please refer to the description of the method embodiments of this application. As an example, the program instructions can be deployed on an electronic device, executed on multiple electronic devices located in one location, or executed on multiple electronic devices distributed across multiple locations and interconnected via a communication network. These multiple electronic devices distributed across multiple locations and interconnected via a communication network can constitute a blockchain system.

[0237] According to one aspect of this application, a computer program product is also provided, comprising a computer program stored in a computer-readable storage medium, including program instructions. A processor of an electronic device reads the program instructions from the computer-readable storage medium and executes the program instructions, enabling the electronic device to perform the content described in the above method embodiments; therefore, further details are omitted here.

[0238] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0239] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for recommending background music, characterized in that, The method includes: Based on the reading text corresponding to the reading audio, the multiple audio segments included in the reading audio are determined, and the text features corresponding to the reading text are obtained. The reading audio is recorded based on the speech of the target object, and the reading text is a pronunciation reference text used to record the reading audio. One audio segment corresponds to one word segment in the reading text. The text features are used to indicate the attribute information of the reading text, including the author and category of the reading text. Obtain one or more of the duration, pitch, and intensity features of each audio segment; The length features of each audio segment are determined as the length features of each word in the read-aloud text; Based on the phonetic length characteristics of each word segment, the sentence length, average word segment duration, and word segment duration variance of each sentence are determined. Based on the sentence length, average word segmentation duration, and word segmentation duration variance of each sentence, the phonetic length features corresponding to each sentence are obtained; The length features of each sentence are concatenated to obtain the length features of the audio recording. The pitch features of each audio segment are determined as the pitch features of each word in the read-aloud text; Based on the pitch characteristics of each word segment, determine the average pitch of each word segment and the pitch variation between word segments in each sentence; Based on the average pitch of each word segment and the pitch variation between word segments, the pitch features corresponding to each sentence are obtained; The pitch features of each sentence are concatenated to obtain the pitch features of the reading audio. Determine the intensity features and noise intensity features of each audio segment, and determine the difference between the intensity features of each audio segment and the noise intensity features in the reading audio as the intensity features of each word segment in the reading text; Based on the intensity characteristics of each word segment, determine the intensity changes between word segments in each sentence; Based on the changes in sound intensity between word segments of each sentence, the sound intensity features corresponding to each sentence are obtained; The intensity features corresponding to each sentence are concatenated to obtain the intensity features of the reading audio. The matching degree prediction model, which has been pre-trained, obtains a first matching degree based on one or more of the duration feature, pitch feature, and intensity feature and the second attribute feature of each background music; wherein, the second attribute feature includes one or more of the rhythm feature, mood feature, range feature, and pitch feature of the background music; The matching degree prediction model obtains the second matching degree based on the text features and the second attribute features; Based on the first matching degree and the second matching degree, the matching degree between the reading audio and each of the background music pieces is obtained; Based on the matching degree between the reading audio and each of the background music tracks, background music recommendations are made.

2. The method according to claim 1, characterized in that, The step of determining multiple audio segments included in the audio reading based on the corresponding text reading the audio reading includes: The audio recording is subjected to speech recognition to obtain the recognized text; Align each word in the identified text with the corresponding word in the read-aloud audio text, using word segmentation as the unit. Based on the alignment result between the identified text and the read-aloud text, the multiple audio segments included in the read-aloud audio are determined.

3. The method according to claim 1 or 2, characterized in that, Obtaining one or more of the duration, pitch, and intensity features of the read-out audio, including: One or more of the average values ​​of the duration features, pitch features, and intensity features corresponding to each sentence are determined as one or more of the duration features, pitch features, and intensity features of the reading audio.

4. A training method for a matching degree prediction model, characterized in that, The method includes: Obtain audio samples for reading aloud, background music samples, and the labeled matching degree of the audio samples for reading aloud and the background music samples; Based on the reading text corresponding to the audio sample, the text features corresponding to the reading text are obtained, and the multiple audio segments included in the audio sample are determined. Each audio segment corresponds to a word segment in the reading text. The text features are used to indicate the attribute information of the reading text, including the author and category of the reading text. Obtain one or more of the duration, pitch, and intensity features of each audio segment; The length features of each audio segment are determined as the length features of each word in the read-aloud text; Based on the phonetic length characteristics of each word segment, the sentence length, average word segment duration, and word segment duration variance of each sentence are determined. Based on the sentence length, average word segmentation duration, and word segmentation duration variance of each sentence, the phonetic length features corresponding to each sentence are obtained; The length features of each sentence are concatenated to obtain the length features of the audio sample. The pitch features of each audio segment are determined as the pitch features of each word in the read-aloud text; Based on the pitch characteristics of each word segment, determine the average pitch of each word segment and the pitch variation between word segments in each sentence; Based on the average pitch of each word segment and the pitch variation between word segments, the pitch features corresponding to each sentence are obtained; The pitch features corresponding to each sentence are concatenated to obtain the pitch features of the audio sample being read aloud; Determine the intensity features and noise intensity features of each audio segment, and determine the difference between the intensity features of each audio segment and the noise intensity features in the reading audio as the intensity features of each word segment in the reading text; Based on the intensity characteristics of each word segment, determine the intensity changes between word segments in each sentence; Based on the changes in sound intensity between word segments of each sentence, the sound intensity features corresponding to each sentence are obtained; The intensity features corresponding to each sentence are concatenated to obtain the intensity features of the audio sample being read aloud. Obtain the attribute features of the background music sample; The first matching degree is obtained by inputting one or more of the duration feature, pitch feature, and intensity feature of the audio sample and the attribute features of the background music sample into the initial matching degree prediction model; wherein, the attribute features of the background music sample include one or more of the rhythm feature, emotion feature, vocal range feature, and pitch feature of the background music sample. The text features of the reading text corresponding to the audio sample and the attribute features of the background music sample are input into the initial matching degree prediction model to obtain the second matching degree. Based on the first matching degree and the second matching degree, the predicted matching degree between the reading audio sample and the background music sample is obtained; With the goal of narrowing the difference between the labeled matching degree and the predicted matching degree, the initial matching degree prediction model is trained to obtain the trained matching degree prediction model; wherein, the matching degree prediction model is used to predict the matching degree between the input reading audio and each input background music.

5. The method according to claim 4, characterized in that, The method further includes: Obtain historical feedback behavior data for the training works, wherein the training works are obtained by superimposing the reading audio samples and the background music samples, and the historical feedback behavior data includes one or more of the following: satisfaction with the background music samples in the training works, the probability of publishing the training works, and the number of interactions with the published training works. The historical feedback behavior data is standardized to obtain the labeled matching degree.

6. The method according to claim 4, characterized in that, The method further includes: Acquire target feedback behavior data for the target work, wherein the target work is obtained by superimposing the reading audio and the target background music, and the target background music is selected from the background music recommendations, which are generated based on the matching degree between each background music and the reading audio; Based on the target feedback behavior data, the labeled matching degree between the reading audio and the target background music is obtained; Based on the labeled matching degree between the reading audio and the target background music, the matching degree prediction model is optimized.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor; The memory is used to store computer programs, the computer programs including program instructions; The processor is configured to call the program instructions from the memory, causing the electronic device to perform the method as described in any one of claims 1-3 or 4-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method as claimed in any one of claims 1-3 or 4-6.

Images