Intelligent vocal accompaniment method and system

Abstract

The invention belongs to the technical field of audio processing, and provides an intelligent vocal accompaniment method and system, and the method comprises the steps: collecting the high and low pitches sung by a user, collecting the voiceprint information of the user, dividing a song selected based on a song selection instruction into human voice data and accompaniment data, and transmitting the human voice data and accompaniment data to the user; the method comprises the following steps: determining a shift tone tuning number for adjusting accompaniment data according to treble and bass corresponding to human voice data and treble and bass singing by a user, playing an audio based on the adjusted accompaniment data, acquiring and continuously detecting singing data of the user, and when the singing data of the user is detected to meet a preset condition, sending the audio to the user. And adding the vocal accompaniment sound generated through the voiceprint information and the human voice data into the audio. The method provided by the scheme can customize the tone of the song for the user, adjust the accompaniment pitch to a range suitable for the user, ensure that the user can more easily singe, and selectively add the accompaniment sound according to the singing condition of the user, thereby guiding the user to adjust the singing mode of the user, improving the singing level of the user, and improving the singing experience of the user. And the singing experience of the user is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of audio processing, and particularly relates to an intelligent karaoke method and system. BACKGROUND

[0002] With the development of audio processing technology, the music karaoke function has begun to enter the daily life of users, and its application in the fields of entertainment and education is becoming more and more widespread. Whether a user is a singing enthusiast or a professional singer, he or she can enjoy a more personalized music experience through the karaoke function.

[0003] Traditional karaoke systems mostly rely on preset audio and accompaniment segments, and lack personalized functions for individual users. This limitation often makes users feel uncomfortable when singing, affecting their singing performance and experience.

[0004] With the development of intelligent audio technology, some existing intelligent karaoke technologies can make simple adjustments according to user input, but there are some common shortcomings, such as limited adjustment precision, inability to accurately adapt to the vocal range differences of users, or inability to provide personalized guidance according to the actual singing situation of users, which cannot meet the needs of different users and make it difficult to achieve truly personalized and intelligent karaoke experience. SUMMARY

[0005] In order to overcome the defects of the prior art, the present application proposes an intelligent karaoke method, which comprises:

[0006] In response to a collection instruction, record the highest note sung by the user as a first high note and the lowest note as a first low note, and collect the voiceprint information of the user;

[0007] In response to a song selection instruction, divide the song selected based on the song selection instruction into vocal data and accompaniment data, calculate the second high note and the second low note corresponding to the vocal data, determine the number of transposition notes through the first high note, the second high note, the first low note and the second low note, adjust the accompaniment data based on the number of transposition notes, and play audio based on the adjusted accompaniment data;

[0008] Obtain and continuously detect the singing data of the user, and when it is detected that the singing data of the user meets a preset condition, add the karaoke sound generated through the voiceprint information and the vocal data to the audio.

[0009] Specifically, the voiceprint information is obtained through a trained voiceprint collection model, and the method for training the voiceprint collection model to collect the voiceprint information of the user comprises:

[0010] extracting human voice data in singing data of a plurality of singers as first training data inputting the voiceprint collection model, and taking the personnel number corresponding to each singer as a label, so that the voiceprint collection model predicts the personnel number corresponding to each human voice data in the first training data;

[0011] determining the number difference between the predicted personnel number and the real personnel number of the voiceprint collection model, continuously optimizing the parameters of the voiceprint collection model through the number difference and retraining the voiceprint collection model until the number difference meets the preset first training requirement, and completing the training of the voiceprint collection model;

[0012] taking the penultimate layer of the trained voiceprint collection model as an output layer, and inputting the singing data of the user into the trained voiceprint collection model, and outputting the voiceprint information of the user through the output layer.

[0013] Further, the accompaniment voice is generated by a trained timbre conversion model, and the method for training the timbre conversion model comprises:

[0014] inputting the singing data of a first singer into the trained voiceprint collection model and obtaining the voiceprint information of the first singer, inputting the voiceprint information of the first singer and the singing data of a second singer as training data into the timbre conversion model, and making the timbre conversion model predict the singing data of the first singer;

[0015] determining the data difference between the predicted singing data and the real singing data of the timbre conversion model, continuously optimizing the parameters of the timbre conversion model through the data difference and retraining the timbre conversion model until the data difference meets the preset second training requirement, and completing the training of the timbre conversion model.

[0016] Optionally, the preset condition includes that the pause time of the user exceeds a preset first time threshold, and the method for detecting whether the singing data of the user meets the preset condition comprises:

[0017] preprocessing the singing data of the user to obtain processed audio data, identifying the user pitch of the processed audio data at each time, and when the detection result of the user pitch at continuous multiple times is empty and the total time value corresponding to the multiple times with empty continuous detection results exceeds the first time threshold, it is judged that the singing data of the user meets the preset condition.

[0018] Optionally, the preset condition includes that the proportion of incorrect singing of the user within a preset second time threshold exceeds a preset proportion value, and the method for detecting whether the singing data of the user meets the preset condition comprises:

[0019] Preprocessing the singing data of the user to obtain processed audio data, identifying the user pitch of the processed audio data at each time, and obtaining the standard pitch of the karaoke voice at each time, determining the time when the user pitch and the standard pitch at the corresponding time are not matched as the wrong pitch time;

[0020] The number of time points corresponding to the second time threshold is determined as the detection number, and when the proportion of the wrong pitch time in the continuous detection number of time points exceeds the preset proportion value, it is judged that the singing data of the user meets the preset condition.

[0021] Specifically, the karaoke voice is added to the audio, comprising:

[0022] Obtaining the current playback timestamp of the audio, aligning the audio and the audio track time of the karaoke voice, and starting to superimpose the audio and the karaoke voice at the next time of the current playback timestamp.

[0023] Preferably, the method further comprises:

[0024] In response to the collection instruction, judging the type of the collection instruction;

[0025] If the collection instruction is of a first type, playing a scale for the user to sing along, obtaining the singing along data of the user, and determining the first high pitch and the first low pitch based on the singing along data;

[0026] If the collection instruction is of a second type, when responding to the song selection instruction for the first time, playing audio based on unadjusted accompaniment data, obtaining the singing data of the user, and determining the first high pitch and the first low pitch based on the singing data.

[0027] Specifically, the formula for determining the number of transposition sounds comprises:

[0028] N=(H'+L') / 2-(H+L) / 2;

[0029] Wherein, N represents the number of transposition sounds, H represents the first high pitch, H' represents the second high pitch, L represents the first low pitch, and L' represents the second low pitch.

[0030] The present application also provides an intelligent karaoke system, comprising:

[0031] The collection module is used for responding to the collection instruction, recording the highest pitch sung by the user as the first high pitch, recording the lowest pitch as the first low pitch, and collecting the voiceprint information of the user;

[0032] The playing module is configured to, in response to a song selection instruction, divide a song selected based on the song selection instruction into vocal data and accompaniment data, calculate a second high pitch and a second low pitch corresponding to the vocal data, determine a number of transposed pitches based on the first high pitch, the second high pitch, the first low pitch and the second low pitch, adjust the accompaniment data based on the number of transposed pitches, and play audio based on the adjusted accompaniment data.

[0033] The singing module is configured to acquire and continuously detect singing data of the user, and when it is detected that the singing data of the user meets a preset condition, add a singing voice generated based on the voiceprint information and the vocal data to the audio.

[0034] The present application also provides a computer readable storage medium, which stores executable instructions for being executed by a processor to implement the intelligent singing method as described above.

[0035] The present application has at least the following beneficial effects:

[0036] The present application can accurately calculate the number of transposed pitches by real-time acquisition of the highest and lowest pitches of the user and the high and low pitches in the vocal data of the selected song, thereby accurately adjusting the accompaniment pitch according to the vocal range of the user, ensuring that the accompaniment pitch is always within the range suitable for the user to sing, meeting the vocal range of the user, so that the user can sing the song more easily, avoiding the problem that the fixed pitch in the traditional accompaniment system cannot meet the needs of different users; by acquiring the voiceprint information and the vocal data of the user, the singing voice suitable for the user can be dynamically generated during the singing process of the user, thereby providing real-time singing guidance for the user, and the singing voice can be selectively added to the audio according to the singing performance and needs of the user, thereby providing immediate support and adjustment for the user during the singing process, optimizing the singing experience of the user, better enjoying the singing music, and also helping the user to continuously improve the singing level.

[0037] Further, the present application can train the voiceprint acquisition model and continuously optimize the prediction accuracy thereof, so as to accurately extract the voiceprint information of the user through the voiceprint acquisition model, and the timbre conversion model can generate appropriate singing voice according to the voiceprint information of the user, so that the singing style of the user can be more accurately simulated through the use of the timbre conversion model, thereby providing more suitable singing for the user, and the singing performance of the user will be highly matched with the generated singing voice, thereby improving the realism and individuality of the singing effect.

[0038] On this basis, the scheme can also judge the singing state of the user according to the pause time and the proportion of wrong notes in the singing process of the user, and provide singing guide at the appropriate time, which helps the user to improve singing skills and performance, by acquiring the playing time stamp of the current audio and aligning the singing voice with the audio time, the perfect synchronization of the singing voice and the original accompaniment audio can be ensured, different collection methods can be provided for different user needs to improve personalized experience, and the introduced accurate transposition number calculation formula can ensure that the adjustment of the accompaniment pitch is more accurate.

[0039] Therefore, the present application provides an intelligent singing method and system, the method can tailor the pitch of the song for the user, adjust the accompaniment pitch to the range suitable for the user, ensure that the user can sing more easily, and selectively add singing voice according to the singing condition of the user, so as to guide the user to adjust his own singing way, improve his own singing level, and improve the singing experience of the user. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0041] Figure 1 The method flowchart of the intelligent singing method provided for embodiment 1 is shown in the figure.

[0042] Figure 2 The method flowchart of the method for training the voiceprint collection model and the tone conversion model is shown in the figure.

[0043] Figure 3 The module structure schematic diagram of the intelligent singing system provided for embodiment 2 is shown in the figure.

[0044] REFERENCE NUMERALS

[0045] 10-acquisition module; 11-first training unit; 12-first cycle unit; 13-model truncation unit; 14-second training unit; 15-second cycle unit; 20-playing module; 30-singing module. DETAILED DESCRIPTION

[0046] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0047] Various embodiments of the present application will be described in detail below. The present application can have various embodiments, and adjustments and changes can be made therein. However, it should be understood that there is no intention to limit various embodiments of the present application to the specific embodiments disclosed herein, but the present application should be understood to encompass all adjustments, equivalents and / or alternatives falling within the spirit and scope of various embodiments of the present application.

[0048] Hereinafter, the term "include" or "may include" used in various embodiments of the present application indicates the presence of the disclosed functions, operations, or elements, and does not limit the addition of one or more functions, operations, or elements. Also, as used in various embodiments of the present application, the terms "include", "have", and their conjugates merely indicate the presence of specific features, numbers, steps, operations, elements, components, or combinations thereof, and should not be construed as excluding the presence or addition of one or more other features, numbers, steps, operations, elements, components, or combinations thereof.

[0049] In various embodiments of the present application, the expression "or" or "at least one of A or / and B" includes any combination of the listed terms or all combinations thereof. For example, the expression "A or B" or "at least one of A or / and B" can include A, can include B, or can include both A and B.

[0050] The expressions (such as "first", "second", etc.) used in various embodiments of the present application can modify various constituent elements in various embodiments, but can not limit the corresponding constituent elements. For example, the above expressions do not limit the order and / or importance of the described elements. The above expressions are used only for the purpose of distinguishing one element from other elements. For example, the first user device and the second user device indicate different user devices, although both are user devices. For example, a first element can be called a second element, and likewise, a second element can be called a first element, without departing from the scope of various embodiments of the present application.

[0051] It should be noted that in the present application, unless explicitly specified and defined otherwise, the terms "mount", "connect", "fixed", etc. should be understood broadly, for example, can be fixed connection, can be detachable connection, or integral connection; can be mechanical connection, or electrical connection; can be direct connection, or indirect connection through intermediate medium; can be internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0052] In the present application, those of ordinary skill in the art need to understand that the terms indicating the orientation or positional relationship herein are based on the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0053] The terms used in various embodiments of the present application are used only for the purpose of describing specific embodiments and are not intended to limit various embodiments of the present application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly dictates otherwise. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which various embodiments of the present application belong. The terms (such as those defined in a generally used dictionary) will be interpreted to have the same meaning as the contextual meaning in the relevant technical field and will not be interpreted to have an idealized meaning or an overly formal meaning, unless clearly defined in various embodiments of the present application.

[0054] Embodiment 1

[0055] See Figure 1 The present embodiment proposes an intelligent karaoke method, which can solve the problem of mismatch between the song the user likes and the user's vocal range in karaoke and the like, support the user to sing along with any song more comfortably, and greatly improve the user's experience. The method comprises:

[0056] S100: In response to a collection instruction, record the highest note sung by the user as a first high note, record the lowest note as a first low note, and collect the voiceprint information of the user.

[0057] In the present embodiment, step S100 can record the first high note and the first low note that the user can sing using two different types of vocal range test methods. When responding to the collection instruction, the type of the collection instruction can be determined. When the collection instruction indicates that the user needs to use a perceptual vocal range test method for collection, a scale for the user to sing along is played, the singing along data of the user is obtained, and the first high note and the first low note are determined based on the singing along data.

[0058] When the collection instruction indicates that the user needs to use a non-perceptual vocal range test method for collection, the audio is played based on unadjusted accompaniment data when responding to the song selection instruction for the first time, the singing data of the user is obtained, and the first high note and the first low note are determined based on the singing data. When the user uses the non-perceptual vocal range test method for collection, the method proposed in the present embodiment can continuously obtain the singing data of the user and constantly update the first high note and the first low note.

[0059] S200: in response to the song selection instruction, dividing a song selected based on the song selection instruction into vocal data and accompaniment data, calculating a second high pitch and a second low pitch corresponding to the vocal data, determining a number of transposition pitches through a first high pitch, the second high pitch, a first low pitch and the second low pitch, adjusting the accompaniment data based on the number of transposition pitches, and playing audio based on the adjusted accompaniment data.

[0060] Preferably, the division of the song into the vocal data and the accompaniment data in step S200 can be achieved by a music track separation model that has completed training; it should be noted that step S200 can divide the content of the song into a human vocal track human as the vocal data and a comp accompaniment track as the accompaniment data, and the formula for determining the number of transposition pitches in step S100 can specifically include:

[0061] N = (H' + L') / 2 - (H + L) / 2;

[0062] wherein N represents the number of transposition pitches, H represents the first high pitch, H' represents the second high pitch, L represents the first low pitch, and L' represents the second low pitch;

[0063] In this embodiment, the number of transposition pitches is an integer, and the number of transposition pitches determines the number of half steps that the accompaniment track needs to be adjusted;

[0064] For example, when the value of N is 3, step S200 will play the accompaniment track after lowering it by three half steps; when the value of N is -1, step S200 will play the accompaniment track after raising it by one half step; and when the value of N is 0, step S200 can directly play the accompaniment track.

[0065] S300: obtaining and continuously detecting singing data of a user, and when it is detected that the singing data of the user meets a preset condition, adding a backing vocal generated through voiceprint information and vocal data to the audio.

[0066] In this embodiment, the backing vocal includes a processed human vocal track human', and step S300 can obtain a current playback timestamp of the audio, align the tracks of the audio and the backing vocal in time, and start superimposing the audio and the backing vocal at the next moment of the current playback timestamp; that is, after obtaining the playback timestamp of the accompaniment track comp, the processed human vocal track human' is jumped to the corresponding timestamp, so that the processed human vocal track human' and the accompaniment track comp are played in superposition;

[0067] Preferably, the method proposed in this embodiment can also support the user to select the time when the accompaniment sound is added, for example, when the user starts singing in the accompaniment mode, the timestamp of the current playing can be recorded in real time according to the played accompaniment track comp, wherein the starting timestamp is 0, if the user needs to sing along at the time of timestamp t, the method proposed in this embodiment can jump the processed human voice track human' to the position of timestamp t, complete the time alignment of the two tracks of the processed human voice track human' and the accompaniment track comp, and then start from time t+1, so that the audio played at each time is generated by superimposing the processed human voice track human' and the accompaniment track comp.

[0068] In this embodiment, the preset conditions include that the user's pause time exceeds a preset first time threshold T1, and / or the proportion of the user's singing wrong notes within a preset second time threshold T2 exceeds a preset proportion value, and the method of detecting whether the user's singing data meets the preset conditions includes:

[0069] Preprocessing the user's singing data to obtain processed audio data, identifying the user's pitch k_u at each time, when the detection results of the user's pitch k_u at continuous multiple times are empty, and the total time value of the multiple times with empty detection results exceeds the first time threshold T1, it is determined that the user's singing data meets the preset conditions;

[0070] and / or, preprocessing the user's singing data to obtain processed audio data, identifying the user's pitch k_u at each time, and obtaining the standard pitch k_v of the accompaniment sound at each time, determining the time when the user's pitch k_u and the standard pitch k_v at the corresponding time do not match as the wrong note time, determining the number of times corresponding to the second time threshold T2 as the detection number, when the proportion of the wrong note time in the multiple times with continuous detection number exceeds the preset proportion value, it is determined that the user's singing data meets the preset conditions.

[0071] Specifically, the preprocessing of the user's singing data can include noise reduction processing by filtering and other noise reduction algorithms, and the method proposed in this embodiment can identify the user's pitch k_u at each time of the processed audio data in real time by using YIN algorithm and / or deep learning network.

[0072] On this basis, the method proposed in this embodiment can also set a preset cancellation condition for canceling the addition of the accompaniment sound to the audio, in an optional implementation, after the accompaniment sound is added to the audio in step S300, if it is detected that the user's singing data no longer meets the preset conditions, the addition of the accompaniment sound to the audio can be cancelled accordingly, that is, the audio is played again only based on the adjusted accompaniment data.

[0073] Specifically, in step S100, the voiceprint information is obtained by using the voiceprint collection model that is trained, please refer to Figure 2 The method for training the voiceprint collection model to collect the voiceprint information of the user includes:

[0074] S110: Extract the vocal data in the singing data of the plurality of singers as the first training data, input the voiceprint collection model, and input the personnel number corresponding to each singer as the label, so that the voiceprint collection model predicts the personnel number corresponding to each vocal data in the first training data.

[0075] In this embodiment, in step S110, the singers who are uniformly distributed in terms of gender, age, language, etc. can be selected to sing a plurality of designated songs to obtain the singing data, and then the vocal data separated from the singing data corresponding to the same designated song can be used to train the voiceprint collection model.

[0076] S120: Determine the number difference between the predicted personnel number and the real personnel number of the voiceprint collection model, continuously optimize the parameters of the voiceprint collection model through the number difference, and retrain the voiceprint collection model until the number difference meets the preset first training requirement, thereby completing the training of the voiceprint collection model.

[0077] S130: The penultimate layer of the trained voiceprint collection model is used as the output layer, and the singing data of the user is input into the trained voiceprint collection model, and the voiceprint information of the user is output through the output layer.

[0078] It should be noted that step S130 completes the generation of the voiceprint collection model by truncating the model, and after the voiceprint collection model is truncated, step S100 can output the voiceprint information of the user by inputting the singing data of the user into the voiceprint collection model.

[0079] Further, the accompaniment voice is generated by using the trained timbre conversion model, and in this embodiment, the voiceprint collection model and the timbre conversion model are a joint training model in stages, and the method for training the timbre conversion model includes:

[0080] S140: Input the singing data of the first singer into the trained voiceprint collection model to obtain the voiceprint information of the first singer, input the voiceprint information of the first singer and the singing data of the second singer into the trained timbre conversion model as the training data, and make the timbre conversion model predict the singing data of the first singer.

[0081] Specifically, step S140 can take any one singer as the first singer and another singer as the second singer, input the singing data x_m of the first singer into the voiceprint collection model that has completed training, at this time, the parameters of the voiceprint collection model have been fixed, and the voiceprint information of the first singer can be output, and then the singing data x_n of the second singer singing the same song and the voiceprint information of the first singer can be input into the timbre conversion model as training data, so that the timbre conversion model predicts the singing data of the first singer, and the singing data x_m' predicted by the timbre conversion model is compared with the real singing data x_m, so as to realize the calculation of loss and the training of the timbre conversion model.

[0082] S150: determining the data difference between the predicted singing data of the timbre conversion model and the real singing data, continuously optimizing the parameters of the timbre conversion model through the data difference and retraining the timbre conversion model until the data difference meets the preset second training requirement, and completing the training of the timbre conversion model.

[0083] Embodiment 2

[0084] This embodiment proposes an intelligent karaoke system, which can be used to realize the intelligent karaoke method proposed in embodiment 1, please refer to Figure 3 , the system comprises:

[0085] The collection module 10 is configured to, in response to a collection instruction, record the highest pitch sung by the user as the first highest pitch and the lowest pitch as the first lowest pitch, and collect the voiceprint information of the user.

[0086] The playing module 20 is configured to, in response to a song selection instruction, divide a song selected based on the song selection instruction into vocal data and accompaniment data, calculate a second highest pitch and a second lowest pitch corresponding to the vocal data, determine a transposition number based on the first highest pitch, the second highest pitch, the first lowest pitch and the second lowest pitch, adjust the accompaniment data based on the transposition number, and play the audio based on the adjusted accompaniment data.

[0087] The karaoke module 30 is configured to acquire and continuously detect the singing data of the user, and when it is detected that the singing data of the user meets a preset condition, add the karaoke sound generated based on the voiceprint information and the vocal data to the audio.

[0088] Specifically, the collection module 10 can use two different types of pitch range test methods to record the first highest pitch and the first lowest pitch that the user can sing, when responding to the collection instruction, the type of the collection instruction can be judged, when the collection instruction indicates that the user needs to use the perceptual pitch range test method for collection, the scale for the user to sing along is played, the singing along data of the user is acquired, and the first highest pitch and the first lowest pitch are determined based on the singing along data.

[0089] When the collection instruction indicates that the user needs to use the non-aware vocal range test mode for collection, the audio is played based on the unadjusted accompaniment data when the user first responds to the song selection instruction, the singing data of the user is obtained, and the first high pitch and the first low pitch are determined based on the singing data; when the user uses the non-aware vocal range test mode for collection, the collection module 10 can continuously obtain the singing data of the user and continuously update the first high pitch and the first low pitch.

[0090] In the embodiment, the karaoke sound includes the processed human voice track human', the accompaniment module 30 can obtain the current playback timestamp of the audio, align the audio with the track time of the karaoke sound, and start superimposing the audio and the karaoke sound at the next moment of the current playback timestamp;

[0091] Preferably, the accompaniment module 30 can also support the user to select the time to join the karaoke sound by himself / herself. For example, when the user starts singing in the accompaniment mode, the current playback timestamp can be recorded in real time according to the played accompaniment track comp, wherein the starting timestamp is 0. If the user needs to sing along at the moment of the timestamp t, the accompaniment module 30 can jump the processed human voice track human' to the position of the timestamp t, complete the time alignment of the two tracks of the processed human voice track human' and the accompaniment track comp, and then start playing the audio at t+1 moment, which is generated by superimposing the processed human voice track human' and the accompaniment track comp.

[0092] In the embodiment, the preset condition includes that the pause time of the user exceeds a preset first time threshold T1, and / or the proportion of incorrect pitches sung by the user within a preset second time threshold T2 exceeds a preset proportion value, and the manner in which the accompaniment module 30 detects whether the singing data of the user meets the preset condition includes:

[0093] The singing data of the user is preprocessed to obtain processed audio data, the user pitch k_u at each moment is recognized, when the detection results of the user pitch k_u at continuous multiple moments are empty, and the total time value of the multiple moments with empty continuous detection results exceeds the first time threshold T1, it is judged that the singing data of the user meets the preset condition;

[0094] And / or, the singing data of the user is preprocessed to obtain processed audio data, the user pitch k_u at each moment is recognized, and the standard pitch k_v of the karaoke sound at each moment is obtained. The moment when the user pitch k_u and the standard pitch k_v at the corresponding moment are not matched is determined as the incorrect pitch moment, the number of moments corresponding to the second time threshold T2 is determined as the detection number, and when the proportion of incorrect pitch moments in the multiple moments with continuous detection number exceeds the preset proportion value, it is judged that the singing data of the user meets the preset condition.

[0095] Specifically, the preprocessing of the user's singing data can include noise reduction processing by a noise reduction algorithm such as filtering, and the accompaniment module 30 can identify the user's pitch ku at each moment in real time by using a YIN algorithm and / or a deep learning network.

[0096] On this basis, the accompaniment module 30 can also set a preset cancel condition for adding the accompaniment sound to the audio, and in an optional embodiment, after the accompaniment module 30 adds the accompaniment sound to the audio, if it is detected that the user's singing data no longer meets the preset condition, the accompaniment sound can be canceled from being added to the audio, i.e., the audio is played again only based on the adjusted accompaniment data.

[0097] Specifically, the collection module 10 can obtain the voiceprint information by using a trained voiceprint collection model, and the collection module 10 can include:

[0098] The first training unit 11 is configured to extract the vocal data in the singing data of a plurality of singers as the first training data input into the voiceprint collection model, and input the personnel number corresponding to each singer as a label, so that the voiceprint collection model predicts the personnel number corresponding to each vocal data in the first training data;

[0099] The first cycle unit 12 is configured to determine the number difference between the predicted personnel number and the real personnel number of the voiceprint collection model, continuously optimize the parameters of the voiceprint collection model through the number difference, and retrain the voiceprint collection model until the number difference meets the preset first training requirement, thereby completing the training of the voiceprint collection model;

[0100] The model truncation unit 13 is configured to use the second-to-last layer of the trained voiceprint collection model as an output layer, and input the user's singing data into the trained voiceprint collection model, and output the user's voiceprint information through the output layer.

[0101] In this embodiment, the first training unit 11 can select singers with uniform distribution in gender, age, language, etc. to sing a plurality of designated songs to obtain singing data, and then the vocal data separated from the singing data corresponding to the same designated song can be used to train the voiceprint collection model;

[0102] The model truncation unit 13 completes the generation of the voiceprint collection model by truncating the model, and after truncating the voiceprint collection model, the collection module 10 can input the user's singing data into the voiceprint collection model, so that the output layer of the voiceprint collection model outputs the voiceprint information of the user.

[0103] Further, the accompaniment sound is generated by a trained timbre conversion model, and in this embodiment, the voiceprint collection model and the timbre conversion model are a joint training model in stages, and the collection module 10 can further include:

[0104] The second training unit 14 is configured to input the singing data of the first singer into the trained voiceprint collection model to obtain voiceprint information of the first singer, and input the voiceprint information of the first singer and the singing data of the second singer as training data into the timbre conversion model to make the timbre conversion model predict the singing data of the first singer.

[0105] The second loop unit 15 is configured to determine a data difference between the predicted singing data of the timbre conversion model and the real singing data, continuously optimize the parameters of the timbre conversion model through the data difference and retrain the timbre conversion model until the data difference meets a preset second training requirement, and complete the training of the timbre conversion model.

[0106] Specifically, the second training unit 14 can take any singer as the first singer and another singer as the second singer, input the singing data x_m of the first singer into the trained voiceprint collection model, at this time, the parameters of the voiceprint collection model have been fixed and the voiceprint information of the first singer can be output, and then input the singing data x_n of the second singer singing the same song and the voiceprint information of the first singer as training data into the timbre conversion model to make the timbre conversion model predict the singing data of the first singer, and compare the predicted singing data x_m' of the timbre conversion model with the real singing data x_m to realize the calculation of loss and the training.

[0107] Embodiment 3

[0108] The embodiment provides a computer readable storage medium, and a computer instruction is stored on the computer readable storage medium. The computer instruction is executed by a processor to realize the steps of the intelligent accompaniment method provided in the above embodiment 1.

[0109] Note that the computer-readable medium can include persistent and non-persistent, movable and non-movable media that can be implemented in any method or technology for storing information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, disk storage, quantum memory, graphene-based storage medium or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, computer-readable medium does not include transitory medium such as modulated data signals and carriers.

[0110] In summary, the present application provides an intelligent karaoke method and system. The method can tailor the pitch of songs for users, adjust the accompaniment pitch to a range suitable for users, ensure that users can sing more easily, and selectively add accompaniment sound according to the singing situation of users, thereby guiding users to adjust their singing manner, improve their singing level, and improve the singing experience of users.

[0111] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An intelligent accompaniment singing method, characterized in that, The method includes: In response to the acquisition command, the highest note sung by the user is recorded as the first high note, the lowest note is recorded as the first low note, and the user's voiceprint information is acquired; In response to a song selection command, the song selected based on the song selection command is divided into vocal data and accompaniment data. The second high note and the second low note corresponding to the vocal data are calculated. The number of transposition notes is determined by the first high note, the second high note, the first low note, and the second low note. The accompaniment data is adjusted based on the number of transposition notes. The audio is played based on the adjusted accompaniment data. The system acquires and continuously monitors the user's singing data. When the user's singing data meets preset conditions, the system adds backing vocals generated from the voiceprint information and the human voice data to the audio.

2. The intelligent accompaniment singing method according to claim 1, characterized in that, The voiceprint information is obtained through a trained voiceprint acquisition model. The method for training the voiceprint acquisition model to acquire the user's voiceprint information includes: The vocal data from the singing data of multiple singers is extracted as the first training data and input into the voiceprint acquisition model. The personnel number corresponding to each singer is used as a label, so that the voiceprint acquisition model can predict the personnel number corresponding to each vocal data in the first training data. The difference between the predicted personnel number and the actual personnel number of the voiceprint acquisition model is determined. The parameters of the voiceprint acquisition model are continuously optimized based on the difference in number, and the voiceprint acquisition model is retrained until the difference in number meets the preset first training requirement, thus completing the training of the voiceprint acquisition model. The penultimate layer of the trained voiceprint acquisition model is used as the output layer, and the user's singing data is input into the trained voiceprint acquisition model. The user's voiceprint information is then output through the output layer.

3. The intelligent accompaniment singing method according to claim 2, characterized in that, The backing vocals are generated using a trained timbre conversion model. The method for training the timbre conversion model includes: The singing data of the first singer is input into the trained voiceprint acquisition model to obtain the voiceprint information of the first singer. The voiceprint information of the first singer and the singing data of the second singer are used as training data and input into the timbre conversion model so that the timbre conversion model can predict the singing data of the first singer. The data difference between the predicted singing data and the actual singing data of the timbre conversion model is determined. The parameters of the timbre conversion model are continuously optimized based on the data difference, and the timbre conversion model is retrained until the data difference meets the preset second training requirement, thus completing the training of the timbre conversion model.

4. The intelligent accompaniment singing method according to claim 1, characterized in that, The preset conditions include the user's pause time exceeding a preset first time threshold. The method for detecting whether the user's singing data meets the preset conditions includes: The user's singing data is preprocessed to obtain processed audio data. The user's pitch at each moment is identified. When the detection result of the user's pitch is empty for multiple consecutive moments, and the total time value corresponding to the multiple consecutive moments with empty detection results exceeds the first time threshold, it is determined that the user's singing data meets the preset condition.

5. The intelligent accompaniment singing method according to claim 1, characterized in that, The preset conditions include the proportion of the user singing wrong notes exceeding a preset proportion value within a preset second time threshold. The method for detecting whether the user's singing data meets the preset conditions includes: The user's singing data is preprocessed to obtain processed audio data. The user's pitch at each moment is identified in the processed audio data, and the standard pitch of the accompaniment is obtained at each moment. The moment when the user's pitch does not match the standard pitch at the corresponding moment is determined as the wrong note moment. The number of moments corresponding to the second time threshold is determined as the detection quantity. When the proportion of wrong notes in multiple consecutive detection times exceeds the preset proportion value, it is determined that the user's singing data meets the preset condition.

6. The intelligent accompaniment singing method according to any one of claims 1-5, characterized in that, Adding backing vocals to the audio includes: Obtain the current playback timestamp of the audio, align the audio with the backing vocals in time, and start superimposing the audio and backing vocals at the next moment after the current playback timestamp.

7. The intelligent accompaniment singing method according to claim 1, characterized in that, The method further includes: In response to the acquisition command, determine the type of the acquisition command; If the acquisition instruction is of the first type, play the scale for the user to sing along, acquire the user's singing data, and determine the first high note and the first low note based on the singing data; If the acquisition command is of the second type, when responding to the song selection command for the first time, the audio is played based on the unadjusted accompaniment data, the user's singing data is acquired, and the first high note and the first low note are determined based on the singing data.

8. The intelligent accompaniment singing method according to claim 1 or 7, characterized in that, The formula for determining the number of transposed notes includes: N = (H' + L') / 2 - (H + L) / 2; Wherein, N represents the number of transposed notes, H represents the first high note, H' represents the second high note, L represents the first low note, and L' represents the second low note.

9. An intelligent karaoke system, characterized in that, The system includes: The acquisition module is used to respond to the acquisition command, record the highest note sung by the user as the first high note, record the lowest note as the first low note, and acquire the user's voiceprint information; The playback module is used to respond to a song selection command, divide the song selected based on the song selection command into vocal data and accompaniment data, calculate the second high note and the second low note corresponding to the vocal data, determine the number of transposition notes through the first high note, the second high note, the first low note and the second low note, adjust the accompaniment data based on the number of transposition notes, and play audio based on the adjusted accompaniment data; The accompaniment module is used to acquire and continuously detect the user's singing data, and when the user's singing data meets preset conditions, it adds the accompaniment voice generated by the voiceprint information and the human voice data to the audio.

10. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the intelligent accompaniment singing method as described in any one of claims 1-8 when executed by a processor.

Images