A smart accompaniment singing method and system

Abstract

This invention belongs to the field of audio processing technology and provides an intelligent accompaniment singing method and system. The method can collect the user's vocal pitch and voiceprint information, then divide the song selected based on the song selection command into vocal data and accompaniment data. The pitch of the accompaniment data is adjusted by comparing the vocal data with the user's vocal pitch, and the audio is played based on the adjusted accompaniment data. The user's singing data is continuously acquired and monitored, and when the user's singing data meets preset conditions, accompaniment vocals generated using the voiceprint information and vocal data are added to the audio. This proposed method can tailor the song's pitch to the user's individual needs, adjusting the accompaniment pitch to a suitable range for the user, ensuring easier singing. It can also selectively add accompaniment vocals based on the user's singing performance, thereby guiding the user to adjust their singing style, improve their singing level, and enhance their singing experience.

Description

Technical Field

[0001] This invention belongs to the field of audio processing technology, specifically relating to an intelligent accompaniment singing method and system. Background Technology

[0002] With the development of audio processing technology, karaoke functionality has begun to enter users' daily lives, especially in the fields of entertainment and education, where its application is becoming increasingly widespread. Whether you are a singing enthusiast or a professional singer, you can enjoy a more personalized music experience through karaoke functionality.

[0003] Traditional karaoke systems rely heavily on preset audio and accompaniment clips, lacking personalized features for individual users. This limitation often makes users feel uncomfortable while singing, affecting their performance and experience.

[0004] With the development of intelligent audio technology, some existing intelligent accompaniment technologies can make simple adjustments based on user input, but they generally have some shortcomings. For example, the adjustment precision is limited, and it is impossible to accurately adapt to the differences in the user's vocal range, or it is impossible to provide personalized guidance based on the user's actual singing situation, thus failing to meet the needs of different users and making it difficult to achieve a truly personalized and intelligent accompaniment experience. Summary of the Invention

[0005] To overcome the shortcomings of the prior art, this invention proposes an intelligent accompaniment singing method, the method comprising:

[0006] In response to the acquisition command, the highest note sung by the user is recorded as the first high note, the lowest note is recorded as the first low note, and the user's voiceprint information is acquired;

[0007] In response to a song selection command, the song selected based on the song selection command is divided into vocal data and accompaniment data. The second high note and the second low note corresponding to the vocal data are calculated. The number of transposition notes is determined by the first high note, the second high note, the first low note, and the second low note. The accompaniment data is adjusted based on the number of transposition notes. The audio is played based on the adjusted accompaniment data.

[0008] The system acquires and continuously monitors the user's singing data. When the user's singing data meets preset conditions, the system adds backing vocals generated from the voiceprint information and the human voice data to the audio.

[0009] Specifically, the voiceprint information is obtained through a trained voiceprint acquisition model. The method for training the voiceprint acquisition model to acquire the user's voiceprint information includes:

[0010] The vocal data from the singing data of multiple singers is extracted as the first training data and input into the voiceprint acquisition model. The personnel number corresponding to each singer is used as a label, so that the voiceprint acquisition model can predict the personnel number corresponding to each vocal data in the first training data.

[0011] The difference between the predicted personnel number and the actual personnel number of the voiceprint acquisition model is determined. The parameters of the voiceprint acquisition model are continuously optimized based on the difference in number, and the voiceprint acquisition model is retrained until the difference in number meets the preset first training requirement, thus completing the training of the voiceprint acquisition model.

[0012] The penultimate layer of the trained voiceprint acquisition model is used as the output layer, and the user's singing data is input into the trained voiceprint acquisition model. The user's voiceprint information is then output through the output layer.

[0013] Furthermore, the backing vocals are generated using a trained timbre conversion model, and the method for training the timbre conversion model includes:

[0014] The singing data of the first singer is input into the trained voiceprint acquisition model to obtain the voiceprint information of the first singer. The voiceprint information of the first singer and the singing data of the second singer are used as training data and input into the timbre conversion model so that the timbre conversion model can predict the singing data of the first singer.

[0015] The data difference between the predicted singing data and the actual singing data of the timbre conversion model is determined. The parameters of the timbre conversion model are continuously optimized based on the data difference, and the timbre conversion model is retrained until the data difference meets the preset second training requirement, thus completing the training of the timbre conversion model.

[0016] Optionally, the preset condition includes the user's pause time exceeding a preset first time threshold, and the method for detecting whether the user's singing data meets the preset condition includes:

[0017] The user's singing data is preprocessed to obtain processed audio data. The user's pitch at each moment is identified. When the detection result of the user's pitch is empty for multiple consecutive moments, and the total time value corresponding to the multiple consecutive moments with empty detection results exceeds the first time threshold, it is determined that the user's singing data meets the preset condition.

[0018] Optionally, the preset condition includes the user singing wrong notes more than a preset proportion within a preset second time threshold, and the method for detecting whether the user's singing data meets the preset condition includes:

[0019] The user's singing data is preprocessed to obtain processed audio data. The user's pitch at each moment is identified in the processed audio data, and the standard pitch of the accompaniment is obtained at each moment. The moment when the user's pitch does not match the standard pitch at the corresponding moment is determined as the wrong note moment.

[0020] The number of moments corresponding to the second time threshold is determined as the detection quantity. When the proportion of wrong notes in multiple consecutive detection times exceeds the preset proportion value, it is determined that the user's singing data meets the preset condition.

[0021] Specifically, adding backing vocals to the audio includes:

[0022] Obtain the current playback timestamp of the audio, align the audio with the backing vocals in time, and start superimposing the audio and backing vocals at the next moment after the current playback timestamp.

[0023] Preferably, the method further includes:

[0024] In response to the acquisition command, determine the type of the acquisition command;

[0025] If the acquisition instruction is of the first type, play the scale for the user to sing along, acquire the user's singing data, and determine the first high note and the first low note based on the singing data;

[0026] If the acquisition command is of the second type, when responding to the song selection command for the first time, the audio is played based on the unadjusted accompaniment data to acquire the user's singing data, and the first high note and the first low note are determined based on the singing data.

[0027] Specifically, the formula for determining the number of transposed notes includes:

[0028] N = (H' + L') / 2 - (H + L) / 2;

[0029] Wherein, N represents the number of transposed notes, H represents the first high note, H' represents the second high note, L represents the first low note, and L' represents the second low note.

[0030] This invention also proposes an intelligent accompaniment singing system, the system comprising:

[0031] The acquisition module is used to respond to the acquisition command, record the highest note sung by the user as the first high note, record the lowest note as the first low note, and acquire the user's voiceprint information;

[0032] The playback module is used to respond to a song selection command, divide the song selected based on the song selection command into vocal data and accompaniment data, calculate the second high note and the second low note corresponding to the vocal data, determine the number of transposition notes through the first high note, the second high note, the first low note and the second low note, adjust the accompaniment data based on the number of transposition notes, and play audio based on the adjusted accompaniment data;

[0033] The accompaniment module is used to acquire and continuously detect the user's singing data, and when the user's singing data meets preset conditions, it adds the accompaniment voice generated by the voiceprint information and the human voice data to the audio.

[0034] The present invention also proposes a computer-readable storage medium storing executable instructions that, when executed by a processor, implement the intelligent accompaniment method as described above.

[0035] The present invention has at least the following beneficial effects:

[0036] The proposed solution can accurately calculate the number of transposition notes by real-time acquisition of the user's highest and lowest notes and combining this with the high and low notes in the vocal data of the selected song. This allows for precise adjustment of the accompaniment pitch according to the user's vocal range, ensuring that the accompaniment pitch is always within a suitable range for the user's singing and conforms to the user's vocal range. This makes it easier for the user to sing songs and avoids the problem that fixed pitch in traditional accompaniment systems cannot meet the needs of different users. By acquiring the user's voiceprint information and vocal data, the solution can dynamically generate suitable backing vocals during the user's singing process, thereby providing the user with real-time backing vocal guidance. Furthermore, it can selectively add backing vocals to the audio according to the user's singing performance and needs, providing the user with immediate support and adjustments during the singing process. This optimizes the user's singing experience, allows them to better enjoy the fun of singing, and helps them continuously improve their singing skills.

[0037] Furthermore, the solution proposed in this invention can train the voiceprint acquisition model and continuously optimize its prediction accuracy to ensure that the voiceprint acquisition model can accurately extract the user's voiceprint information. The timbre conversion model can generate a suitable backing vocal based on the user's voiceprint information. By using the timbre conversion model, the user's singing style can be simulated more accurately, thereby providing a more suitable backing vocal. The user's singing performance will be highly matched with the generated backing vocal, thereby improving the realism and personalization of the backing vocal effect.

[0038] Building upon this foundation, the solution can also assess a user's singing performance based on pause times and the proportion of wrong notes during their singing, and provide backing vocal guidance at appropriate times. This helps users improve their singing skills and performance. By acquiring the playback timestamp of the current audio and aligning the backing vocals with the audio time, perfect synchronization between the backing vocals and the original accompaniment audio can be ensured. Different acquisition methods can be provided for different user needs, enhancing the personalized experience. The introduced precise transposition pitch calculation formula ensures more accurate adjustment of the accompaniment pitch.

[0039] Therefore, this invention proposes an intelligent accompaniment singing method and system. The method proposed in this solution can customize the pitch of a song for the user, adjust the accompaniment pitch to a suitable range for the user, ensure that the user can sing more easily, and selectively add backing vocals according to the user's singing situation, thereby guiding the user to adjust their singing style, improve their singing level, and enhance the user's singing experience. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating the intelligent accompaniment singing method provided in Example 1;

[0042] Figure 2 A schematic diagram illustrating the process for training the voiceprint acquisition model and the timbre conversion model;

[0043] Figure 3 This is a schematic diagram of the module structure of the intelligent accompaniment system provided in Example 2.

[0044] Figure Labels

[0045] 10-Acquisition module; 11-First training unit; 12-First loop unit; 13-Model truncation unit; 14-Second training unit; 15-Second loop unit; 20-Playback module; 30-Accompaniment module. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0047] Various embodiments of the invention will be described more fully below. The invention may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of the invention to the specific embodiments disclosed herein, but rather the invention should be understood to cover all modifications, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of the invention.

[0048] In the following, the terms “comprising” or “may include” as used in various embodiments of the invention indicate the presence of the disclosed functions, operations, or elements, and do not limit the addition of one or more functions, operations, or elements. Furthermore, as used in various embodiments of the invention, the terms “comprising,” “having,” and their cognates are intended only to indicate a specific feature, number, step, operation, element, component, or combination of the foregoing, and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, elements, components, or combinations of the foregoing, or the possibility of adding one or more combinations of the foregoing.

[0049] In various embodiments of the invention, the expression "or" or "at least one of A and / or B" includes any combination or all combinations of the words listed simultaneously. For example, the expression "A or B" or "at least one of A and / or B" may include A, may include B, or may include both A and B.

[0050] The expressions used in the various embodiments of the present invention (such as "first," "second," etc.) may modify various constituent elements in the various embodiments, but do not limit the corresponding constituent elements. For example, the above expressions do not limit the order and / or importance of the elements. The above expressions are only used for the purpose of distinguishing one element from other elements. For example, a first user device and a second user device refer to different user devices, although both are user devices. For example, a first element may be referred to as a second element without departing from the scope of the various embodiments of the present invention, and similarly, a second element may also be referred to as a first element.

[0051] It should be noted that, in this invention, unless otherwise explicitly specified and defined, terms such as "installation," "connection," and "fixation" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0052] In this invention, those skilled in the art should understand that the terms indicating orientation or positional relationship in the text are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the purpose of facilitating the description of this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0053] The terminology used in the various embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to limit the various embodiments of the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the various embodiments of the invention pertain. The terms (such as those defined in a generally used dictionary) are to be interpreted as having the same meaning as in the context of the relevant technical field and are not to be interpreted as having an idealized or overly formal meaning, unless clearly defined in the various embodiments of the invention.

[0054] Example 1

[0055] Please see Figure 1 This embodiment proposes an intelligent accompaniment singing method that can solve the problem of mismatch between a user's favorite song and their personal vocal range in scenarios such as karaoke. It allows users to more comfortably use any song for accompaniment singing, greatly improving the user experience. The method includes:

[0056] S100: In response to the acquisition command, the highest note sung by the user is recorded as the first high note and the lowest note is recorded as the first low note, and the user's voiceprint information is acquired.

[0057] In this embodiment, step S100 can use two different types of pitch range testing methods to record the first high note and the first low note that the user can sing. When responding to the acquisition command, the type of acquisition command can be determined. When the acquisition command indicates that the user needs to use a perceived pitch range testing method for acquisition, the scale used to make the user sing along is played, the user's singing data is obtained, and the first high note and the first low note are determined based on the singing data.

[0058] When the acquisition command indicates that the user needs to use the non-perceptive range test method for acquisition, the audio is played based on the unadjusted accompaniment data when responding to the song selection command for the first time, the user's singing data is acquired, and the first high note and the first low note are determined based on the singing data. When the user uses the non-perceptive range test method for acquisition, the method proposed in this embodiment can continuously acquire the user's singing data and continuously update the first high note and the first low note.

[0059] S200: In response to the song selection command, the song selected based on the song selection command is divided into vocal data and accompaniment data. The second high note and the second low note corresponding to the vocal data are calculated. The transposition number is determined by the first high note, the second high note, the first low note and the second low note. The accompaniment data is adjusted based on the transposition number, and the audio is played based on the adjusted accompaniment data.

[0060] Preferably, dividing the song into vocal data and accompaniment data in step S200 can be achieved by completing a trained music track segmentation model; it should be noted that step S200 can divide the song content into a vocal track (human) as vocal data and an accompaniment track (comp) as accompaniment data, and the formula for determining the number of transposition notes in step S100 may specifically include:

[0061] N = (H' + L') / 2 - (H + L) / 2;

[0062] Where N represents the number of transposed notes, H represents the first high note, H' represents the second high note, L represents the first low note, and L' represents the second low note;

[0063] In this embodiment, the transposition number is an integer, and the transposition number determines the number of semitones that need to be adjusted in the accompaniment track;

[0064] For example, when the value of N is 3, step S200 will lower the accompaniment track by three semitones before playing it; when the value of N is -1, step S200 will raise the accompaniment track by one semitone before playing it; and when the value of N is 0, step S200 can directly play the accompaniment track.

[0065] S300: Acquires and continuously detects the user's singing data, and when the user's singing data meets the preset conditions, adds the accompaniment voice generated by voiceprint information and human voice data to the audio.

[0066] In this embodiment, the backing vocals include the processed human voice track human'. Step S300 can obtain the current playback timestamp of the audio, align the audio and backing vocal tracks in time, and start superimposing the audio and backing vocals at the next moment after the current playback timestamp; that is, after obtaining the playback timestamp of the backing track comp, the processed human voice track human' jumps to the corresponding timestamp, thereby superimposing the processed human voice track human' and the backing track comp.

[0067] Preferably, the method proposed in this embodiment can also support users to choose the time to add backing vocals. For example, when a user turns on the backing vocal mode to sing, the current playback timestamp can be recorded in real time according to the playing backing track comp, where the starting timestamp is 0. If the user needs to sing along at timestamp t, the method proposed in this embodiment can jump the processed human voice track human' to the position of timestamp t, complete the time alignment of the two tracks human' and backing track comp, and then from time t+1, the audio played at each time is generated by superimposing the processed human voice track human' and backing track comp.

[0068] In this embodiment, the preset conditions include the user's pause time exceeding a preset first time threshold T1, and / or the proportion of the user singing wrong notes within a preset second time threshold T2 exceeding a preset proportion value. The method for detecting whether the user's singing data meets the preset conditions includes:

[0069] The user's singing data is preprocessed to obtain processed audio data. The user's pitch k_u at each moment of the processed audio data is identified. When the detection results for the user's pitch k_u are empty for multiple consecutive moments, and the total time value corresponding to the multiple consecutive moments with empty detection results exceeds the first time threshold T1, it is determined that the user's singing data meets the preset conditions.

[0070] And / or, preprocess the user's singing data to obtain processed audio data, identify the user's pitch k_u at each moment of the processed audio data, and obtain the standard pitch k_v of the accompaniment at each moment. The moment when the user's pitch k_u does not match the standard pitch k_v at the corresponding moment is determined as the wrong note moment. The number of moments corresponding to the second time threshold T2 is determined as the detection number. When the proportion of wrong note moments in multiple consecutive detection moments exceeds the preset proportion value, it is determined that the user's singing data meets the preset conditions.

[0071] Specifically, the preprocessing of the user's singing data may include noise reduction processing through noise reduction algorithms such as filtering. The method proposed in this embodiment can identify and process the user's pitch k_u at each moment of the audio data in real time using the YIN algorithm and / or deep learning networks.

[0072] Based on this, the method proposed in this embodiment can also set a preset cancellation condition for canceling the addition of backing vocals to the audio; in an optional implementation, after adding backing vocals to the audio in step S300, if it is detected that the user's singing data no longer meets the preset condition, the addition of backing vocals to the audio can be cancelled accordingly, that is, the audio is played again based only on the adjusted accompaniment data.

[0073] Specifically, in step S100, the voiceprint information is obtained through the trained voiceprint acquisition model. Please refer to [link to relevant documentation]. Figure 2 Methods for training a voiceprint acquisition model to collect user voiceprint information include:

[0074] S110: Extract the vocal data from the singing data of multiple singers as the first training data and input it into the voiceprint acquisition model. Use the personnel number corresponding to each singer as a label so that the voiceprint acquisition model can predict the personnel number corresponding to each vocal data in the first training data.

[0075] In this embodiment, in step S110, singers who are evenly distributed in terms of gender, age, language, etc. can be selected to sing multiple specified songs to obtain singing data. Then, the voice data separated from the singing data corresponding to the same specified song can be used to train the voiceprint acquisition model.

[0076] S120: Determine the difference between the predicted personnel number and the actual personnel number in the voiceprint acquisition model, continuously optimize the parameters of the voiceprint acquisition model based on the difference in number, and retrain the voiceprint acquisition model until the difference in number meets the preset first training requirement, thus completing the training of the voiceprint acquisition model.

[0077] S130: The penultimate layer of the trained voiceprint acquisition model is used as the output layer, and the user's singing data is input into the trained voiceprint acquisition model. The user's voiceprint information is then output through the output layer.

[0078] It should be noted that step S130 generates the voiceprint acquisition model by truncating the model. After truncating the voiceprint acquisition model, step S100 can output the user's voiceprint information by inputting the user's singing data into the voiceprint acquisition model.

[0079] Furthermore, the backing vocals are generated through a trained timbre conversion model. In this embodiment, the voiceprint acquisition model and the timbre conversion model are jointly trained in stages. The method for training the timbre conversion model includes:

[0080] S140: Input the singing data of the first singer into the trained voiceprint acquisition model and obtain the voiceprint information of the first singer. Input the voiceprint information of the first singer and the singing data of the second singer as training data into the timbre conversion model so that the timbre conversion model can predict the singing data of the first singer.

[0081] Specifically, in step S140, any singer can be designated as the first singer and another singer as the second singer. The singing data x_m of the first singer is input into the trained voiceprint acquisition model. At this point, the parameters of the voiceprint acquisition model are fixed and it can output the voiceprint information of the first singer. Then, the singing data x_n of the second singer singing the same song and the voiceprint information of the first singer can be used as training data to input into the timbre conversion model, so that the timbre conversion model can predict the singing data of the first singer. The singing data x_m' predicted by the timbre conversion model is compared with the real singing data x_m, thereby realizing the calculation of loss for training.

[0082] S150: Determine the data difference between the predicted singing data and the actual singing data of the timbre conversion model, continuously optimize the parameters of the timbre conversion model based on the data difference, and retrain the timbre conversion model until the data difference meets the preset second training requirement, thus completing the training of the timbre conversion model.

[0083] Example 2

[0084] This embodiment proposes an intelligent accompaniment system, which can be used to implement the intelligent accompaniment method proposed in Embodiment 1. Please refer to [link to embodiment]. Figure 3 The system includes:

[0085] The acquisition module 10 is used to respond to the acquisition command, record the highest note sung by the user as the first high note, record the lowest note as the first low note, and acquire the user's voiceprint information;

[0086] The playback module 20 is used to respond to the song selection command, divide the song selected based on the song selection command into vocal data and accompaniment data, calculate the second high note and the second low note corresponding to the vocal data, determine the transposition number through the first high note, the second high note, the first low note and the second low note, adjust the accompaniment data based on the transposition number, and play the audio based on the adjusted accompaniment data.

[0087] The accompaniment module 30 is used to acquire and continuously detect the user's singing data, and when the user's singing data meets the preset conditions, it adds the accompaniment voice generated by voiceprint information and human voice data to the audio.

[0088] Specifically, the acquisition module 10 can use two different types of pitch range testing methods to record the first high note and the first low note that the user can sing. When responding to an acquisition command, it can determine the type of acquisition command. When the acquisition command indicates that the user needs to use a perceived pitch range testing method for acquisition, it plays a scale for the user to sing along, acquires the user's singing data, and determines the first high note and the first low note based on the singing data.

[0089] When the acquisition command indicates that the user needs to use the non-perceptive range test method for acquisition, the acquisition module 10 plays the audio based on the unadjusted accompaniment data when responding to the song selection command for the first time, acquires the user's singing data, and determines the first high note and the first low note based on the singing data; when the user uses the non-perceptive range test method for acquisition, the acquisition module 10 can continuously acquire the user's singing data and continuously update the first high note and the first low note.

[0090] In this embodiment, the backing vocals include the processed human voice track "human". The backing vocal module 30 can obtain the current playback timestamp of the audio, align the audio and backing vocal tracks in time, and start superimposing the audio and backing vocals at the next moment after the current playback timestamp.

[0091] Preferably, the accompaniment module 30 can also support the user to select the time to add accompaniment sounds. For example, when the user turns on the accompaniment mode to sing, the accompaniment module 30 can record the current playback timestamp in real time according to the playing accompaniment track comp, where the start timestamp is 0. If the user needs to sing along at timestamp t, the accompaniment module 30 can jump the processed human voice track human' to the position of timestamp t, complete the time alignment of the processed human voice track human' and the accompaniment track comp, and then starting from time t+1, make the audio played at each time point generated by superimposing the processed human voice track human' and the accompaniment track comp.

[0092] In this embodiment, the preset conditions include the user's pause time exceeding a preset first time threshold T1, and / or the proportion of the user singing wrong notes within a preset second time threshold T2 exceeding a preset proportion value. The accompaniment module 30 detects whether the user's singing data meets the preset conditions in the following ways:

[0093] The user's singing data is preprocessed to obtain processed audio data. The user's pitch k_u at each moment of the processed audio data is identified. When the detection results for the user's pitch k_u are empty for multiple consecutive moments, and the total time value corresponding to the multiple consecutive moments with empty detection results exceeds the first time threshold T1, it is determined that the user's singing data meets the preset conditions.

[0094] And / or, preprocess the user's singing data to obtain processed audio data, identify the user's pitch k_u at each moment of the processed audio data, and obtain the standard pitch k_v of the accompaniment at each moment. The moment when the user's pitch k_u does not match the standard pitch k_v at the corresponding moment is determined as the wrong note moment. The number of moments corresponding to the second time threshold T2 is determined as the detection number. When the proportion of wrong note moments in multiple consecutive detection moments exceeds the preset proportion value, it is determined that the user's singing data meets the preset conditions.

[0095] Specifically, the preprocessing of the user's singing data may include noise reduction processing through filtering and other noise reduction algorithms. The accompaniment module 30 may use the YIN algorithm and / or deep learning network to identify and process the user's pitch k_u at each moment of the audio data in real time.

[0096] Based on this, the backing vocal module 30 can also set preset cancellation conditions for canceling the addition of backing vocals to the audio; in an optional implementation, after the backing vocal module 30 adds backing vocals to the audio, if it detects that the user's singing data no longer meets the preset conditions, it can cancel the addition of backing vocals to the audio accordingly, that is, play the audio again based only on the adjusted accompaniment data.

[0097] Specifically, the acquisition module 10 can acquire voiceprint information through a trained voiceprint acquisition model. The acquisition module 10 may include:

[0098] The first training unit 11 is used to extract the vocal data from the singing data of multiple singers as the first training data input to the voiceprint acquisition model, and use the personnel number corresponding to each singer as a label, so that the voiceprint acquisition model can predict the personnel number corresponding to each vocal data in the first training data.

[0099] The first loop unit 12 is used to determine the difference between the predicted personnel number and the actual personnel number of the voiceprint acquisition model. The parameters of the voiceprint acquisition model are continuously optimized by the difference in number and the voiceprint acquisition model is retrained until the difference in number meets the preset first training requirement, and the training of the voiceprint acquisition model is completed.

[0100] The model truncation unit 13 is used to take the penultimate layer of the trained voiceprint acquisition model as the output layer, input the user's singing data into the trained voiceprint acquisition model, and output the user's voiceprint information through the output layer.

[0101] In this embodiment, the first training unit 11 can select singers who are evenly distributed in terms of gender, age, language, etc. to sing multiple specified songs to obtain singing data. Then, the voice data separated from the singing data corresponding to the same specified song can be used to train the voiceprint acquisition model.

[0102] The model truncation unit 13 generates the voiceprint acquisition model by truncating the model. After truncating the voiceprint acquisition model, the acquisition module 10 can input the user's singing data into the voiceprint acquisition model, so that the output layer of the voiceprint acquisition model outputs the user's voiceprint information accordingly.

[0103] Furthermore, the backing vocals are generated through a trained timbre conversion model. In this embodiment, the voiceprint acquisition model and the timbre conversion model are jointly trained in stages. The acquisition module 10 may also include:

[0104] The second training unit 14 is used to input the singing data of the first singer into the trained voiceprint acquisition model and obtain the voiceprint information of the first singer. The voiceprint information of the first singer and the singing data of the second singer are used as training data to input into the timbre conversion model so that the timbre conversion model can predict the singing data of the first singer.

[0105] The second loop unit 15 is used to determine the data difference between the predicted singing data and the actual singing data of the timbre conversion model. The parameters of the timbre conversion model are continuously optimized by the data difference and the timbre conversion model is retrained until the data difference meets the preset second training requirement, thus completing the training of the timbre conversion model.

[0106] Specifically, the second training unit 14 can use any singer as the first singer and another singer as the second singer. The singing data x_m of the first singer is input into the trained voiceprint acquisition model. At this time, the parameters of the voiceprint acquisition model are fixed and can output the voiceprint information of the first singer. Then, the singing data x_n of the second singer singing the same song and the voiceprint information of the first singer can be used as training data to input into the timbre conversion model, so that the timbre conversion model can predict the singing data of the first singer. The singing data x_m' predicted by the timbre conversion model is compared with the real singing data x_m, thereby realizing the calculation of loss for training.

[0107] Example 3

[0108] This embodiment provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the intelligent accompaniment method proposed in Embodiment 1 above.

[0109] It should be noted that computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0110] In summary, this invention proposes an intelligent accompaniment singing method and system. The method proposed in this solution can customize the pitch of a song for the user, adjust the accompaniment pitch to a suitable range for the user, ensure that the user can sing more easily, and selectively add backing vocals according to the user's singing situation, thereby guiding the user to adjust their singing style, improve their singing level, and enhance the user's singing experience.

[0111] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An intelligent accompaniment singing method, characterized in that, The method includes: In response to the acquisition command, the highest note sung by the user is recorded as the first high note, the lowest note is recorded as the first low note, and the user's voiceprint information is acquired; In response to a song selection command, the song selected based on the command is divided into vocal data and accompaniment data. The second high note and second low note corresponding to the vocal data are calculated. The transposition pitch is determined using the first high note, the second high note, the first low note, and the second low note. The accompaniment data is adjusted based on the transposition pitch, and audio is played based on the adjusted accompaniment data. The formula for determining the transposition pitch includes: N = (H' + L') / 2 - (H + L) / 2; where N represents the transposition pitch, H represents the first high note, H' represents the second high note, L represents the first low note, and L' represents the second low note. Acquire and continuously detect the user's singing data, and when the user's singing data meets preset conditions, add the accompaniment voice generated by the voiceprint information and the human voice data to the audio; The voiceprint information is obtained through a trained voiceprint acquisition model. The method for training the voiceprint acquisition model to acquire the user's voiceprint information includes: The vocal data from the singing data of multiple singers is extracted as the first training data and input into the voiceprint acquisition model. The personnel number corresponding to each singer is used as a label, so that the voiceprint acquisition model can predict the personnel number corresponding to each vocal data in the first training data. The difference between the predicted personnel number and the actual personnel number of the voiceprint acquisition model is determined. The parameters of the voiceprint acquisition model are continuously optimized based on the difference in number, and the voiceprint acquisition model is retrained until the difference in number meets the preset first training requirement, thus completing the training of the voiceprint acquisition model. The penultimate layer of the trained voiceprint acquisition model is used as the output layer, and the user's singing data is input into the trained voiceprint acquisition model. The user's voiceprint information is then output through the output layer. The backing vocals are generated using a trained timbre conversion model. The method for training the timbre conversion model includes: The singing data of the first singer is input into the trained voiceprint acquisition model to obtain the voiceprint information of the first singer. The voiceprint information of the first singer and the singing data of the second singer are used as training data and input into the timbre conversion model so that the timbre conversion model can predict the singing data of the first singer. The data difference between the predicted singing data and the actual singing data of the timbre conversion model is determined. The parameters of the timbre conversion model are continuously optimized based on the data difference, and the timbre conversion model is retrained until the data difference meets the preset second training requirement, thus completing the training of the timbre conversion model.

2. The intelligent accompaniment singing method according to claim 1, characterized in that, The preset conditions include the user's pause time exceeding a preset first time threshold. The method for detecting whether the user's singing data meets the preset conditions includes: The user's singing data is preprocessed to obtain processed audio data. The user's pitch at each moment is identified. When the detection result of the user's pitch is empty for multiple consecutive moments, and the total time value corresponding to the multiple consecutive moments with empty detection results exceeds the first time threshold, it is determined that the user's singing data meets the preset condition.

3. The intelligent accompaniment singing method according to claim 1, characterized in that, The preset conditions include the proportion of the user singing wrong notes exceeding a preset proportion value within a preset second time threshold. The method for detecting whether the user's singing data meets the preset conditions includes: The user's singing data is preprocessed to obtain processed audio data. The user's pitch at each moment is identified in the processed audio data, and the standard pitch of the accompaniment is obtained at each moment. The moment when the user's pitch does not match the standard pitch at the corresponding moment is determined as the wrong note moment. The number of moments corresponding to the second time threshold is determined as the detection quantity. When the proportion of wrong notes in multiple consecutive detection times exceeds the preset proportion value, it is determined that the user's singing data meets the preset condition.

4. The intelligent accompaniment singing method according to any one of claims 1-3, characterized in that, Adding backing vocals to the audio includes: Obtain the current playback timestamp of the audio, align the audio with the backing vocals in time, and start superimposing the audio and backing vocals at the next moment after the current playback timestamp.

5. The intelligent accompaniment singing method according to claim 1, characterized in that, The method further includes: In response to the acquisition command, determine the type of the acquisition command; If the acquisition instruction is of the first type, play the scale for the user to sing along, acquire the user's singing data, and determine the first high note and the first low note based on the singing data; If the acquisition command is of the second type, when responding to the song selection command for the first time, the audio is played based on the unadjusted accompaniment data to acquire the user's singing data, and the first high note and the first low note are determined based on the singing data.

6. An intelligent karaoke system, characterized in that, The system includes: The acquisition module is used to respond to the acquisition command, record the highest note sung by the user as the first high note, record the lowest note as the first low note, and acquire the user's voiceprint information; The playback module is used to respond to a song selection command, divide the song selected based on the song selection command into vocal data and accompaniment data, calculate the second high note and the second low note corresponding to the vocal data, determine the transposition pitch number using the first high note, the second high note, the first low note, and the second low note, adjust the accompaniment data based on the transposition pitch number, and play audio based on the adjusted accompaniment data; the formula for determining the transposition pitch number includes: N = (H' + L') / 2 - (H + L) / 2; where N represents the transposition pitch number, H represents the first high note, H' represents the second high note, L represents the first low note, and L' represents the second low note; The backing vocal module is used to acquire and continuously detect the user's singing data, and when the user's singing data meets preset conditions, it adds the backing vocals generated by the voiceprint information and the human voice data to the audio. The voiceprint information is obtained through a trained voiceprint acquisition model, and the accompaniment voice is generated through a trained timbre conversion model. The acquisition module includes: The first training unit is used to extract the vocal data from the singing data of multiple singers as the first training data and input it into the voiceprint acquisition model, and use the personnel number corresponding to each singer as a label, so that the voiceprint acquisition model can predict the personnel number corresponding to each vocal data in the first training data. The first loop unit is used to determine the difference between the predicted personnel number and the actual personnel number of the voiceprint acquisition model, continuously optimize the parameters of the voiceprint acquisition model based on the difference in number, and retrain the voiceprint acquisition model until the difference in number meets the preset first training requirement, thus completing the training of the voiceprint acquisition model. The model truncation unit is used to take the penultimate layer of the trained voiceprint acquisition model as the output layer, input the user's singing data into the trained voiceprint acquisition model, and output the user's voiceprint information through the output layer. The second training unit is used to input the singing data of the first singer into the trained voiceprint acquisition model and obtain the voiceprint information of the first singer, and input the voiceprint information of the first singer and the singing data of the second singer as training data into the timbre conversion model, so that the timbre conversion model can predict the singing data of the first singer. The second loop unit is used to determine the data difference between the predicted singing data and the actual singing data of the timbre conversion model, continuously optimize the parameters of the timbre conversion model based on the data difference, and retrain the timbre conversion model until the data difference meets the preset second training requirement, thus completing the training of the timbre conversion model.

7. A computer-readable storage medium, characterized in that, It stores executable instructions for use by a processor to implement the intelligent accompaniment singing method as described in any one of claims 1-5.

Images