Data determination method and apparatus, electronic device, storage medium, and product

Abstract

The present disclosure provides a data determination method and device, electronic equipment, storage medium and product. The method comprises: obtaining original dry sound data, the original dry sound data comprising original melody and original lyrics; performing transformation processing on the original melody to obtain at least two groups of transformed melodies, and / or performing rewriting processing on the original lyrics to obtain at least two groups of rewritten lyrics; inputting the original lyrics and the transformed melodies into a first target expert model to generate first expanded dry sound data, and / or inputting the original melody and the rewritten lyrics into a second target expert model to generate second expanded dry sound data; and determining target dry sound data based on the first expanded dry sound data and the second expanded dry sound data. The data scale expansion can be realized on the basis of a small amount of original dry sound data, and the generalization performance of the song synthesis model is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a data determination method, apparatus, electronic device, storage medium, and product. Background Technology

[0002] The training dry audio data for related vocal synthesis models mainly relies on human composition, lyrics writing, singer recording, and manual annotation. The high labor costs and long production cycle result in a limited amount of dry audio data, leading to poor generalization performance of vocal synthesis models. Summary of the Invention

[0003] This disclosure provides a data determination method, apparatus, electronic device, storage medium, and product to solve the problem in related technologies where the limited amount of dry audio data that can be generated leads to poor generalization performance of singing synthesis models.

[0004] A first aspect of this disclosure provides a data determination method, the method comprising:

[0005] Obtain raw audio data, which includes the original melody and original lyrics; Transform the original melody to obtain at least two sets of transformed melodies, and / or rewrite the original lyrics to obtain at least two sets of rewritten lyrics; The original lyrics and the transformed melody are input into the first target expert model to generate the first expanded dry vocal data, and / or the original melody and the rewritten lyrics are input into the second target expert model to generate the second expanded dry vocal data; The target dry sound data is determined based on the first and second augmented dry sound data.

[0006] In one embodiment, before inputting the original lyrics and the transformed melody into a first target expert model to generate first expanded dry vocal data, or before inputting the original melody and the rewritten lyrics into a second target expert model to generate second expanded dry vocal data, the method provided in this disclosure includes: Obtain a first training dataset and a second training dataset. The first training dataset contains dry vocal data corresponding to fixed lyrics and at least two different melodies. The second training dataset contains dry vocal data corresponding to fixed melodies and at least two different lyrics. A lyrics expert model is trained based on the first training dataset to obtain a first target expert model, and a melody expert model is trained based on the second training dataset to obtain a second target expert model.

[0007] In one embodiment, the original melody is transformed to obtain at least two sets of transformed melodies, including: Obtain the melody constraint parameters of the raw dry audio data. The melody constraint parameters include rhythm dimension parameters, mode dimension parameters, and style dimension parameters. Input the melody constraint parameters and the original melody into the target language model to generate at least two sets of transformed melodies.

[0008] In one embodiment, the original lyrics are rewritten to obtain at least two sets of rewritten lyrics, including: Obtain the lyrics constraint parameters of the raw dry audio data. The lyrics constraint parameters include style dimension, language dimension and rhythm dimension. Input the lyrics constraint parameters and the original lyrics into the target language model to generate at least two sets of rewritten lyrics.

[0009] In one embodiment, the original lyrics and the transformed melody are input into a first target expert model to generate first augmented dry vocal data, including: Input the original lyrics and the transformed melody into the first target expert model to generate at least two sets of first candidate dry vocal data; The first augmented dry data is determined based on the audio quality score of the first candidate dry data in each group.

[0010] In one embodiment, the original melody and the rewritten lyrics are input into a second target expert model to generate second augmented dry vocal data, including: Input the original melody and the rewritten lyrics into the second objective expert model to generate at least two sets of second candidate dry vocal data; The second augmented dry data is determined based on the audio quality score of each group of second candidate dry data.

[0011] A second aspect of this disclosure provides a data determination apparatus, the apparatus comprising: The acquisition unit is used to acquire raw dry audio data, which includes the original melody and original lyrics. The processing unit is used to transform the original melody to obtain at least two sets of transformed melodies, and / or to rewrite the original lyrics to obtain at least two sets of rewritten lyrics. The generation unit is used to input the original lyrics and the transformed melody into a first target expert model to generate first expanded dry vocal data, and / or input the original melody and the rewritten lyrics into a second target expert model to generate second expanded dry vocal data; The determining unit is used to determine the target dry sound data based on the first expanded dry sound data and the second expanded dry sound data.

[0012] A third aspect of this disclosure provides an electronic device comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods described in the first aspect of this disclosure.

[0013] A fourth aspect of this disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in the first aspect of this disclosure.

[0014] A fifth aspect of this disclosure provides a computer program product including a computer program that, when executed by a processor, implements the methods described in the first aspect of this disclosure.

[0015] In summary, this disclosure proposes a data determination method, which includes: acquiring raw dry vocal data, the raw dry vocal data including raw melody and raw lyrics; transforming the raw melody to obtain at least two sets of transformed melodies, and / or rewriting the raw lyrics to obtain at least two sets of rewritten lyrics; inputting the raw lyrics and transformed melody into a first target expert model to generate first expanded dry vocal data, and / or inputting the raw melody and rewritten lyrics into a second target expert model to generate second expanded dry vocal data; and determining target dry vocal data based on the first expanded dry vocal data and the second expanded dry vocal data.

[0016] According to the solution provided in this disclosure, by realizing melody transformation and lyric rewriting based on the original dry vocal data, and automatically generating expanded dry vocal data in combination with expert models, and determining the target dry vocal data based on the expanded dry vocal data, it is possible to achieve large-scale data expansion on the basis of a small amount of original dry vocal data, thereby improving the generalization performance of the singing synthesis model.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0019] Figure 1 A flowchart illustrating a data determination method provided in an embodiment of this disclosure; Figure 2 A schematic diagram of a melody expert model provided in an embodiment of this disclosure; Figure 3 A flowchart illustrating a method for generating first augmented dry acoustic data provided in an embodiment of this disclosure; Figure 4 A flowchart illustrating a method for generating second augmented dry acoustic data provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of the structure of a data determination device provided in an embodiment of the present disclosure; Figure 6 This is a schematic diagram of the hardware composition structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0020] Embodiments of this disclosure are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0021] To facilitate a better understanding of the technical solutions described in the embodiments of this disclosure by those skilled in the art, the technical terms in the embodiments of this disclosure are explained as follows before introducing them.

[0022] Singing synthesis algorithms refer to converting user-input melodies and lyrics into dry audio, which refers to pure human vocal audio without background music.

[0023] Existing data determination methods typically rely on human annotators with music expertise to manually generate training data through a recording and annotation process. A typical data annotation process involves: pre-setting melody and lyrics libraries, recording the singer's vocals, manually annotating and aligning, and obtaining triples (melody, lyrics, vocals). While this type of method can obtain relatively high-quality annotated data, the scale of the data constructed is generally small due to its high labor costs and long cycle.

[0024] Furthermore, the data duration of mainstream public datasets (such as M4Singer and GTSinger) is less than 100 hours, which is insufficient to support the training requirements of singing voice synthesis models for high generalization and strong robustness.

[0025] The training dry audio data for related vocal synthesis models mainly relies on human composition, lyrics writing, singer recording, and manual annotation. The high labor costs and long production cycle result in a limited amount of dry audio data, leading to poor generalization performance of vocal synthesis models.

[0026] To address the shortcomings of related technologies, this disclosure achieves melody transformation and lyric rewriting based on the original dry vocal data, automatically generates expanded dry vocal data using an expert model, and determines the target dry vocal data based on the expanded dry vocal data. This enables large-scale data expansion based on a small amount of original dry vocal data, thereby improving the generalization performance of the vocal synthesis model.

[0027] The present disclosure will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0028] The data determination method provided in this disclosure can be applied to scenarios involving the determination of training datasets for artificial intelligence (AI) vocal synthesis, such as in scenarios involving virtual singers, AI ringtones, and AIGC vocal content production. The execution entity of this method can be a music application backend data processing server or a cloud-based AI computing platform.

[0029] like Figure 1 As shown, Figure 1 This is a flowchart illustrating a data determination method provided in an embodiment of this disclosure. The data determination method provided in this embodiment includes the following steps: Step 101: Obtain raw dry vocal data, which includes the original melody and original lyrics; In one embodiment, the raw dry audio data is a clean human voice audio data source that has been professionally recorded, noise-reduced, and finely labeled, without any accompaniment music.

[0030] In one embodiment, the original melody is audio melody information that has been parameterized. The original melody may include melody parameters such as beats per minute (BPM), key, note pitch, and note start and end duration; the original lyrics are text lyrics that have been semantically and phonetically verified and have a fixed sentence structure.

[0031] In one embodiment, raw vocal data can be generated by selecting professional singers who meet the singing criteria to record dry vocals, extracting melody parameters with the help of audio software, and simultaneously organizing and proofreading the corresponding lyrics. For copyrighted complete songs, raw vocal data can also be obtained by separating the accompaniment from the song.

[0032] Step 102: Transform the original melody to obtain at least two sets of transformed melodies, and / or rewrite the original lyrics to obtain at least two sets of rewritten lyrics; In one embodiment, melody transformation processing refers to using the original melody as a basis, while retaining the core features of the original melody, changing the rhythm, mode, and style to generate multiple sets of melodies adapted to fixed lyrics.

[0033] In one embodiment, lyrics rewriting refers to generating multiple versions of differentiated lyrics based on the original lyrics by configuring aesthetically pleasing constraints such as style, language, and rhythm, while preserving the semantics of the original lyrics.

[0034] In one embodiment, the original melody can be transformed by inputting the rhythm, mode, and style constraint parameters of the original melody into a large language model, resulting in at least two sets of transformed melodies.

[0035] In one embodiment, the original melody can also be transformed by using fixed music theory rules such as transposition, duration scaling, and the addition or subtraction of ornaments to obtain at least two sets of transformed melodies.

[0036] In one embodiment, at least two sets of rewritten lyrics can be generated by inputting the style, language, and word count constraints of the original lyrics into a large language model; alternatively, based on the initially generated rewritten lyrics, a second rhyme scheme and semantic preservation can be added using a large language model to generate the final rewritten lyrics.

[0037] In one embodiment, the original lyrics can also be rewritten by fixing the original sentence structure and replacing the content words to obtain at least two sets of rewritten lyrics.

[0038] Step 103: Input the original lyrics and the transformed melody into the first target expert model to generate the first expanded dry vocal data, and / or input the original melody and the rewritten lyrics into the second target expert model to generate the second expanded dry vocal data; In one embodiment, the first target expert model is a lyrics expert model pre-trained using multiple melody samples with the same lyrics, and the second target expert model is a melody expert model pre-trained using multiple lyrics samples with the same melody. The lyrics expert model and the melody expert model can use deep neural network structures such as Diffusion Transformer (DiT) and U-Net (UNet).

[0039] In one embodiment, the dry audio data is pure human voice audio data after removing background music. The first expanded dry audio data is synthesized by combining fixed original lyrics with various modified melodies, and the second expanded dry audio data is synthesized by combining fixed original melodies with various rewritten lyrics.

[0040] In one embodiment, the original lyrics can be paired sequentially with each group of transformed melodies and input into the first target expert model one pair at a time to generate the first expanded dry vocal data; alternatively, the original lyrics can be paired with all the transformed melodies and input into the first target expert model in parallel to generate the first expanded dry vocal data; alternatively, the transformed melodies can be grouped according to style and mode and input into the first target expert model in batches to generate the first expanded dry vocal data.

[0041] In one embodiment, inputting the original melody and the rewritten lyrics into the second target expert model to generate the second expanded dry vocal data can be achieved in the same way as inputting the original lyrics and the transformed melody into the first target expert model to generate the first expanded dry vocal data, which will not be described again here.

[0042] Step 104: Determine the target dry sound data based on the first and second augmented dry sound data.

[0043] In one embodiment, the target dry audio data is the dry audio data that is ultimately used for training the singing synthesis model after the first augmented dry audio data and the second augmented dry audio data have undergone audio quality screening, summarization and integration.

[0044] In one embodiment, the target dry sound data can be one set or at least two sets. The number of target dry sound data is not limited in this application.

[0045] In one embodiment, a preset audio quality scoring algorithm can be used to score the sound quality of each piece of dry audio data in the first and second expanded dry audio data. Dry audio data with a sound quality score higher than a preset score can be selected as target dry audio data. The preset score is a pre-set threshold value, such as a preset score of 80 points. Dry audio data with a sound quality score higher than 80 points can be selected as target dry audio data.

[0046] By acquiring raw dry vocal data containing the original melody and lyrics, performing variant processing on the melody and lyrics respectively, and using two trained expert models to generate augmented dry vocal data, the target dry vocal data can be determined. This approach can solve the problems of high cost of manual annotation and insufficient dataset size in existing models, thereby improving the generalization performance of the vocal synthesis model.

[0047] In one embodiment, before inputting the original lyrics and the transformed melody into a first target expert model to generate first expanded dry vocal data, or before inputting the original melody and the rewritten lyrics into a second target expert model to generate second expanded dry vocal data, the data determination method includes: Obtain a first training dataset and a second training dataset. The first training dataset contains dry vocal data corresponding to fixed lyrics and at least two different melodies. The second training dataset contains dry vocal data corresponding to fixed melodies and at least two different lyrics. In one embodiment, the first training dataset is a set of dry vocal samples formed by combining a fixed single lyric with multiple versions of differentiated melodies and professionally recorded and labeled, and the second training dataset is a set of dry vocal samples formed by combining a fixed single melody with multiple versions of differentiated lyrices and professionally recorded and labeled.

[0048] In one embodiment, materials can be selected from a compliant copyright music library or original content to create a first training dataset consisting of multiple melodic samples with the same lyrics and a second training dataset consisting of multiple lyric samples with the same melody.

[0049] In one embodiment, when constructing the first training dataset, the selected lyrics can be labeled with the number of characters, vowels, tones, and sentence pause parameters. The composer then creates differentiated melodies with multiple rhythms, modes, and styles. After music theory and lyrics matching verification, the corresponding singers record dry vocals. After the labeling verification is completed, the first training dataset is obtained.

[0050] For example, select lyric fragments that are semantically complete and have comprehensive pronunciation coverage (including core vowels such as a, i, u, ü, and all tones of the first, second, third, and fourth tones, without rare characters or ambiguous expressions). These can be selected from publicly available copyrighted lyric libraries or created independently (they must conform to everyday language scenarios, such as emotional expression or life narrative). At the same time, the "melody compatibility" of the lyrics needs to be verified—through evaluation by music theory experts, it must be ensured that the lyric sentence structure (such as 3-8 characters per line) can adapt to different rhythmic patterns (such as 4 / 4 time, 3 / 4 time) and vocal ranges (such as soprano C4-C6, baritone A2-A4), to avoid limiting melody creation due to a fixed lyric structure. Use text annotation tools (such as LabelStudio and Notion) to annotate the selected lyrics and clarify key information: the number of characters in each line of lyrics, the position of the core vowels (such as "wind" (eng) and "warm" (uan) in "spring breeze is warm"), the tone distribution (annotate the specific tone value of each character, such as "spring" is level tone (1) and "wind" is level tone (1)), and the semantic pauses (such as the pauses in "spring breeze / warm, summer rain / cool"), to form a standardized lyrics parameter document. Invite 2-3 music composers with over 5 years of experience to create multiple versions of melodies based on a fixed lyric parameter document. The melodies must cover three core differentiating dimensions: ① Rhythmic differences (e.g., 4 / 4 time signature lyrical rhythm, 3 / 4 time signature waltz rhythm, 6 / 8 time signature folk rhythm); ② Modal and range differences (e.g., C major soprano melody, A minor baritone melody, G major children's melody, with a range span of 8-12 degrees to ensure vocal performance); ③ Stylistic differences (e.g., pop, traditional Chinese style, folk style, matching the semantic meaning of the lyrics, such as adapting traditional Chinese style lyrics to a pentatonic scale melody). The final result should be 8-12 differentiated melody versions. A review panel consisting of two music theory experts and one vocal teacher was formed to conduct a double verification of the generated melodies: first, a lyrics-melody compatibility verification, checking whether the melody rhythm matches the pauses in the lyrics and whether the pitch fluctuations match the semantic and emotional content of the lyrics (e.g., lyrical lyrics should avoid rapid rhythms); second, a performance feasibility verification, testing whether the melody's range is within the comfortable range for human singing and whether the transitions between notes are smooth through piano playing, eliminating melody versions with poor compatibility or that are difficult to sing, and finally retaining 6-10 qualified melody versions, and generating a standardized melody parameter document (including BPM, mode, note duration, and pitch value) for each version. Based on the vocal range and style requirements of each melody version, corresponding professional singers are matched (e.g., soprano melodies are matched with singers with a C4-C6 vocal range, baritone melodies are matched with singers with an A2-A4 vocal range). The selection criteria are consistent with those for recording the same melody with multiple lyrics (more than 5 years of singing experience, pitch error ≤ 5 cents, rhythm deviation ≤ 10 milliseconds). Each melody version is recorded by one suitable singer to avoid sample quality defects caused by mismatch in singer vocal range.Before recording, provide the singer with a fixed lyrics text and a standardized parameter document for the corresponding melody, and arrange a 1-hour rehearsal time to ensure that the singer is familiar with the pronunciation of the lyrics and the rhythm of the melody; the recording adopts a whole-section recording and phrase-by-phrase supplementary recording mode: first record the whole melody 2-3 times, and then record the sections of each phrase of the melody that are prone to pitch or rhythm deviation (such as high notes, key changes) 3-5 times separately; the recording engineer uses audio analysis tools (such as Melodyne) to monitor pitch and rhythm in real time, and immediately re-records any sections with pitch deviation exceeding 5 cents, rhythm deviation exceeding 10 milliseconds, or unclear pronunciation; after the recording is completed, the singer, recording engineer, and composer jointly review it, and retain 1-2 optimal recording samples for each melody version.

[0051] In one embodiment, when constructing the second training dataset, the parameters of BPM, mode, note duration, and pitch of the selected melody can be first calibrated to form a standardized document. Then, multiple sets of lyrics covering multiple vowels and full tones are designed and screened by language experts. The dry audio is recorded by professional singers who meet the pitch and rhythm error indicators. After wavelet noise reduction, 20ms frame-by-frame audio annotation, and sampling quality inspection, the second training dataset is obtained.

[0052] For example, select melodic fragments with clear characteristics from publicly available copyrighted music libraries (e.g., 300 seconds in length, with a vocal range of 8-10 degrees, and containing typical rhythmic patterns of the verse / chorus), or create original melodies that conform to the rules of human vocal performance, ensuring that the melody is free of noise and key ambiguity, and has the stability for repeatable performance. Combine the selected fragments together, and have them verified by professional singers to ensure that the included notes and pitches cover all notes, including mainstream notes and note transitions. Use professional audio processing tools (such as Audacity, Cubase) to quantify the parameters of the selected melody, annotating key information including: tempo (BPM, accurate to 1 BPM), key (e.g., C major, A minor), note duration (in milliseconds, indicating the start and end times of each note), and pitch (in semitones, indicating the specific pitch value of each note, e.g., C4 corresponds to 60 semitones), forming a standardized melody parameter document to avoid melody deviations during subsequent recording. Design 10-15 sets of differentiated lyrics for everyday language scenarios (such as emotional expression and life description). The length of each set of lyrics should match the phrase structure of a fixed melody (e.g., 4 lines / segment, with the number of characters in each line corresponding to the number of beats in the melody). Ensure the lyrics contain no obscure characters, are unambiguous, and cover different vowels (e.g., a, i, u, ü) and tones (high level, rising, falling-rising, falling), covering common pronunciation combinations in human singing to avoid insufficient model generalization due to monotonous lyrics pronunciation. Invite 3 language professionals to review the designed lyrics, focusing on semantic coherence, pronunciation fluency, and fit with the melody. Eliminate lyrics that are semantically incoherent, difficult to pronounce, or do not match the rhythm of the melody, ultimately retaining 8-12 sets of lyrics that meet the requirements for recording. Select 2-3 singers with over 5 years of professional singing experience, pitch error ≤ 5 cents, and rhythm deviation ≤ 10 milliseconds to participate in the recording, ensuring that the singers can accurately match the pitch and rhythm of the fixed melody to avoid sample quality defects due to insufficient singing ability. Before recording, standardized melody parameter documents and lyrics text are provided to the singer to familiarize them with the melody rhythm and lyrics pronunciation. During recording, a one-line-per-sentence recording mode is adopted, with each line of lyrics recorded 3-5 times. The recording engineer monitors the pitch, rhythm, and pronunciation clarity in real time. Any segment with a pitch deviation exceeding 5 cents, a rhythm deviation exceeding 10 milliseconds, or unclear pronunciation is immediately re-recorded. After the entire recording is completed, the singer and recording engineer jointly review it to ensure that each segment meets the recording standards. Finally, 1-2 optimal recording samples are retained for each set of lyrics. Adaptive noise reduction algorithms (such as wavelet transform-based noise reduction methods) are used to denoise the dry audio, ensuring that the signal-to-noise ratio of the dry audio after noise reduction is ≥35dB, with no significant loss of sound quality. The dry audio samples are annotated frame by frame (20 milliseconds long) using audio annotation tools (such as LabelStudio and Praat). The annotation content includes MIDI note information, pitch, and vocal characteristics (tenor / soprano / bass / alto).30% of the labeled samples are randomly selected and their quality is verified by third-party professional audio evaluators. The labeling error (such as time axis deviation and fundamental frequency deviation) is calculated. The time axis labeling error is required to be ≤5 milliseconds and the fundamental frequency labeling error is required to be ≤2Hz. If the sampling pass rate is less than 95%, all samples are re-labeled and verified until the accuracy requirements are met.

[0053] For example, in the first training dataset, the same standardized melody is matched with 8 to 12 sets of clean dry audio corresponding to compliant lyrics, and in the second training dataset, the same standard lyrics are matched with 6 to 10 sets of clean dry audio corresponding to compliant melodies. All dry audio is real human singing audio with the accompaniment removed.

[0054] A lyrics expert model is trained based on the first training dataset to obtain a first target expert model, and a melody expert model is trained based on the second training dataset to obtain a second target expert model.

[0055] In one embodiment, when training the melody expert model, standardized melody notes, note durations, and lyrics are used as model inputs, and the Mel spectrum features corresponding to the recorded dry sound are used as model outputs. The model iteratively converges using the first training dataset, and a second target expert model is obtained after training.

[0056] In one embodiment, when training the lyrics expert model, fixed lyrics and various melody parameters are used as model inputs, and the corresponding dry sound Mel spectrum is used as model outputs. The network parameters are optimized based on the second training dataset to obtain the first target expert model.

[0057] In one embodiment, the model training process employs diffusion model training logic, which completes feature learning through Mel spectrum extraction, noise injection, multi-granularity conditional coding, multi-layer attention, and feedforward transformation.

[0058] Exemplary, in one embodiment, such as Figure 2 As shown, Figure 2 This is a schematic diagram of a melody expert model provided in an embodiment of this disclosure. Figure 2The overall network architecture diagram for the DiTSinger model training phase is presented, which can be divided into two parts: the overall left-hand link and the internal layer decomposition of the right-hand diffusion transformer module (DiTBlock). The model input includes three types of input data: fine-grained conditions, training waveform audio, and coarse-grained conditions. Fine-grained conditions include lyric scores and pitch information, while coarse-grained conditions include time steps and singer timbre information. The inputs are processed by their respective embedding layers. The fine-grained condition data is sent to the conditional encoder, and the training waveform is sequentially passed through a Mel spectrogram extractor and noise injection before being fed into the token downsampling module. The coarse-grained condition data undergoes feature mapping via a multilayer perceptron (MLP). All three feature paths are then converged into the multilayer diffusion transformer module (DiTBlock). The DiTBlock output features are fed into the Token UpSampler module, which ultimately outputs predicted noise tokens. The right side shows a magnified view of the DiTBlock's internal structure, arranged from bottom to top with three branching layers. Each layer contains adaptive layer normalization and scaling factors, corresponding to hyperparameters α1 / α2 / α3, γ1 / γ2 / γ3, and β1 / β2 / β3. The bottom layer, based on fine-grained tokens, noisy Mel tokens, and coarse-grained tokens, first passes them through a multi-head self-attention module with rotational position encoding and QK-normalization (RoPE and QK-Norm). The middle layer connects to a masked multi-head cross-attention module with QK-normalization (Masked Multi-Head Cross-Attention with QK-Norm). The top layer is configured with a pointwise feedforward network. FeedForward), the results of each layer's computation are summed by adding the residuals, where Coarse-grained Tokens additionally access MLP to provide parameters for normalization at each layer.

[0059] This embodiment constructs a first training dataset and a second training dataset, and trains two types of dedicated expert models for different scenarios, so that the first target expert model can be adapted to generate vocals with different melodies based on fixed lyrics, and the second target expert model can be adapted to generate vocals with different lyrics based on fixed melodies.

[0060] In one embodiment, the original melody is transformed to obtain at least two sets of transformed melodies, including: Obtain the melody constraint parameters of the raw dry audio data. The melody constraint parameters include rhythm dimension parameters, mode dimension parameters, and style dimension parameters. In one embodiment, the melody constraint parameters are parameters extracted from the original melody of the original dry vocal data and used to constrain the boundaries of melody generation.

[0061] In one embodiment, the rhythm dimension parameter refers to melody beat-related data, which may include, but is not limited to, the original time signature, BPM baseline value, original note value ratios, and phrase segmentation points; the mode dimension parameter refers to melody pitch and music theory-related data, which may include, but is not limited to, the original mode type, the vocal range, and the range of key interval constraints in the melody; the style dimension parameter refers to genre-specific data, which may include, but is not limited to, genre labels such as pop, folk, and traditional Chinese style, as well as the corresponding genre's signature scale and melody progression rules.

[0062] In one embodiment, the original melody waveform and Musical Instrument Digital Interface (MIDI) information can be analyzed using professional audio analysis tools to extract the corresponding parameters of rhythm and mode.

[0063] In one embodiment, style dimension parameters can also be added based on manual music theory annotations to form melody constraint parameters.

[0064] For example, for a 4 / 4 time signature, BPM 120 original melody in C major with an ancient Chinese style, the following parameters are extracted: rhythm dimension parameters: time signature 4 / 4, reference BPM 120, eighth notes account for 60%; mode dimension parameters: C major, range C4-C6, retaining the core intervals of thirds and fifths; style dimension parameters: ancient Chinese style, using pentatonic scale writing rules. The above parameters together constitute the entire set of melody constraint parameters.

[0065] Input the melody constraint parameters and the original melody into the target language model to generate at least two sets of transformed melodies.

[0066] In one embodiment, the target language model is a large language model with music theory generation capabilities.

[0067] In one embodiment, the original MIDI data of the original melody and melody constraint parameters are input into the target language model. The output differentiation can be achieved by adjusting parameters such as the model control temperature. Candidate melodies are generated in parallel by combining fixed music theory transformation rules. After the candidate melodies are generated, music theory compliance verification, lyric adaptation verification and sampling manual review are performed in sequence to eliminate non-compliant segments. Finally, at least two sets of differentiated transformed melodies that meet the constraint requirements are retained.

[0068] For example, output differentiation can be achieved by adjusting parameters such as temperature in the model. Specifically, each output of the target language model represents a set of probabilities. If there are 20,000 possible outputs, then each output will contain 20,000 probability numbers, which sum to 1. By adjusting the model's temperature coefficient, the probability distribution can be further adjusted. For instance, by dividing all probabilities by a temperature coefficient, when the temperature coefficient is less than 1, the originally larger probabilities become significantly larger, thus increasing the likelihood of them being sampled. When the temperature coefficient is greater than 1, such as when the temperature coefficient is 100, all probabilities are reduced by a factor of 100, narrowing the probability gap. For example, if two numbers were originally 1 and 2, after being reduced by a factor of 100, they become 0.01 and 0.02, making their sampling probabilities roughly the same.

[0069] This embodiment limits the generation range by splitting constraints such as rhythm, mode, and style, and realizes automated melody variant generation based on the target language model. This ensures that the transformed melody fits the core features of the original melody, while also diversifying the melody content and efficiently expanding the melody material.

[0070] For example, determine the rhythmic transformation parameters, including time signature conversion (e.g., 4 / 4 to 3 / 4), BPM fluctuation range (original BPM ± 20), and note value ratio adjustment (e.g., changing a rhythm dominated by eighth notes to a combination of sixteenth and quarter notes, with specific note duration percentages noted); specify the mode conversion rules (e.g., C major to A minor, G major to E minor), while limiting the pitch range (e.g., the vocal comfort zone C4-C6), and marking key interval relationships (e.g., maintaining the core intervals of the original melody in thirds and fifths); input the MIDI parameters of the original melody (including note pitch, duration, and velocity value) and the above constraint parameters into the large language model, generating 3 sets of basic melody variants with different constraint combinations in the first round; in the second round, add instructions for each variant with a melody fluency ≥ 90% and a similarity to the core motif of the original melody of 40%-60%, deriving 2-3 optimized variants, and using the model's built-in music theory verification module to filter out candidate variants without music theory errors. Deterministic transformations are performed based on a music theory rule base, including: ① rhythmic scaling (scaling note values ​​proportionally while maintaining rhythmic structure); ② modal shifting (transposing the melody as a whole to different pitches in the same key); ③ ornamentation addition (inserting appoggiaturas, trills, and other ornamentation patterns at long notes). Each rule generates 2-3 variants to ensure complementarity with the model's generated results. Music analysis tools (such as Sibelius and MuseScore) are used to check the modal consistency, interval rationality (eliminating unreasonable leaps exceeding an octave), and rhythmic logic of the variations (avoiding chaotic off-beat accents), with a pass rate of ≥95%. The generated melody variations are matched with fixed lyrics, and variations with serious rhythmic discrepancies with lyrical phrasing are eliminated by calculating the corresponding ratio of the number of melody notes to the number of lyrics (ideally 1:1 to 2:1). 30% of the candidate variations are randomly selected, and the music producer evaluates the accuracy of stylistic features (such as whether the ancient style variation contains pentatonic scale features) and auditory fluency. Finally, 8-12 versions of melody variations with significant differences and meeting the quality standards are retained to form a diverse melody variation library.

[0071] In one embodiment, the original lyrics are rewritten to obtain at least two sets of rewritten lyrics, including: Obtain the lyrics constraint parameters of the raw dry audio data. The lyrics constraint parameters include style dimension, language dimension and rhythm dimension. In one embodiment, the lyrics constraint parameters are original lyrics extracted based on the original dry vocal data, and are used to constrain the boundary of the large model rewriting.

[0072] In one embodiment, the style dimension parameter refers to the limiting information related to the lyric writing style. The style dimension parameter may include, but is not limited to, style tags such as pop, folk, and traditional Chinese style, and corresponding style word constraints; the language dimension parameter refers to the limiting information related to the language of the text. The language dimension parameter may include, but is not limited to, language types such as Chinese, English, and Japanese, and corresponding language grammar rules and prosody norms; the rhythm dimension parameter refers to the text constraint information that adapts to the melody rhythm. The rhythm dimension parameter may include, but is not limited to, the original number of characters per sentence, the allowed fluctuation range of the number of characters, and the requirements for sentence pause points.

[0073] In one embodiment, lyric constraint parameters can be manually annotated and generated based on the original lyrics and the accompanying melody beat information.

[0074] In one embodiment, the lyrics constraint parameters can also be obtained by automatically associating and extracting them with text parsing tools and melody parameters.

[0075] For example, the original 7-character Chinese classical style lyrics have the following style dimension parameters: classical style, limited to misty rain, pavilion-like imagery; language dimension parameter: Chinese; rhythm dimension parameter: the number of characters per line can fluctuate between 6 and 8. The lyrics constraint parameters are formed by combining the above content.

[0076] Input the lyrics constraint parameters and the original lyrics into the target language model to generate at least two sets of rewritten lyrics.

[0077] In one embodiment, the original lyrics and lyric constraint parameters belong to the target language model and are used as prompt words for the target language model. The first round of text generation is performed by configuring parameters such as temperature coefficient of multiple models. Then, semantic retention and rhyme change instructions are added for a second iteration of generation. Candidate lyrics are screened, and 30% of the samples are randomly selected from the candidate lyrics for manual review. Texts that do not match the melody and rhythm are removed, and finally at least two sets of rewritten lyrics that meet the differentiation criteria are obtained.

[0078] For example, diverse control parameters are explicitly embedded in the model prompts, including style dimensions (such as pop, folk, and traditional Chinese style, which require annotation of core style feature words, such as traditional Chinese style requiring the inclusion of imagery such as "misty rain" and "pavilion"), language dimensions (supporting bilingual or multilingual conversion such as Chinese-English and Chinese-Japanese, and annotating grammatical rules and cultural adaptation requirements, such as Japanese lyrics requiring conformity to the "five-seven tone" rhyme scheme), and rhythm dimensions (specifying the range of word count fluctuations per sentence, such as allowing 6-8 word changes after rewriting if the original sentence has 7 words, and annotating the pause positions within the sentence), to ensure that the model generation accurately matches diverse needs. In the first round of generation, three sets of different constraint parameters are input for the same original lyrics (the richness of the output content can be adjusted by controlling parameters such as temperature) to obtain three versions of basic rewritten text. In the second round, based on the results of the first round, supplementary instructions such as "semantic retention ≥ 80%" and "rhyme adjustment" (e.g., changing from "a rhyme" to "i rhyme") are added to each version of the text to further generate 2-3 versions of derivative text. Finally, the model's built-in diversity assessment module (DeepSeek can perform this function) is used to filter out lyrics with significant differences. The melody and rhythm parameters (such as BPM and phrase division) corresponding to the original lyrics are simultaneously input into the model, requiring the number of words in the rewritten lyrics to match the rhythm (e.g., when BPM=120, the number of words in each line of lyrics must correspond to the 8 eighth notes in 4 / 4 time) to avoid rhythmic disconnect. At the same time, 30% of the samples are manually sampled to ensure the compatibility of style, language and melody, ultimately forming a high-quality rewritten lyrics library covering multiple styles, languages ​​and rhythms.

[0079] This embodiment limits the scope of rewriting by setting lyric constraint parameters such as style, language, and rhythm, and uses a language model to generate multiple versions of lyrics in batches. This ensures that the rewritten content fits the core meaning and original melody of the original lyrics, while also enabling the diversification and expansion of lyric materials.

[0080] In one embodiment, the original lyrics and the transformed melody are input into a first target expert model to generate first augmented dry vocal data, including: Input the original lyrics and the transformed melody into the first target expert model to generate at least two sets of first candidate dry vocal data; In one embodiment, the first target expert model is a lyrics expert model trained based on a first training dataset, which can receive different melody inputs and output corresponding human voice audio under fixed lyrics conditions.

[0081] In one embodiment, the original lyrics are fixed standard lyrics text, and the transformed melody is a set of differentiated melody materials obtained through parameter constraints and compliance screening.

[0082] In one embodiment, the dry audio data is audio data with the accompaniment removed and only the pure human voice audio retained. The first candidate dry audio data is the dry audio data output by the model inference after the same set of original lyrics are combined with a single transformed melody.

[0083] In one embodiment, each set of transformed melodies and original lyrics are sequentially input into the first target expert model for forward inference. Each set of inputs generates multiple audio tracks with different details, resulting in multiple sets of first candidate dry audio data.

[0084] For example, a fixed original lyric is paired with 5 different melodies. Each set of inputs is used by the model to generate 3 first candidate dry vocal data. All dry vocal data are aggregated to form the first candidate dry vocal data.

[0085] The first augmented dry data is determined based on the audio quality score of the first candidate dry data in each group.

[0086] In one embodiment, the audio quality score is used to determine the quality of the singing performance of the dry vocal data. The scoring dimensions may include indicators such as pitch error, rhythm deviation, audio signal-to-noise ratio, and vocal fluency.

[0087] In one embodiment, a pre-trained audio quality scoring model can be used to score the first candidate dry audio data respectively to obtain an audio quality score for each group of first candidate dry audio data; alternatively, the audio quality score for each group of first candidate dry audio data can be calculated by weighting the pitch, rhythm, and sound quality according to preset scoring rules.

[0088] In one embodiment, a quality score threshold is preset, and first candidate dry sound data with a score greater than the preset threshold is selected. The first candidate dry sound data that passes the selection is used as the first augmented dry sound data.

[0089] This embodiment generates multiple candidate dry audio samples in batches using a model, then selects high-quality samples based on audio quality scores, and removes samples with sound quality defects to ensure that the final first augmented dry audio data has stable and reliable audio quality.

[0090] In one embodiment, the original melody and the rewritten lyrics are input into a second target expert model to generate second augmented dry vocal data, including: Input the original melody and the rewritten lyrics into the second objective expert model to generate at least two sets of second candidate dry vocal data; In one embodiment, the second target expert model is a melody expert model trained based on a second training dataset. The second target expert model is able to adapt to different input lyrics and synthesize corresponding human voice audio under fixed melody conditions.

[0091] In one embodiment, the original melody is a standard melody material with fixed parameters, and the rewritten lyrics are multi-version differentiated lyric texts that have undergone multi-dimensional constraints, verification and screening.

[0092] In one embodiment, the second candidate dry vocal data is multiple candidate dry vocal data generated by model inference after fixing the original melody and combining it with a single rewritten lyric.

[0093] In one embodiment, the original melody and the rewritten lyrics are paired and combined one by one, and then input into the second target expert model for inference calculation. Each pair of inputs can generate multiple dry vocal data with different details, forming the second candidate dry vocal data.

[0094] For example, a single fixed original melody is paired with 6 different rewritten lyrics, and each input is used by the model to generate 2 to 3 second candidate dry vocal datasets.

[0095] The second augmented dry data is determined based on the audio quality score of each group of second candidate dry data.

[0096] In one embodiment, a pre-trained audio quality scoring model can be used to score the second candidate dry audio data respectively to obtain an audio quality score for each group of second candidate dry audio data; alternatively, the audio quality score for each group of second candidate dry audio data can be calculated by weighting the pitch, rhythm, and sound quality according to preset scoring rules.

[0097] In one embodiment, a quality score threshold is preset, and second candidate dry sound data with scores greater than the preset threshold are selected. The qualified second candidate dry sound data is used as the second augmented dry sound data.

[0098] This embodiment utilizes a melody expert model to generate multiple candidate dry audio samples in batches, and combines audio quality scoring to complete the screening of high-quality samples, eliminating synthesized flawed audio, thus effectively ensuring the overall audio quality of the second augmented dry audio data.

[0099] In one embodiment, such as Figure 3 As shown, Figure 3 This is a flowchart illustrating a method for generating first augmented dry acoustic data according to an embodiment of the present disclosure. Figure 3The process can be divided into two stages: training the lyrics expert model on the left and expanding the melody data on the right. In the left-side model training stage, the original melody is fed into a large language model to generate multiple sets of rewritten melodies. The original lyrics and rewritten melodies are recorded by singers to obtain a small amount of data. Based on this small amount of data, the expert model is trained to complete the training of the lyrics expert model. In the right-side data expansion stage, the original lyrics (which can be the same as or different from the original lyrics in the training stage) and the original melody are input. The original melody generates multiple versions of differentiated rewritten melodies based on the large language model. The original lyrics are fixed, and each set of rewritten melodies is input into the trained lyrics expert model. The model automatically generates clean dry audio in batches, ultimately obtaining a large amount of data. This process first relies on a small amount of manually recorded material to complete the training of the dedicated lyrics expert model, and then uses the large language model to generate diverse melodies in batches and relies on the expert model to automatically synthesize audio. This can get rid of the shortcomings of relying on singers to record massive amounts of dry audio and achieve large-scale expansion of the training dataset at low cost.

[0100] In one embodiment, such as Figure 4 As shown, Figure 4 This is a flowchart illustrating a method for generating second augmented dry acoustic data according to an embodiment of the present disclosure. Figure 4 The process is divided into two parts: the left side is the training process of the melody expert model, and the right side is the expansion of lyrics data based on the trained melody expert model. In the training process, the original lyrics and melody are input first. The original lyrics are input into a large language model to obtain rewritten lyrics. The singer records a small amount of dry vocal data based on the original melody and the rewritten lyrics. The expert model is then trained to obtain a converged melody expert model. In the expansion process on the right, the original lyrics are input and multiple sets of rewritten lyrics are generated through the large language model. The original melody and each set of rewritten lyrics are input into the trained melody expert model, which automatically generates expanded dry vocal data in batches, ultimately forming a large amount of target dry vocal data. This can overcome the shortcomings of relying on singers to record massive amounts of dry vocal data and achieve large-scale expansion of the training dataset at low cost.

[0101] In summary, the solution provided in this public disclosure is as follows: By performing melody transformation and lyric rewriting based on the original dry vocal data, and automatically generating extended dry vocal data using an expert model, and determining the target dry vocal data based on the extended dry vocal data, large-scale data expansion can be achieved on a small amount of original dry vocal data, thereby improving the generalization performance of the vocal synthesis model.

[0102] To implement the data determination method provided in this disclosure, this disclosure also provides a data determination apparatus, such as... Figure 5 As shown. Figure 5This is a schematic diagram of a data determination device provided in an embodiment of the present disclosure. The data determination device 500 includes: Acquisition unit 501 is used to acquire raw dry vocal data, which includes the original melody and original lyrics; The processing unit 502 is used to transform the original melody to obtain at least two sets of transformed melodies, and / or to rewrite the original lyrics to obtain at least two sets of rewritten lyrics. The generation unit 503 is used to input the original lyrics and the transformed melody into a first target expert model to generate first expanded dry vocal data, and / or input the original melody and the rewritten lyrics into a second target expert model to generate second expanded dry vocal data; The determining unit 504 is used to determine the target dry sound data based on the first expanded dry sound data and the second expanded dry sound data.

[0103] In one embodiment, the data determination device 500 further includes a training unit, which is used for: Obtain a first training dataset and a second training dataset. The first training dataset contains dry vocal data corresponding to fixed lyrics and at least two different melodies. The second training dataset contains dry vocal data corresponding to fixed melodies and at least two different lyrics. A lyrics expert model is trained based on the first training dataset to obtain a first target expert model, and a melody expert model is trained based on the second training dataset to obtain a second target expert model.

[0104] In one embodiment, the processing unit 502 is specifically used for: Obtain the melody constraint parameters of the raw dry audio data. The melody constraint parameters include rhythm dimension parameters, mode dimension parameters, and style dimension parameters. Input the melody constraint parameters and the original melody into the target language model to generate at least two sets of transformed melodies.

[0105] In one embodiment, the processing unit 502 is specifically used for: Obtain the lyrics constraint parameters of the raw dry audio data. The lyrics constraint parameters include style dimension, language dimension and rhythm dimension. Input the lyrics constraint parameters and the original lyrics into the target language model to generate at least two sets of rewritten lyrics.

[0106] In one embodiment, the generation unit 503 is specifically used for: Input the original lyrics and the transformed melody into the first target expert model to generate at least two sets of first candidate dry vocal data; The first augmented dry data is determined based on the audio quality score of the first candidate dry data in each group.

[0107] In one embodiment, the generation unit 503 is specifically used for: Input the original melody and the rewritten lyrics into the second objective expert model to generate at least two sets of second candidate dry vocal data; The second augmented dry data is determined based on the audio quality score of each group of second candidate dry data.

[0108] It should be noted that the data determination device provided in the above embodiments is only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the data determination device can be divided into different program modules to complete all or part of the processing described above. In addition, the data determination device provided in the above embodiments and the data determination method provided in the embodiments of this disclosure belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0109] Figure 6 This is a schematic diagram of the hardware composition structure of the electronic device provided in the embodiments of this disclosure, such as... Figure 6 As shown, the electronic device 600 includes at least one processor 602; and a memory 601 communicatively connected to the at least one processor 602; wherein the memory 601 stores instructions executable by the at least one processor 602, the instructions being executed by the at least one processor 602 to implement the steps of the data determination method of the present disclosure embodiments.

[0110] Optionally, the electronic device may specifically be a data determining device in the embodiments of this application, and the electronic device may implement the corresponding processes implemented by the data determining device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.

[0111] It is understood that the electronic device also includes a communication interface 603. Various components in the electronic device are coupled together via a bus system 604. It is understood that the bus system 604 is used to implement communication between these components. In addition to a data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 6 The general designated all buses as Bus System 604.

[0112] It is understood that memory 601 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 601 described in this embodiment of the invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0113] The methods disclosed in the above embodiments can be applied to or implemented by processor 602. Processor 602 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 602 or by instructions in software form. Processor 602 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 602 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, specifically memory 601. Processor 602 reads information from memory 601 and, in conjunction with its hardware, completes the steps of the aforementioned methods.

[0114] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned method.

[0115] This disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, which are used to cause a computer to execute the steps of the data determination method of the present invention.

[0116] Optionally, the computer-readable storage medium can be applied to the data determining apparatus in the embodiments of this application, and the computer instructions cause the computer to execute the corresponding processes implemented by the data determining apparatus in the various methods of the embodiments of this application. For the sake of brevity, these will not be described in detail here.

[0117] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the data determination method provided in this embodiment of the invention.

[0118] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0119] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0120] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0121] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0122] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0123] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data determination method, characterized in that, include: Obtain raw audio data, which includes the original melody and original lyrics; The original melody is transformed to obtain at least two sets of transformed melodies, and / or the original lyrics are rewritten to obtain at least two sets of rewritten lyrics; The original lyrics and the transformed melody are input into a first target expert model to generate first expanded dry vocal data, and / or the original melody and the rewritten lyrics are input into a second target expert model to generate second expanded dry vocal data; Based on the first augmented dry sound data and the second augmented dry sound data, the target dry sound data is determined.

2. The method according to claim 1, characterized in that, Before inputting the original lyrics and the transformed melody into a first target expert model to generate first expanded dry vocal data, or inputting the original melody and the rewritten lyrics into a second target expert model to generate second expanded dry vocal data, the method includes: Obtain a first training dataset and a second training dataset. The first training dataset contains dry vocal data corresponding to fixed lyrics and at least two different melodies. The second training dataset contains dry vocal data corresponding to fixed melodies and at least two different lyrics. A lyrics expert model is trained based on the first training dataset to obtain a first target expert model, and a melody expert model is trained based on the second training dataset to obtain a second target expert model.

3. The method according to claim 1, characterized in that, The transformation process of the original melody yields at least two sets of transformed melodies, including: Obtain the melody constraint parameters of the original dry audio data, which include rhythm dimension parameters, mode dimension parameters, and style dimension parameters; The melody constraint parameters and the original melody are input into the target language model to generate at least two sets of transformed melodies.

4. The method according to claim 1, characterized in that, The rewriting process of the original lyrics yields at least two sets of rewritten lyrics, including: Obtain the lyrics constraint parameters of the raw dry audio data, the lyrics constraint parameters including style dimension, language dimension and rhythm dimension; Input the lyrics constraint parameters and the original lyrics into the target language model to generate at least two sets of rewritten lyrics.

5. The method according to claim 1, characterized in that, The step of inputting the original lyrics and the transformed melody into a first target expert model to generate first augmented dry vocal data includes: The original lyrics and the transformed melody are input into the first target expert model to generate at least two sets of first candidate dry vocal data. The first augmented dry data is determined based on the audio quality score of the first candidate dry data in each group.

6. The method according to claim 1, characterized in that, The step of inputting the original melody and the rewritten lyrics into the second target expert model to generate second augmented dry vocal data includes: The original melody and the rewritten lyrics are input into the second target expert model to generate at least two sets of second candidate dry vocal data. The second augmented dry data is determined based on the audio quality score of the second candidate dry data in each group.

7. A data determination device, characterized in that, include: An acquisition unit is used to acquire raw dry audio data, which includes the original melody and original lyrics; The processing unit is used to transform the original melody to obtain at least two sets of transformed melodies, and / or to rewrite the original lyrics to obtain at least two sets of rewritten lyrics. The generation unit is used to input the original lyrics and the transformed melody into a first target expert model to generate first expanded dry vocal data, and / or input the original melody and the rewritten lyrics into a second target expert model to generate second expanded dry vocal data; The determining unit is used to determine the target dry sound data based on the first expanded dry sound data and the second expanded dry sound data.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

Images