A healing music generation method and system based on AIGC

Abstract

The application discloses a kind of based on AIGC's healing music generation method and system, comprising: (1) obtaining the music preference text description and specified audio segment corresponding to user;(2) the text description obtained is extracted, and based on audio sequence generation model obtains a decodable audio format sequence;(3) the audio segment obtained is encoded, and based on self-attention mechanism obtains another decodable audio format sequence;(4) the two decodable audio format sequences obtained above are processed using cross-attention mechanism and decoded, to obtain the audio segment for healing;(5) based on user feedback evaluation, fine-tuning training step (2) in audio sequence generation model, using large language model updates text description in step (1), and repeats above steps (1) ~ (4), until the audio segment that user is satisfied is obtained.The healing music generated by the application can meet the personalized needs of patients, and has a perfect feedback mechanism.

Description

Technical Field

[0001] The present invention relates to the technical field of music generation, and in particular to a method and system for generating healing music based on AIGC. Background Art

[0002] In recent years, with the serious development and spread of mental illness, mental illness treatment methods have received widespread attention. Music therapy, as an important mental illness treatment method, aims to mobilize patients' emotions through music, change their psychological state, reduce the side effects of drug treatment, and improve patient compliance.

[0003] A Chinese patent document with publication number CN116484043A discloses a music therapy method and system, including a cloud platform for providing public information and collecting data, an APP, a mini-program or a song-ordering device connected to the cloud platform signal. The method includes: obtaining music data listened to by the user; collecting scene data of the user using a microphone; collecting basic psychological information data of the user; calculating the user's emotional degree based on the collected music data, scene data and basic psychological information data, pre-processing, evaluating and classifying the user's mental state respectively, and generating an evaluation log; recommending songs for music therapy to the user based on the evaluation log, and the user independently selecting the corresponding songs for therapy.

[0004] A Chinese patent document with publication number CN117064402A discloses a method for identifying the feedback of music on brain waves. By constructing an EEG signal data ontology and analyzing the power spectral density characteristics of EEG signals under different emotions, real-time emotion recognition and music therapy can be achieved.

[0005] However, traditional music therapy methods have many limitations. For example, the audio categories involved in music therapy are single, ignoring the uniqueness and diversity of patients' needs and preferences; the feedback mechanism is not perfect, and it is impossible to adjust the content and elements of music therapy in a timely manner based on the patient's feedback after treatment, which limits the therapeutic effect of music therapy for mental illness. Summary of the Invention

[0006] The present invention provides a method and system for generating healing music based on AIGC, so that the generated healing music can meet the personalized needs of patients and has a complete feedback mechanism.

[0007] A healing music generation method based on AIGC, comprising the following steps:

[0008] (1) Obtain the user's corresponding music preference text description and specified audio clip;

[0009] (2) Feature extraction is performed on the acquired text description to obtain text feature encoding that contains both semantic and emotional information, and a decodable audio format sequence is obtained based on the audio sequence generation model;

[0010] (3) Encode the acquired audio clips and obtain another decodable audio format sequence based on the self-attention mechanism;

[0011] (4) Using the cross-attention mechanism to process the two decodable audio format sequences obtained above and decode them to obtain a fused audio clip for healing;

[0012] (5) Based on user feedback, fine-tune the audio sequence generation model in the training step (2), use the large language model to update the text description in step (1), and repeat the above steps (1) to (4) until an audio clip that satisfies the user is obtained.

[0013] In step (1), the music preference text description includes the music preference keywords input by the user and a combination of the music preference keywords selected by the user from existing music preference keywords;

[0014] The designated audio clips include music clips that the user actively expresses and creates during music therapy, and audio clips that the user selects from existing music clips.

[0015] The specific process of step (2) is:

[0016] The text is processed based on the text encoding model to obtain the text semantic feature encoding; the text is processed based on the sentiment dictionary to obtain the text sentiment feature encoding; the text semantic feature encoding and the text sentiment feature encoding are concatenated to obtain a text feature encoding that contains both semantic and sentiment information; then, based on the audio sequence generation model, the text feature encoding is used to predict a decodable audio format sequence.

[0017] The process of processing text based on the text encoding model to obtain text semantic feature encoding is as follows:

[0018] The BERT model is used to preprocess the input text. For each word, a word feature vector, a position feature vector, and a distinguishing feature vector are obtained. These three vectors are summed to obtain the input vector, which is the encoded text information.

[0019] Then, a multi-layer stacked encoder structure is used to simultaneously focus on the relationship between each word vector and other word vectors in the input sequence, capturing contextual information from the entire sequence; by calculating the relevance of the target word with all the words in the sentence and normalizing the relevance, the relevance and the encoding of all words are weighted and summed to obtain the text semantic feature encoding of the target word.

[0020] The audio sequence generation model uses a residual method to cascade multiple vector quantization modules, performs residual vector quantization on the text feature encoding, and sums the outputs of each vector quantization module to obtain a decodable audio format sequence;

[0021] Among them, the vector quantization module matches the continuous input vector with a pre-defined codebook, finds the closest codeword, and replaces the original input, effectively converting a large amount of continuous data into a smaller discrete representation, realizing the discretization of continuous vectors, and obtaining discrete sequence data; the residual method is to encode the text features in the output of the first vector quantization module and perform a residual calculation on the original input, and use the result as the input of the second vector quantization module, and so on, thereby improving the reconstruction accuracy and reducing the codebook size requirement.

[0022] The specific process of step (3) is:

[0023] The audio is processed based on the audio encoder to obtain audio feature encoding. The audio feature encoding obtained by processing based on the self-attention mechanism dynamically processes features at different time scales, including pitch, speaking speed, and emotional changes, and finally obtains a decodable audio format sequence.

[0024] The specific process of step (4) is:

[0025] A cross-attention mechanism is used to process decodable audio format sequences of two different modalities, allowing the sequence of one modality to extract information from the other modality. By fusing the semantic and emotional features of the text and the musical features of the audio, a decodable audio format sequence with complementary features is obtained. The audio decoder is then used to process the above-mentioned fused decodable audio format sequence to reconstruct the audio clip used for healing.

[0026] In step (5), the audio sequence generation model trained in step (2) is fine-tuned, specifically:

[0027] Obtain the patient's evaluation of the audio after receiving music therapy, generate the audio and evaluation scores as metadata, establish a feedback dataset, and based on this dataset, use the LoRA fine-tuning method to fine-tune the model weight parameters.

[0028] In step (5), the text description in step (1) is updated using the large language model, specifically:

[0029] The large language model is used to analyze direct user feedback on the defects of the audio generated by the current model, summarize the possible reasons why the original text description performs poorly on the current task, and output a series of reflection results. A reflection prompt template is defined to guide the large language model to reflect on itself.

[0030] After obtaining a series of reflection results, we use the large language model to continue analyzing the improvement strategies for the above results, and generate more suitable text descriptions based on the original text descriptions to re-participate in audio generation.

[0031] A healing music generation system based on AIGC includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned healing music generation method.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The present invention is based on the AIGC healing music generation method, which guides the model to complete the decoding and generation of healing music fragments through the decodable audio format sequence obtained by encoding and fusing text descriptions and audio fragments, and uses the feedback mechanism to improve the adaptability of the model audio generation, thereby improving the therapeutic effect of music therapy for mild mental illness. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of a method for generating healing music based on AIGC according to an embodiment of the present invention;

[0035] Figure 2 A schematic diagram of a flow chart for encoding a text description in one embodiment of the present invention;

[0036] Figure 3 1. A schematic diagram of a process for encoding an audio clip in one embodiment of the present invention;

[0037] Figure 4 The figure is a flowchart of fusing a decodable audio format sequence and decoding to generate audio in one embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be described in further detail below with reference to the accompanying drawings and examples. It should be noted that the following examples are intended to facilitate understanding of the present invention and do not have any limiting effect on the present invention.

[0039] like Figure 1 As shown, a healing music generation method based on AIGC includes the following steps:

[0040] S101: Obtain a music preference text description and a specified audio clip.

[0041] The music preference text description can be a music preference keyword input by the user or a combination of existing music preference keywords selected by the user. Its content mainly includes: music type (music genre, such as classical music, jazz, pop, folk, electronic music, etc.), emotional association (emotional response to specific music, such as relaxation, happiness, calmness, or relief of stress, anxiety, anger, etc.), rhythm and speed (for example: slow rhythm, soft or strong rhythm, fast beat), musical elements (for example: specific instrument names) and other important information.

[0042] The designated audio clips can be music clips that the patient actively expresses and creates during music therapy, or they can be audio selected by the user from existing music clips. These music clips may include important information such as the patient's emotional expression, subconscious associations (deep memories, unresolved emotional conflicts or personal experiences), and the patient's familiarity and comfort.

[0043] S102: extract features from the text and obtain a decodable audio format sequence based on an audio sequence generation model.

[0044] Based on the text encoding model, the text is processed to obtain the text semantic feature encoding; based on the sentiment dictionary, the text is processed to obtain the text sentiment feature encoding, and the above two feature encodings are spliced ​​to obtain the text feature encoding that contains both semantic and sentiment information. Then, based on the audio sequence generation model, the text feature encoding is used to predict the decodable audio format sequence, which facilitates the subsequent completion of encoding fusion and decoding processing.

[0045] The audio sequence generation model performs residual vector quantization on text feature encoding, which involves cascading multiple vector quantization modules using a residual method and summing the outputs of each module to obtain a decodable audio format sequence. The vector quantization module can match continuous input vectors with predefined discrete vectors (codebooks), find the closest codeword, and replace the original input, effectively converting large amounts of continuous data into smaller discrete representations, discretizing continuous vectors, and obtaining discrete sequence data. The residual method encodes text features in the output of the first vector quantization module and performs a residual calculation on the original input. The result is used as the input to the second vector quantization module, and so on, thereby improving reconstruction accuracy and reducing codebook size requirements.

[0046] S103: Encode the audio and obtain an encodable audio format sequence based on the self-attention mechanism.

[0047] Based on an audio encoder, process the audio to obtain an audio feature encoding, and based on the self-attention mechanism, process the audio feature encoding to leverage its global modeling advantage, capture long-range dependencies, and dynamically process features at different time scales (such as pitch, speech rate, emotional changes, etc.), resulting in a sequence of decodable audio formats.

[0048] S104, Use the cross-attention mechanism to process the above two sequences of decodable audio formats and decode them to obtain the generated audio segment.

[0049] Use the cross-attention mechanism to process the above two sequences of decodable audio formats, fuse the semantic and emotional features of the text and the musical features of the audio (such as pitch, rhythm), etc., to obtain a sequence of decodable audio formats with fused complementary features, and then use an audio decoder to process the above fused sequence of decodable audio formats to reconstruct the audio segment generated by the model.

[0050] S105, Based on user feedback evaluation, fine-tune the training of the audio sequence generation model to optimize the model's prediction effect.

[0051] [[ID=1']]Obtain the patient's evaluation of the audio after receiving music therapy, use "generated audio + evaluation score" as metadata to establish a feedback dataset, and based on the above dataset, use the LoRA fine-tuning method to fine-tune the model weight parameters. The principle is to use low-dimensional matrix factorization to approximate the high-dimensional parameter matrix of the model, that is, add a bypass structure to the network. The bypass is the multiplication of two matrices A and B, where the dimension of matrix A is d×r, and the dimension of matrix B is r×d, where r << d. Then the number of parameters of this bypass will be much smaller than the original network parameter W0 (d×d). In the specific training process, freeze the original network parameter W0, only train the bypass parameters A and B, and finally directly combine the bypass parameters and the original parameters to achieve low-cost and high-efficiency fine-tuning training of the model, that is:

[0052] W0x + ΔWx = W0x + BAx

[0053] Where ΔW represents the parameter update in the model fine-tuning process. While reducing the training cost, the above method further improves the audio sequence generation model's ability to understand keyword text information when predicting a sequence of decodable audio formats using text feature encoding.

[0054] S106, Based on user feedback evaluation, use a large language model to update the text description to obtain a more suitable text description.

[0055] Use a large language model to analyze the user's direct feedback on the defects of the audio generated by the current model, summarize the possible reasons for the poor performance of the original text description in the current task, and output a series of reflection results.

[0056] It should be noted that in the provided text, there seems to be an error in the label of the line starting with "Obtain the patient's evaluation...". It should probably be instead of [[ID=1']]. This has been corrected in the translation for consistency.A reflection prompt template is defined to guide the large language model to reflect on itself. The template contains a task description, defect feedback, and detailed instructions for requesting reflection. For example:

[0057] "I am writing a text prompt for the text-to-audio task. The current prompt is: {original text description}. However, the audio generated based on this prompt has these problems: {user feedback defects}. Please summarize the reasons for the above problems in the generated audio and use <start>and <end>Segment each reflection result.”

[0058] After obtaining a series of reflection results, we used the large language model to continue analyzing improvement strategies for the above results, and generated more suitable text descriptions based on the original text descriptions to re-participate in audio generation, thereby solving the defects of user feedback and improving the model's performance in audio generation tasks.

[0059] For example, when the original text is described as "soft ambient music, mainly piano, accompanied by gentle rain sounds, suitable for listening in quiet time", users report that they "fail to resonate, lack immersion, and feel sad after listening". Referring to the above template, the main reasons for reflecting on the output using the large language model are:

[0060] <start>The resulting audio lacks resonance, likely because the audio lacks clarity in its emotional delivery, resulting in the music's melody and orchestration failing to evoke an emotional response in the listener. The combination of a soft piano and the sound of rain should create a warm or calming atmosphere, but perhaps the melody is too bland or monotonous to form an emotional connection. <end>

[0061] <start>The lack of immersion in the audio may be due to the poorly controlled volume balance between the background rain and the piano. Rain should be a background element, enhancing the atmosphere rather than distracting from it. The lack of detail in the sound design, along with its lack of spatiality and depth, may leave the listener feeling somewhat monotonous and unable to fully immerse themselves. <end>

[0062] <start>Audio that makes users feel sad may be due to piano tunes or chord choices that are too low or slow. While soft music is generally relaxing, dark tones or excessive use of minor keys and harmonies can easily bring negative emotional associations and induce sadness. <end>

[0063] Based on these results, a large language model was used to regenerate a more suitable text description: "A soft and light piano melody, accompanied by the gentle sound of rain, creates a warm and relaxing atmosphere. The piano tone is mainly in major, combined with the harmonious background sound of rain, highlighting the sense of tranquility without losing vitality, aiming to resonate with the audience and create a comfortable and immersive listening experience." This achieves the update of the text description and improves the audio generation effect.

[0064] The above-mentioned healing music generation method based on AIGC first obtains the patient's music preference text description and the patient's specified audio clip, then performs feature extraction on the above text, and obtains a decodable audio format sequence based on text features based on the audio sequence generation model, and encodes the audio, and obtains a decodable audio format sequence based on audio features based on the self-attention mechanism, and then uses the cross-attention mechanism to process the above two decodable audio format sequences and decode them to obtain the generated audio clip, and uses the above audio clip to complete the patient's music therapy. At the same time, based on user feedback evaluation, fine-tune the training audio sequence generation model and optimize the model prediction effect. In addition, based on user feedback evaluation, the large language model is used to update the text description to obtain a more suitable text description of music preferences. The healing music generation method based on AIGC of this application will guide the model to complete the decoding and generation of healing music clips through the decodable audio format sequence obtained by encoding and fusing the text description and audio clip respectively, and use the feedback mechanism to improve the adaptability of the model audio generation, thereby improving the therapeutic effect of music therapy for mild mental illness.

[0065] Based on the above embodiment, Figure 2 The steps of extracting text features and obtaining a decodable audio format sequence based on the audio sequence generation model are decomposed and refined. Figure 2 As shown, the implementation process includes the following:

[0066] S201, based on the text encoding model, processing the text to obtain text semantic feature encoding.

[0067] After obtaining the text sentence input, the BERT model is first used to preprocess the input sentence. For each word, the word feature vector Et, position feature vector Ep and distinction feature vector Es are obtained respectively. These three vectors are summed to obtain the input vector Et, that is, the encoded text information.

[0068] Et=Et+Ep+Es

[0069] A multi-layered encoder architecture then simultaneously focuses on the relationship between each word vector and the other word vectors in the input sequence, capturing contextual information from the entire sequence. The target word's textual semantic feature encoding, Fs, is obtained by calculating the relevance of the target word with all words in the sentence, normalizing the relevance, and taking a weighted sum of the relevance and the encoding of all words.

[0070] S202: Based on the sentiment dictionary, the text is processed to obtain a text sentiment feature code.

[0071] Load the sentiment dictionary (such as the Canadian National Research Council Sentiment Dictionary) to obtain the association information between words and emotions, and perform word segmentation on the input text. For each word in the text, look up its association with each emotion category (such as anger, expectation, disgust, fear, joy, etc.) in the sentiment dictionary, generate the corresponding sentiment embedding vector, and aggregate the sentiment vectors of each word to obtain the text sentiment feature encoding Fe.

[0072] S203: Concatenate the two feature codes to obtain a text feature code.

[0073] The text semantic feature coding and text emotional feature coding are concatenated to obtain the text feature coding (including semantics and emotion) F, where the text semantic feature coding dimension of a single word is Ds and the text emotional feature coding dimension is De. Then the dimension D of the text feature coding (including semantics and emotion) is Ds+De.

[0074] S204: Based on the audio sequence generation model, the decodable audio format sequence is predicted using the above text feature encoding.

[0075] The audio sequence generation model is used to predict a discrete decodable audio format sequence Tt based on the text feature encoding (including semantics and emotions) F.

[0076] It can be understood that this embodiment provides a possible implementation method for obtaining a decodable audio format sequence containing text semantics and emotional information, which lays the foundation for subsequent feature fusion of text and audio.

[0077] Based on the above embodiment, Figure 3 The audio is encoded and the steps of obtaining the encodable audio format sequence based on the self-attention mechanism are decomposed and refined. Figure 3 As shown, the implementation process includes the following:

[0078] S301: Based on an audio encoder, process the audio to obtain audio feature code.

[0079] After obtaining the audio input, the audio is encoded using the autoencoder of the high-fidelity neural audio compressor, and the representation compression of the audio signal is completed with the help of the natural abstract feature extraction ability of the neural network, resulting in an audio feature code Fm with higher dimension and lower frame rate.

[0080] S302: Process the audio feature encoding based on a self-attention mechanism to obtain a decodable audio format sequence.

[0081] The above-mentioned audio feature encoding Fm is a discrete representation extracted from the original audio signal. They represent different segments or frames of audio and contain the encoding of audio features within a certain time segment. Applying the self-attention mechanism on the above sequence allows a single discrete representation to be updated in combination with the information of the entire audio sequence, thereby capturing the long-term dependencies and complex structures in the audio signal, and improving the sequence's ability to express features at different time scales (such as pitch, speaking speed, emotional changes, etc.). The main principle is: for each discrete representation Xi (the length of the audio sequence is n, and i can take any value from 1 to n), three vectors are generated through linear transformation, namely: query vector Qi, key vector Ki and value vector Vi, where Wq, Wk, and Wv are learnable weight matrices, namely:

[0082] Qi=WqXi Ki=WkXi Vi=WvXi

[0083] Perform dot product between the query vector Qi of each discrete representation Xi and the key vector Kj of all other discrete representations Xj (the audio sequence length is n, and j takes all values ​​from 1 to n in turn), calculate their similarity, and normalize them to obtain the attention weight, that is:

[0084]

[0085] Finally, the value vectors Vj of all other discrete representations Xj are weighted and summed according to the above attention weights, and the values ​​of all discrete representations Xi are updated, that is:

[0086]

[0087] At this time, the new discrete representation Xi not only contains its own information, but also integrates the information in the entire audio sequence, and finally obtains a decodable audio format sequence Tm containing audio features.

[0088] It can be understood that this embodiment provides a possible implementation method for obtaining a decodable audio format sequence containing audio feature information, which lays the foundation for subsequent feature fusion of text and audio.

[0089] Based on the above embodiment, Figure 4 The steps of using the cross-attention mechanism to process the above two decodable audio format sequences and decode them to obtain the generated audio clips are decomposed and refined. Figure 4 As shown, the implementation process includes the following:

[0090] S401, using a cross attention mechanism to process the above two decodable audio format sequences to obtain a fused decodable audio format sequence.

[0091] The cross-attention mechanism allows sequences from one modality to extract information from another modality when processing sequences from different modalities. For two encodable audio format sequences from text and audio respectively, one of the sequences is used as the query and the sequence of the other modality is used as the key and value. By calculating the similarity between the query modality and the key modality, the representation of the value modality is weighted and combined to generate a new representation of the query modality.

[0092] For example, for each discrete representation Si of a text sequence (the length of the text sequence is n s , i can be from 1 to n s Any value) generates the query vector Qi through linear transformation, for each discrete representation Xj of the audio sequence (the length of the audio sequence is n x , j is taken from 1 to n x All values ​​of ) generate key vector Kj and value vector Vj, where Wq, Wk, Wv are learnable weight matrices, namely:

[0093] Qi=WqSi Kj=WkXj Vj=WvXj

[0094] Then calculate the similarity between the query vector Qi from the text sequence and the key vector Kj from the audio sequence, and calculate the attention weight by dot product and normalization, that is:

[0095]

[0096] Finally, the value vectors Vj of all other discrete representations Xj are weighted and summed according to the above attention weights, and the values ​​of all discrete representations Si are updated, that is:

[0097]

[0098] In this example, the cross-attention from text to audio and the cross-attention from audio to text are calculated separately. That is, the query vector is calculated using the decodable audio format sequence Tt from the text and the decodable audio format sequence Tm from the audio, respectively. The values ​​of Tt and Tm are updated, and they are fused by weighted sum and splicing to obtain the fused decodable audio format sequence T, thereby realizing the mutual fusion and enhancement of the two modal features.

[0099] S402: Process the fused decodable audio format sequence using an audio decoder to obtain an audio file generated by the model.

[0100] After obtaining the fused decodable audio format sequence T, the sequence is decoded using the decoder of the high-fidelity neural audio compressor to restore the discrete sequence information into continuous audio features, and these features are converted into the final audio waveform through the reconstruction network to obtain the generated audio file.

[0101] It can be understood that this embodiment provides a possible implementation method of fusing two decodable audio format sequences and decoding and converting them into audio to obtain the final audio output.

[0102] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the scope of protection of the present invention.< / end> < / start> < / end> < / start> < / end> < / start> < / end> < / start>

Claims

1. A healing music generation method based on AIGC, characterized in that: The following steps are involved: (1) Obtain the user's corresponding music preference text description and specified audio clip; (2) Feature extraction is performed on the acquired text description to obtain text feature encoding that contains both semantic and emotional information, and a decodable audio format sequence is obtained based on the audio sequence generation model; (3) Encode the acquired audio clips and obtain another decodable audio format sequence based on the self-attention mechanism; (4) Using the cross-attention mechanism to process the two decodable audio format sequences obtained above and decode them to obtain a fused audio clip for healing; (5) Based on user feedback, fine-tune the audio sequence generation model in the training step (2), use the large language model to update the text description in step (1), and repeat the above steps (1) to (4) until an audio clip that satisfies the user is obtained.

2. The AIGC-based healing music generation method according to claim 1, characterized in that: In step (1), the music preference text description includes the music preference keywords input by the user and a combination of the music preference keywords selected by the user from existing music preference keywords; The designated audio clips include music clips that the user actively expresses and creates during music therapy, and audio clips that the user selects from existing music clips.

3. The AIGC-based healing music generation method according to claim 1, characterized in that: The specific process of step (2) is: Process the text based on the text encoding model to obtain the text semantic feature encoding; The text is processed based on the sentiment dictionary to obtain the text sentiment feature encoding; the text semantic feature encoding and the text sentiment feature encoding are spliced ​​together to obtain a text feature encoding that contains both semantic and sentiment information; then, based on the audio sequence generation model, the text feature encoding is used to predict a decodable audio format sequence.

4. The AIGC-based healing music generation method according to claim 3, characterized in that: The process of processing text based on the text encoding model to obtain text semantic feature encoding is as follows: The BERT model is used to preprocess the input text. For each word, a word feature vector, a position feature vector, and a distinguishing feature vector are obtained. These three vectors are summed to obtain the input vector, which is the encoded text information. Then, a multi-layer stacked encoder structure is used to simultaneously focus on the relationship between each word vector and other word vectors in the input sequence, capturing contextual information from the entire sequence; by calculating the relevance of the target word with all the words in the sentence and normalizing the relevance, the relevance and the encoding of all words are weighted and summed to obtain the text semantic feature encoding of the target word.

5. The AIGC-based healing music generation method according to claim 1, characterized in that: In step (2), the audio sequence generation model cascades multiple vector quantization modules in a residual manner, performs residual vector quantization on the text feature encoding, and sums the outputs of each vector quantization module to obtain a decodable audio format sequence; Among them, the vector quantization module matches the continuous input vector with a pre-defined codebook, finds the closest codeword, and replaces the original input, effectively converting a large amount of continuous data into a smaller discrete representation, realizing the discretization of continuous vectors, and obtaining discrete sequence data; the residual method is to encode the text features in the output of the first vector quantization module and perform a residual calculation on the original input, and use the result as the input of the second vector quantization module, and so on, thereby improving the reconstruction accuracy and reducing the codebook size requirement.

6. The AIGC-based healing music generation method according to claim 1, characterized in that: The specific process of step (3) is: The audio is processed based on the audio encoder to obtain audio feature encoding. The audio feature encoding obtained by processing based on the self-attention mechanism dynamically processes features at different time scales, including pitch, speaking speed, and emotional changes, and finally obtains a decodable audio format sequence.

7. The AIGC-based healing music generation method according to claim 1, characterized in that: The specific process of step (4) is: A cross-attention mechanism is used to process decodable audio format sequences of two different modalities, allowing the sequence of one modality to extract information from the other modality. By fusing the semantic and emotional features of the text and the musical features of the audio, a decodable audio format sequence with complementary features is obtained. The audio decoder is then used to process the above-mentioned fused decodable audio format sequence to reconstruct the audio clip used for healing.

8. The AIGC-based healing music generation method according to claim 1, characterized in that: In step (5), the audio sequence generation model trained in step (2) is fine-tuned, specifically: Obtain the patient's evaluation of the audio after receiving music therapy, generate the audio and evaluation scores as metadata, establish a feedback dataset, and based on this dataset, use the LoRA fine-tuning method to fine-tune the model weight parameters.

9. The AIGC-based healing music generation method according to claim 1, characterized in that: In step (5), the text description in step (1) is updated using the large language model, specifically: The large language model is used to analyze direct user feedback on the defects of the audio generated by the current model, summarize the possible reasons why the original text description performs poorly on the current task, and output a series of reflection results. A reflection prompt template is defined to guide the large language model to reflect on itself. After obtaining a series of reflection results, we use the large language model to continue analyzing the improvement strategies for the above results, and generate more suitable text descriptions based on the original text descriptions to re-participate in audio generation.

10. A healing music generation system based on AIGC, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the healing music generation method according to any one of claims 1 to 9.

Images