Method and system for generating healing music based on AIGC

Abstract

The invention discloses a method and a system for generating healing music based on AIGC. The method comprises the following steps: (1) acquiring music preference text description and a specified audio clip corresponding to a user; (2) performing feature extraction on the obtained text description, and obtaining a decodable audio format sequence based on an audio sequence generation model; (3) carrying out coding processing on the obtained audio clips, and obtaining another decodable audio format sequence based on a self-attention mechanism; (4) processing the obtained two decodable audio format sequences by using a cross attention mechanism and decoding to obtain audio clips for healing; and (5) based on user feedback evaluation, finely adjusting and training the audio sequence generation model in the step (2), updating the text description in the step (1) by using a large language model, and repeating the steps (1) to (4) until an audio clip satisfying the user is obtained. According to the invention, the generated healing music can meet individual requirements of patients, and a perfect feedback mechanism is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of music generation, and in particular to a method and system for generating healing music based on AIGC. Background Art

[0002] In recent years, as mental illness has become more serious, the treatment of mental illness has received widespread attention. Music therapy, as an important treatment for mental illness, aims to mobilize the patient's emotions through music, change the patient's psychological state, reduce the side effects of drug treatment, and improve the patient's compliance with treatment.

[0003] A Chinese patent document with publication number CN116484043A discloses a music therapy method and system thereof, including a cloud platform for providing public information and collecting data, an APP, a mini-program or a song-ordering device connected to the cloud platform by signal, the method comprising: obtaining music data listened to by a user; collecting scene data in which the user uses a microphone; collecting basic psychological information data of the user; calculating the user's emotional degree based on the collected music data, scene data and basic psychological information data, pre-processing, evaluating and classifying the user's mental state respectively, and generating an evaluation log; recommending songs for music therapy to the user based on the evaluation log, and the user independently selecting the corresponding song for therapy.

[0004] A Chinese patent document with publication number CN117064402A discloses a method for identifying the feedback of music on brain waves. By constructing an EEG signal data ontology and analyzing the power spectral density characteristics in EEG signals under different emotions, real-time emotion recognition and music therapy can be achieved.

[0005] However, traditional music therapy methods have many limitations. For example, the audio categories involved in music therapy are single, ignoring the uniqueness and diversity of patients' needs and preferences; the feedback mechanism is not perfect, and it is impossible to adjust the content and elements of music therapy in time according to the patient's feedback after treatment, which limits the therapeutic effect of music therapy for mental illness. Summary of the invention

[0006] The present invention provides a method and system for generating healing music based on AIGC, so that the generated healing music can meet the personalized needs of patients and has a complete feedback mechanism.

[0007] A healing music generation method based on AIGC, comprising the following steps:

[0008] (1) Obtain the user's corresponding music preference text description and specified audio clip;

[0009] (2) extracting features from the acquired text description to obtain text feature encoding that contains both semantic and emotional information, and obtaining a decodable audio format sequence based on an audio sequence generation model;

[0010] (3) Encode the acquired audio clips and obtain another decodable audio format sequence based on the self-attention mechanism;

[0011] (4) using a cross-attention mechanism to process the two decodable audio format sequences obtained above and decode them to obtain a fused audio clip for healing;

[0012] (5) Based on user feedback and evaluation, fine-tune the audio sequence generation model in the training step (2), use the large language model to update the text description in step (1), and repeat the above steps (1) to (4) until an audio clip that satisfies the user is obtained.

[0013] In step (1), the music preference text description includes a combination of music preference keywords input by the user and the existing music preference keywords selected by the user;

[0014] The designated audio clips include music clips that the user actively expresses and creates during music therapy, and audio clips that the user selects from existing music clips.

[0015] The specific process of step (2) is as follows:

[0016] The text is processed based on the text encoding model to obtain the text semantic feature encoding; the text is processed based on the sentiment dictionary to obtain the text sentiment feature encoding; the text semantic feature encoding and the text sentiment feature encoding are concatenated to obtain the text feature encoding that contains both semantic and sentiment information; then, based on the audio sequence generation model, the text feature encoding is used to predict and obtain a decodable audio format sequence.

[0017] The process of processing text based on the text encoding model to obtain the text semantic feature encoding is as follows:

[0018] The BERT model is used to preprocess the input text. For each word, the word feature vector, position feature vector, and distinguishing feature vector are obtained. The three vectors are summed to obtain the input vector, that is, the encoded text information.

[0019] Then, a multi-layer stacked encoder structure is used to simultaneously focus on the relationship between each word vector and other word vectors in the input sequence to capture contextual information from the entire sequence; by calculating the relevance of the target word with all the words in the sentence and normalizing the relevance, the relevance and the encoding of all words are weighted and summed to obtain the text semantic feature encoding of the target word.

[0020] The audio sequence generation model cascades multiple vector quantization modules in a residual manner, performs residual vector quantization on the text feature encoding, and sums the outputs of each vector quantization module to obtain a decodable audio format sequence;

[0021] Among them, the vector quantization module matches the continuous input vector with the pre-defined codebook, finds the closest codeword, and replaces the original input, effectively converting a large amount of continuous data into a smaller discrete representation, realizing the discretization of continuous vectors, and obtaining discrete sequence data; the residual method is to encode the text features in the output of the first vector quantization module and perform a residual calculation on the original input, and use the result as the input of the second vector quantization module, and so on, thereby improving the reconstruction accuracy and reducing the codebook size requirement.

[0022] The specific process of step (3) is as follows:

[0023] The audio is processed based on the audio encoder to obtain audio feature encoding. The audio feature encoding obtained by processing based on the self-attention mechanism dynamically processes features on different time scales, including pitch, speaking speed, and emotional changes, and finally obtains a decodable audio format sequence.

[0024] The specific process of step (4) is as follows:

[0025] The cross-attention mechanism is used to process decodable audio format sequences of two different modalities, allowing the sequence of one modality to extract information from the other modality. By fusing the semantic and emotional features of the text and the musical features of the audio, a decodable audio format sequence with complementary features is obtained. The audio decoder is then used to process the above-mentioned fused decodable audio format sequence to reconstruct the audio clip for healing.

[0026] In step (5), the audio sequence generation model in step (2) is fine-tuned, specifically:

[0027] The patient's evaluation of the audio after music therapy was obtained, and the generated audio and evaluation scores were used as metadata to establish a feedback dataset. Based on the dataset, the LoRA fine-tuning method was used to fine-tune the model weight parameters.

[0028] In step (5), the text description in step (1) is updated using the large language model, specifically:

[0029] Use the large language model to analyze the user's direct feedback on the defects of the audio generated by the current model, summarize the possible reasons why the original text description performs poorly on the current task, and output a series of reflection results; define a reflection prompt template to guide the large language model to reflect on itself;

[0030] After obtaining a series of reflection results, we use the large language model to continue analyzing the improvement strategies for the above results, and generate more suitable text descriptions based on the original text descriptions to re-participate in audio generation.

[0031] A healing music generation system based on AIGC includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the above-mentioned healing music generation method.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The healing music generation method based on AIGC of the present invention will guide the model to complete the decoding and generation of healing music fragments through the decodable audio format sequence obtained by encoding and fusing text descriptions and audio fragments respectively, and use the feedback mechanism to improve the adaptability of the model audio generation, thereby improving the therapeutic effect of music therapy for mild mental illness. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flow chart of a healing music generation method based on AIGC according to an embodiment of the present invention;

[0035] Figure 2 A schematic diagram of a flow chart of encoding a text description in one embodiment of the present invention;

[0036] Figure 3 A schematic diagram of a process of encoding an audio clip in one embodiment of the present invention;

[0037] Figure 4 The present invention is a flowchart of fusing a decodable audio format sequence and decoding to generate audio in one embodiment of the present invention. DETAILED DESCRIPTION

[0038] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0039] like Figure 1 As shown, a healing music generation method based on AIGC includes the following steps:

[0040] S101, obtaining a music preference text description and a specified audio segment.

[0041] The music preference text description can be the music preference keywords input by the user or a combination of the existing music preference keywords selected by the user. Its content mainly includes: music type (music genre, such as classical music, jazz, pop, folk, electronic music, etc.), emotional association (emotional response to specific music, such as relaxation, happiness, calmness, or relief of stress, anxiety, anger, etc.), rhythm and speed (for example: slower, soft or stronger, faster beats), music elements (for example: specific instrument names) and other important information.

[0042] The designated audio clip can be a music clip that the patient actively expresses and creates during music therapy, or it can be audio selected by the user from existing music clips. These music clips may include important information such as the patient's emotional expression, subconscious associations (deep memories, unresolved emotional conflicts or personal experiences), and the patient's familiarity and comfort.

[0043] S102, extracting features from the text, and obtaining a decodable audio format sequence based on an audio sequence generation model.

[0044] Based on the text encoding model, the text is processed to obtain the text semantic feature encoding; based on the sentiment dictionary, the text is processed to obtain the text sentiment feature encoding, and the above two feature encodings are concatenated to obtain the text feature encoding that contains both semantic and sentiment information. Then, based on the audio sequence generation model, the text feature encoding is used to predict the decodable audio format sequence, which is convenient for the subsequent encoding fusion and decoding processing.

[0045] The audio sequence generation model performs residual vector quantization on the text feature encoding, that is, multiple vector quantization modules are cascaded in a residual manner, and the output of each module is summed to obtain a decodable audio format sequence. The vector quantization module can match the continuous input vector with the pre-defined discrete vector (codebook), find the closest codeword, and replace the original input, effectively converting a large amount of continuous data into a smaller discrete representation, discretizing the continuous vector, and obtaining discrete sequence data. The residual method is to encode the text feature in the output of the first vector quantization module and perform a residual calculation on the original input, and use the result as the input of the second vector quantization module, and so on, thereby improving the reconstruction accuracy and reducing the codebook size requirement.

[0046] S103, encode the audio and obtain an encodable audio format sequence based on the self-attention mechanism.

[0047] Based on an audio encoder, the audio is processed to obtain an audio feature encoding, and based on the self-attention mechanism, the audio feature encoding is processed to leverage its global modeling advantage, capture long-range dependencies, and dynamically process features at different time scales (such as pitch, speech rate, emotional changes, etc.), resulting in a sequence of decodable audio formats.

[0048] S104. Use the cross-attention mechanism to process the above two sequences of decodable audio formats and decode them to obtain the generated audio segment.

[0049] Use the cross-attention mechanism to process the above two sequences of decodable audio formats, fuse the semantic and emotional features of the text and the musical features of the audio (such as pitch, rhythm), etc., to obtain a sequence of decodable audio formats after fusing complementary features, and then use an audio decoder to process the above fused sequence of decodable audio formats to reconstruct the audio segment generated by the model.

[0050] S105. Based on user feedback evaluation, fine-tune the training of the audio sequence generation model to optimize the model prediction effect.

[0051] Obtain the patient's evaluation of the audio after receiving music therapy, use "generated audio + evaluation score" as metadata to establish a feedback dataset, and based on the above dataset, use the LoRA fine-tuning method to fine-tune the model weight parameters. The principle is to use low-rank matrix factorization to approximate the high-dimensional parameter matrix of the model, that is, add a bypass structure to the network. The bypass is the multiplication of two matrices A and B, where the dimension of matrix A is d×r, and the dimension of matrix B is r×d, where r << d. Then the number of parameters of this bypass will be much smaller than the original network parameter W0 (d×d). In the specific training process, freeze the original network parameter W0, only train the bypass parameters A and B, and finally directly combine the bypass parameters and the original parameters to achieve low-cost and high-efficiency fine-tuning training of the model, that is:

[0052] W0x + ΔWx = W0x + BAx

[0053] Where ΔW represents the parameter update during the model fine-tuning. While the above method reduces the training cost, it further improves the audio sequence generation model's ability to understand keyword text information when predicting a sequence of decodable audio formats using text feature encoding.

[0054] S106. Based on user feedback evaluation, use a large language model to update the text description to obtain a more suitable text description.

[0055] Use a large language model to analyze the user's direct feedback on the defects of the audio generated by the current model, and summarize the possible reasons for the poor performance of the original text description in the current task, and output a series of reflection results.

[0056] A reflection prompt template is defined to guide the large language model to reflect on itself. The template contains task description, defect feedback, and detailed instructions for requesting reflection, for example:

[0057] “I am writing a text prompt for the text-to-audio task. The current prompt is: {original text description}. However, the generated audio based on this prompt has these problems: {user feedback defects}. Please summarize the reasons for the above problems in the generated audio and use <start>and <end>Segment each reflection result. ”

[0058] After obtaining a series of reflection results, we used the large language model to continue analyzing the improvement strategies for the above results, and generated more suitable text descriptions based on the original text descriptions to re-participate in audio generation, thereby solving the defects of user feedback and improving the model's performance in audio generation tasks.

[0059] For example, when the original text is described as "soft ambient music, mainly piano, with gentle rain sounds, suitable for listening in quiet times", users report that there is a problem of "failure to resonate, lack of immersion, and feeling sad after listening". Referring to the above template, the main reasons for reflecting on the output using the large language model are:

[0060] <start>The resulting audio lacks resonance, perhaps because the audio is not clear enough in terms of emotion, resulting in the music's melody and score not being able to evoke an emotional response from the listener. The combination of a soft piano and the sound of rain is supposed to create a warm or calming atmosphere, but perhaps the melody is too bland or monotonous to form an emotional connection. <end>

[0061] <start>The lack of immersion in the audio may be because the volume balance between the background rain and the piano is not well controlled. The rain sound should be a background element that enhances the atmosphere rather than distracting. The sound design is not detailed enough, lacking a sense of space and layers, which may make the listener feel a little monotonous and unable to be fully immersed. <end>

[0062] <start>Audio that makes users feel sad may be due to piano tunes or chord choices that are too low or slow. Although soft music is generally relaxing, if the tone is dark or too many minor keys and harmonies are used, it can easily bring negative emotional associations to people and induce sadness. <end>

[0063] Based on the above results, a more suitable text description is regenerated using a large language model: "The soft and light piano melody, accompanied by the light sound of rain, creates a warm and relaxing atmosphere. The piano tone is mainly major, combined with the harmonious background sound of rain, highlighting the sense of tranquility without losing vitality, aiming to resonate with the audience and create a comfortable and immersive listening experience.", thereby updating the text description and improving the audio generation effect.

[0064] The above-mentioned healing music generation method based on AIGC first obtains the patient's music preference text description and the patient's specified audio clip, then extracts features from the above text, and obtains a decodable audio format sequence based on text features based on the audio sequence generation model, and encodes the audio, and obtains a decodable audio format sequence based on audio features based on the self-attention mechanism, and then uses the cross-attention mechanism to process the above two decodable audio format sequences and decode them to obtain the generated audio clip, and uses the above audio clip to complete the patient's music therapy, and at the same time, based on user feedback evaluation, fine-tune the training audio sequence generation model to optimize the model prediction effect, and based on user feedback evaluation, use the large language model to update the text description to obtain a more suitable text description of music preferences. The healing music generation method based on AIGC of this application will guide the model to complete the decoding and generation of healing music clips through the decodable audio format sequence obtained by encoding and fusing the text description and audio clips, and improve the model audio generation adaptability with the help of the feedback mechanism, thereby improving the therapeutic effect of music therapy for mild mental illness.

[0065] Based on the above embodiments, Figure 2 The steps of extracting features from text and obtaining a decodable audio format sequence based on the audio sequence generation model are broken down and refined. Figure 2 As shown, the implementation process includes the following:

[0066] S201, based on the text encoding model, processing the text to obtain text semantic feature encoding.

[0067] After obtaining the text sentence input, the BERT model is first used to preprocess the input sentence. For each word, the word feature vector Et, the position feature vector Ep and the distinguishing feature vector Es are obtained respectively. These three vectors are summed up to obtain the input vector Et, that is, the encoded text information.

[0068] Et=Et+Ep+Es

[0069] Then, a multi-layered encoder structure is used to simultaneously focus on the relationship between each word vector and other word vectors in the input sequence, capturing contextual information from the entire sequence. The text semantic feature encoding Fs of the target word is obtained by calculating the relevance of the target word with all the words in the sentence and normalizing the relevance. The relevance and the encoding of all words are weighted and summed.

[0070] S202, based on the sentiment dictionary, processing the text to obtain text sentiment feature encoding.

[0071] Load the sentiment dictionary (such as the Canadian National Research Council Sentiment Dictionary) to obtain the association information between words and emotions, and perform word segmentation on the input text. For each word in the text, look up its association with each emotion category (such as anger, expectation, disgust, fear, joy, etc.) in the sentiment dictionary, generate the corresponding sentiment embedding vector, and aggregate the sentiment vectors of each word to obtain the text sentiment feature encoding Fe.

[0072] S203, concatenating the above two feature codes to obtain a text feature code.

[0073] Concatenate the text semantic feature coding and the text sentiment feature coding to obtain the text feature coding (including semantics and sentiment) F, where the text semantic feature coding dimension of a single word is Ds and the text sentiment feature coding dimension is De, then the dimension D of the text feature coding (including semantics and sentiment) is D=Ds+De.

[0074] S204: Based on the audio sequence generation model, the decodable audio format sequence is predicted using the above text feature encoding.

[0075] The audio sequence generation model is used to predict the discrete decodable audio format sequence Tt based on the text feature encoding (including semantics and sentiment) F.

[0076] It can be understood that this embodiment provides a possible implementation method for obtaining a decodable audio format sequence containing text semantics and emotional information, which lays a foundation for the subsequent feature fusion of text and audio.

[0077] Based on the above embodiments, Figure 3 The audio is encoded and the steps of obtaining the encodable audio format sequence based on the self-attention mechanism are decomposed and refined. Figure 3 As shown, the implementation process includes the following:

[0078] S301, based on an audio encoder, processing the audio to obtain audio feature coding.

[0079] After obtaining the audio input, the audio is encoded using the autoencoder of the high-fidelity neural audio compressor, and the representation and compression of the audio signal are completed with the help of the natural abstract feature extraction ability of the neural network to obtain an audio feature code Fm with higher dimension and lower frame rate.

[0080] S302: Based on a self-attention mechanism, the audio feature encoding is processed to obtain a decodable audio format sequence.

[0081] The above audio feature encoding Fm is a discrete representation extracted from the original audio signal. They represent different segments or frames of the audio and contain the encoding of the audio features within a certain time segment. Applying the self-attention mechanism on the above sequence allows a single discrete representation to be updated in combination with the information of the entire audio sequence, thereby capturing the long-term dependencies and complex structures in the audio signal, and improving the sequence's ability to express features on different time scales (such as pitch, speaking speed, emotional changes, etc.). The main principle is: for each discrete representation Xi (the length of the audio sequence is n, and i can take any value from 1 to n), three vectors are generated through linear transformation, namely: query vector Qi, key vector Ki and value vector Vi, where Wq, Wk, and Wv are learnable weight matrices, namely:

[0082] Qi=WqXi Ki=WkXi Vi=WvXi

[0083] Perform dot product between the query vector Qi of each discrete representation Xi and the key vector Kj of all other discrete representations Xj (the length of the audio sequence is n, and j takes all values ​​from 1 to n in turn), calculate their similarity, and normalize them to obtain the attention weight, that is:

[0084]

[0085] Finally, the value vectors Vj of all other discrete representations Xj are weighted and summed according to the above attention weights, and the values ​​of all discrete representations Xi are updated, that is:

[0086]

[0087] At this time, the new discrete representation Xi not only contains its own information, but also integrates the information in the entire audio sequence, and finally obtains a decodable audio format sequence Tm containing audio features.

[0088] It can be understood that this embodiment provides a possible implementation method for obtaining a decodable audio format sequence containing audio feature information, which lays a foundation for subsequent feature fusion of text and audio.

[0089] Based on the above embodiments, Figure 4 The steps of processing the above two decodable audio format sequences and decoding them using the cross-attention mechanism to obtain the generated audio clips are decomposed and refined. Figure 4 As shown, the implementation process includes the following:

[0090] S401, using a cross attention mechanism to process the above two decodable audio format sequences to obtain a fused decodable audio format sequence.

[0091] The cross-attention mechanism allows the sequence of one modality to extract information from another modality when processing sequences from different modalities. For two encodable audio format sequences from text and audio respectively, one of the sequences is used as the query and the sequence of the other modality is used as the key and value. By calculating the similarity between the query modality and the key modality, the representation of the value modality is weighted and combined to generate a new representation of the query modality.

[0092] For example, for each discrete representation Si of a text sequence (the length of the text sequence is n s , i can be from 1 to n s Any value) generates the query vector Qi through linear transformation. For each discrete representation Xj of the audio sequence (the length of the audio sequence is n x , j is taken from 1 to n x All values ​​of ) generate key vector Kj and value vector Vj, where Wq, Wk, Wv are learnable weight matrices, namely:

[0093] Qi=WqSi Kj=WkXj Vj=WvXj

[0094] Then calculate the similarity between the query vector Qi from the text sequence and the key vector Kj from the audio sequence, and calculate the attention weight by dot product and normalization, that is:

[0095]

[0096] Finally, the value vectors Vj of all other discrete representations Xj are weighted and summed according to the above attention weights, and the values ​​of all discrete representations Si are updated, that is:

[0097]

[0098] In this example, the cross-attention from text to audio and the cross-attention from audio to text are calculated separately, that is, the query vector is calculated using the decodable audio format sequence Tt from the text and the decodable audio format sequence Tm from the audio, the values ​​of Tt and Tm are updated, and they are fused by weighted sum and splicing to obtain the fused decodable audio format sequence T, thereby realizing the mutual fusion and enhancement of the two modal features.

[0099] S402, using an audio decoder to process the fused decodable audio format sequence to obtain an audio file generated by the model.

[0100] After obtaining the fused decodable audio format sequence T, the sequence is decoded using the decoder of the high-fidelity neural audio compressor to restore the discrete sequence information into continuous audio features, and these features are converted into the final audio waveform through the reconstruction network to obtain the generated audio file.

[0101] It can be understood that this embodiment provides a possible implementation method of fusing two decodable audio format sequences and decoding and converting them into audio to obtain the final audio output.

[0102] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.< / end> < / start> < / end> < / start> < / end> < / start> < / end> < / start>

Claims

1. A healing music generation method based on AIGC, characterized in that: The following steps are involved: (1) Obtain the user's corresponding music preference text description and specified audio clip; (2) extracting features from the acquired text description to obtain text feature encoding that contains both semantic and emotional information, and obtaining a decodable audio format sequence based on an audio sequence generation model; (3) Encode the acquired audio clips and obtain another decodable audio format sequence based on the self-attention mechanism; (4) using a cross-attention mechanism to process the two decodable audio format sequences obtained above and decode them to obtain a fused audio clip for healing; (5) Based on user feedback and evaluation, fine-tune the audio sequence generation model in the training step (2), use the large language model to update the text description in step (1), and repeat the above steps (1) to (4) until an audio clip that satisfies the user is obtained.

2. The AIGC-based healing music generation method according to claim 1, characterized in that: In step (1), the music preference text description includes a combination of music preference keywords input by the user and the existing music preference keywords selected by the user; The designated audio clips include music clips that the user actively expresses and creates during music therapy, and audio clips that the user selects from existing music clips.

3. The AIGC-based healing music generation method according to claim 1, characterized in that: The specific process of step (2) is as follows: Process the text based on the text encoding model to obtain the text semantic feature encoding; The text is processed based on the sentiment dictionary to obtain the text sentiment feature coding; the text semantic feature coding and the text sentiment feature coding are concatenated to obtain the text feature coding containing both semantic and sentiment information; then, based on the audio sequence generation model, the text feature coding is used to predict and obtain a decodable audio format sequence.

4. The AIGC-based healing music generation method according to claim 3, characterized in that: The process of processing text based on the text encoding model to obtain the text semantic feature encoding is as follows: The BERT model is used to preprocess the input text. For each word, the word feature vector, position feature vector, and distinguishing feature vector are obtained. The three vectors are summed to obtain the input vector, that is, the encoded text information. Then, a multi-layer stacked encoder structure is used to simultaneously focus on the relationship between each word vector and other word vectors in the input sequence to capture contextual information from the entire sequence; by calculating the relevance of the target word with all the words in the sentence and normalizing the relevance, the relevance and the encoding of all words are weighted and summed to obtain the text semantic feature encoding of the target word.

5. The AIGC-based healing music generation method according to claim 1, characterized in that: In step (2), the audio sequence generation model cascades multiple vector quantization modules in a residual manner, performs residual vector quantization on the text feature encoding, and sums the outputs of each vector quantization module to obtain a decodable audio format sequence; Among them, the vector quantization module matches the continuous input vector with the pre-defined codebook, finds the closest codeword, and replaces the original input, effectively converting a large amount of continuous data into a smaller discrete representation, realizing the discretization of continuous vectors, and obtaining discrete sequence data; the residual method is to encode the text features in the output of the first vector quantization module and perform a residual calculation on the original input, and use the result as the input of the second vector quantization module, and so on, thereby improving the reconstruction accuracy and reducing the codebook size requirement.

6. The AIGC-based healing music generation method according to claim 1, characterized in that: The specific process of step (3) is as follows: The audio is processed based on the audio encoder to obtain audio feature encoding. The audio feature encoding obtained by processing based on the self-attention mechanism dynamically processes features on different time scales, including pitch, speaking speed, and emotional changes, and finally obtains a decodable audio format sequence.

7. The AIGC-based healing music generation method according to claim 1, characterized in that: The specific process of step (4) is as follows: The cross-attention mechanism is used to process decodable audio format sequences of two different modalities, allowing the sequence of one modality to extract information from the other modality. By fusing the semantic and emotional features of the text and the musical features of the audio, a decodable audio format sequence with complementary features is obtained. The audio decoder is then used to process the above-mentioned fused decodable audio format sequence to reconstruct the audio clip for healing.

8. The AIGC-based healing music generation method according to claim 1, characterized in that: In step (5), the audio sequence generation model in step (2) is fine-tuned, specifically: The patient's evaluation of the audio after music therapy was obtained, and the generated audio and evaluation scores were used as metadata to establish a feedback dataset. Based on the dataset, the LoRA fine-tuning method was used to fine-tune the model weight parameters.

9. The AIGC-based healing music generation method according to claim 1, characterized in that: In step (5), the text description in step (1) is updated using the large language model, specifically: Use the large language model to analyze the user's direct feedback on the defects of the audio generated by the current model, summarize the possible reasons why the original text description performs poorly on the current task, and output a series of reflection results; define a reflection prompt template to guide the large language model to reflect on itself; After obtaining a series of reflection results, we use the large language model to continue analyzing the improvement strategies for the above results, and generate more suitable text descriptions based on the original text descriptions to re-participate in audio generation.

10. A healing music generation system based on AIGC, characterized in that: It comprises a memory and one or more processors, wherein the memory stores executable codes, and when the one or more processors execute the executable codes, they are used to implement the healing music generation method described in any one of claims 1-9.

Images