Voice music creation and deduction system and method based on AI digital human

Abstract

The invention belongs to the technical field of AI, and discloses a voice music creation and deduction system and method based on an AI digital human. The invention provides a voice music creation and deduction system based on an AI digital human. The system comprises a data collection module, a data processing module, a central control module, a feature extraction module, a music mapping and generating module, a music deduction module, a user feedback and adjustment module and a test optimization module. According to the technical scheme, the AI digital human can create music works with novelty and artistry, and aesthetic needs of different users are met. And vivid music deduction: the music deduction of the AI digital human is more natural and vivid, and the audio-visual experience close to the real human singing can be brought to the user. And personalized music experience: the system can adjust creation and deduction strategies according to preferences and feedback of the user, so that the personalized music experience is provided for the user.

Description

Technical Field

[0001] The present invention belongs to the field of AI technology, and in particular relates to a voice music creation and interpretation system and method based on AI digital human. Background Art

[0002] With the rapid development of artificial intelligence technology, AI digital humans have become the new favorite in the field of digital entertainment. This technical solution aims to build a voice music creation and interpretation system based on AI digital humans. Through technologies such as deep learning and natural language processing, digital humans can independently create and interpret music works, providing users with a new music experience. How to convert the voice of AI digital humans into musical elements with melody and rhythm is the primary problem facing the system. How to ensure that the music works created by AI digital humans are novel and artistic, and avoid repetition and monotony. How to make the music interpretation of AI digital humans more natural and realistic, so as to achieve an effect close to that of real human singing.

[0003] Through the above analysis, the problems and defects of the prior art are as follows:

[0004] (1) How to convert the AI ​​digital human’s voice into musical elements with melody and rhythm is the primary problem facing the system.

[0005] (2) How to ensure that the music works created by AI digital humans are novel and artistic, and avoid repetition and monotony.

[0006] (3) How to make the AI ​​digital human’s musical performance more natural and realistic, so as to achieve an effect close to that of real human singing. Summary of the invention

[0007] In view of the problems existing in the prior art, the present invention provides a voice music creation and interpretation system and method based on AI digital human.

[0008] The present invention is implemented as follows: a voice music creation and interpretation system based on AI digital human, comprising:

[0009] Data collection module, used to collect music data and AI digital human voice data;

[0010] A data processing module, connected to the data collection module, is used to clean and extract features from the collected data;

[0011] The central control module is connected to the data processing module to control the operation of the entire system and coordinate the data interaction between the modules;

[0012] The feature extraction module is connected to the central control module and uses the deep learning model to extract features from the AI ​​digital human’s voice data;

[0013] The music mapping and generation module establishes a mapping relationship between speech features and music elements based on the output of the feature extraction module and generates music works;

[0014] The music interpretation module is used to simulate the sound of real human singing and synchronize the expressions and movements of the AI ​​digital human with the rhythm of the music;

[0015] User feedback and adjustment module, used to collect user feedback and adjust music creation and interpretation strategies based on the feedback;

[0016] Test optimization module, used to perform functional testing and performance optimization on each module of the system;

[0017] Among them, the music mapping and generation module is characterized by converting the voice features of AI digital people into notes, melodies and rhythms through mapping relationships, and using music generation algorithms to generate musical melodies and rhythms to form a complete musical work; the music interpretation module is characterized by using sound synthesis technology to simulate human singing sounds, and using motion capture and expression recognition technology to synchronize the expressions and movements of AI digital people with the rhythm of music, while using sound effect libraries and audio processing technology to simulate the environmental sound effects of different singing scenes.

[0018] Further, the feature extraction module extraction method:

[0019] 1) Deep learning model: Design or select a deep learning model suitable for speech analysis, including LSTM or Transformer, to extract pitch, rhythm, intensity, and timbre features from speech;

[0020] 2) Preprocessing: including speech noise reduction and normalization processing to ensure the quality of input data.

[0021] Further, the test optimization module tests the optimization method:

[0022] (1) Functional testing: Perform functional testing on each module of the system to ensure that it operates normally and meets the design requirements;

[0023] (2) User experience testing: Invite users to conduct experience testing, collect user feedback, and optimize the system based on the feedback;

[0024] (3) Performance optimization: Optimize the performance of the system.

[0025] Furthermore, the music mapping and generation module:

[0026] 1) Music mapping;

[0027] Rule definition: Establishing mapping rules between pitch and notes, speech speed and rhythm; this requires the support of music theory, as well as creative thinking including how to artistically transform language features into musical elements;

[0028] Software implementation: Develop an algorithm to automatically convert the extracted speech features into musical elements; including notes, chords, and rhythms;

[0029] 2) Music generation;

[0030] Music synthesizer: Use MIDI synthesizer or other music software to generate music;

[0031] Innovative guarantee: introducing random algorithms and style diversity.

[0032] Furthermore, the music interpretation module:

[0033] Voice Synthesis: Apply advanced voice synthesis techniques, including WaveNet, to generate near-realistic human voices;

[0034] Motion synchronization: Use motion capture technology to synchronize the expressions and movements of AI digital people with the music;

[0035] Environmental sound effects: According to the style and scene of the music, simulate the corresponding environmental sound effects, including reverberation, to enhance the sense of presence;

[0036] Furthermore, the user feedback and adjustment module:

[0037] User Interface: Develop a user-friendly user interface that allows users to upload voice files, select a music style, and receive generated music;

[0038] Feedback mechanism: Establish a feedback system where users can evaluate the generated music and provide suggestions for improvement;

[0039] Data analysis: Analyze user feedback to optimize models and mapping rules.

[0040] Another object of the present invention is to provide a method for creating and performing voice music based on AI digital human, comprising:

[0041] Step 1: Collect a large amount of music data and AI digital human voice data through the data collection module; pre-process the collected data through the data processing module, including data cleaning and feature extraction;

[0042] Step 2: The central control module uses a deep learning model to extract features from the AI ​​digital human’s voice data through a feature extraction module to extract key features such as pitch, rhythm, and volume;

[0043] Step 3: A mapping relationship between speech features and music elements is established through a music mapping and generation module, and the extracted speech features are converted into notes, melodies and rhythms to generate musical melodies and rhythms to form a complete music work;

[0044] Step 4: Use sound synthesis technology to simulate the sound of real human singing through the music interpretation module, and use motion capture and expression recognition technology to synchronize the expression and movement of the AI ​​digital human with the rhythm of the music, while simulating the environmental sound effects of different singing scenes;

[0045] Step 5: Establish a user feedback system through the user feedback and adjustment module to collect user feedback on music creation and interpretation, and adjust the music creation strategy and interpretation method based on the feedback to meet the user's personalized needs; perform functional testing and optimization on each module of the system through the test optimization module.

[0046] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method for creating and interpreting voice music based on AI digital humans.

[0047] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the method for creating and interpreting voice music based on AI digital humans.

[0048] Another object of the present invention is to provide an information data processing terminal, which is used to implement the voice music creation and interpretation system based on AI digital human.

[0049] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0050] First, innovative music creation: Through this technical solution, AI digital humans can create novel and artistic music works to meet the aesthetic needs of different users.

[0051] Realistic music interpretation: The AI ​​digital human's music interpretation is more natural and realistic, and can bring users an audio-visual experience close to that of real human singing.

[0052] Personalized music experience: The system can adjust the creation and performance strategies according to the user's preferences and feedback to provide users with a personalized music experience.

[0053] Second, the AI ​​digital human voice music creation and interpretation system of the present invention specifically solves several key technical problems existing in traditional music creation and singing, and achieves significant technical progress:

[0054] 1. Limitations of automated music creation:

[0055] In the prior art, automated music creation systems usually lack understanding and response to user emotions and personalized needs. By combining advanced music data processing and AI models, the present invention can analyze the user's voice input, capture their emotions and intentions, and then create music works that reflect personal emotions.

[0056] 2. Challenges of high-quality music performance:

[0057] In the existing automated music interpretation technology, it is often difficult to achieve the expressiveness and appeal of human artistic performances. The present invention uses advanced sound synthesis technology and motion capture system to realize the humanized interpretation of AI digital people, so that they can show expressions and movements synchronized with the rhythm of music when singing, enhancing the realism and viewing value of the performance.

[0058] 3. Difficulties in achieving personalized music experience:

[0059] Traditional music creation and interpretation systems are difficult to meet the personalized needs of users on a large scale. The system of the present invention can continuously learn and adapt to user preferences by integrating user feedback and adjustment modules, and adjust the details of music creation and interpretation in real time to better serve the personalized experience of users.

[0060] Significant technological advances achieved:

[0061] 1. Enhanced creative flexibility and efficiency:

[0062] The present invention combines natural language processing and music generation algorithms, so that the conversion from speech to music is not limited to simple pattern matching, but can perform in-depth emotional analysis and creative expression, greatly improving the flexibility and efficiency of music creation.

[0063] 2. Improved performance quality and interactivity:

[0064] Through motion capture and advanced sound synthesis technology, AI digital humans can perform music in a way that is very close to real people, and their expressiveness and interactivity far exceed traditional AI singing systems.

[0065] 3. Realize truly personalized music services:

[0066] The system continuously learns user feedback and optimizes music content based on feedback, truly realizing the personalization of music services. Each user can get their own music experience to meet their unique emotional and aesthetic needs.

[0067] The present invention significantly improves the quality, efficiency and personalization of music creation and interpretation by integrating innovative AI technology and user interaction design, and promotes the development of music creation and interpretation technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a structural block diagram of the voice music creation and interpretation system based on AI digital human provided by an embodiment of the present invention.

[0069] Figure 2 It is a flow chart of the feature extraction module extraction method provided by an embodiment of the present invention.

[0070] Figure 3 It is a flow chart of the test optimization method of the test optimization module provided in an embodiment of the present invention.

[0071] Figure 4 It is a flow chart of the voice music creation and interpretation method based on AI digital human provided by an embodiment of the present invention.

[0072] Figure 1 In: 1. Data collection module; 2. Data processing module; 3. Central control module; 4. Feature extraction module; 5. Music mapping and generation module; 6. Music interpretation module; 7. User feedback and adjustment module; 8. Test optimization module. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0074] In the AI ​​digital human-based voice music creation and interpretation system, the signal and data processing process is the core of the system operation. The following are the detailed signal and data processing steps:

[0075] 1. Data Collection Module

[0076] Function: Collect two types of data: one is the audio data of various musical works, including music of different styles and instruments. The other is the voice data of AI digital people, covering voice samples of different emotions, intonations and intensities.

[0077] Operation: Use online APIs, music libraries, and voice recording devices for data collection to ensure data diversity and richness.

[0078] 2. Data processing module

[0079] Preprocessing: includes data cleaning (noise removal, standardization of audio length, etc.) and data formatting to adapt to subsequent processing needs.

[0080] Feature extraction: Extract key audio features, such as frequency, rhythm, pitch, volume, and emotional characteristics of speech, using FFT (Fast Fourier Transform) and other signal processing techniques.

[0081] 3. Feature extraction module

[0082] Deep learning models: Use models such as CNN (convolutional neural network) or RNN (recurrent neural network) to analyze and learn deep features of speech and music data.

[0083] Output: The model outputs numerical representations of pitch, rhythm, and volume, providing basic data for music mapping and generation.

[0084] 4. Music Mapping and Generation Module

[0085] Establishing mapping relationship: According to the mapping rules between the extracted speech features and the musical elements (such as the high and low intonation corresponds to the high and low pitch), they are converted into the basic components of music.

[0086] Music Generation: Combined with music theory (harmony, rhythm, structure, etc.), algorithms are used to synthesize the mapped data into complete musical works.

[0087] 5. Music interpretation module

[0088] Voice synthesis: Use advanced voice synthesis technology to simulate real human voice singing, including intonation, emotion, etc.

[0089] Motion synchronization: Combined with motion capture technology, the expressions and body movements of AI digital people are synchronized with the rhythm of music to enhance the realism and expressiveness of the performance.

[0090] Environmental sound simulation: simulate the corresponding environmental sound effects (such as echo, reverberation) according to the scene of the music performance (such as concert hall, outdoors, etc.).

[0091] 6. User feedback and adjustment module

[0092] Feedback collection: Collect audience comments and suggestions on musical works and performances through the user interface.

[0093] Adjustment strategy: Based on the collected feedback, adjust the parameters or style of music creation and performance to better meet the personalized needs of users.

[0094] 7. Test optimization module

[0095] Functional testing: Verifies the functionality of each module and the overall performance of the system to ensure that the parts work together.

[0096] System optimization: Adjust system configuration and algorithms based on test results to improve system stability and efficiency.

[0097] Through the above detailed signal and data processing flow, the system can achieve efficient conversion from speech to music, and interpret it in a highly personalized and dynamic way through AI digital humans, creating new art forms while meeting the diverse needs of users. Figure 1 As shown, a voice music creation and interpretation system based on AI digital human provided by an embodiment of the present invention includes:

[0098] Data collection module 1, data processing module 2, central control module 3, feature extraction module 4, music mapping and generation module 5, music interpretation module 6, user feedback and adjustment module 7, test optimization module 8.

[0099] Data collection module 1, connected to data processing module 2, is used to collect a large amount of music data and AI digital human voice data;

[0100] The data processing module 2 is connected to the central control module 3 and is used to pre-process the collected data, including data cleaning and feature extraction;

[0101] The central control module 3 is connected with the data processing module 2, the feature extraction module 4, the music mapping and generation module 5, the music interpretation module 6, the user feedback and adjustment module 7, and the test optimization module 8, and is used to control the normal operation of each module;

[0102] The feature extraction module 4 is connected to the central control module 3 and is used to extract the features of the AI ​​digital human’s voice data using a deep learning model to extract key features such as pitch, rhythm, and volume;

[0103] The music mapping and generation module 5 is connected to the central control module 3 and is used to establish a mapping relationship between speech features and music elements, and convert the extracted speech features into notes, melodies and rhythms, generate musical melodies and rhythms, and form a complete music work;

[0104] The music interpretation module 6 is connected to the central control module 3 and is used to simulate the sound of real human singing by using sound synthesis technology, synchronize the expression and movement of the AI ​​digital human with the music rhythm by using motion capture and expression recognition technology, and simulate the environmental sound effects of different singing scenes;

[0105] The user feedback and adjustment module 7 is connected to the central control module 3 and is used to establish a user feedback system, collect user feedback on music creation and interpretation, and adjust the strategy of music creation and interpretation method according to the feedback to meet the personalized needs of users;

[0106] The test optimization module 8 is connected to the central control module 3 and is used to perform functional testing and optimization on each module of the system.

[0107] The working principle of the AI ​​digital human voice music creation and interpretation system can be elaborated in detail in combination with specific algorithms and mathematical models. The following is a detailed description of the working principle of the main modules of the system:

[0108] Data collection module:

[0109] This module uses crawler technology to collect a large amount of music data and AI digital human voice data from music platforms, social media and other channels. These data include music works of different styles, AI digital human recordings, singing videos, etc.

[0110] Data processing module:

[0111] Data cleaning: Use statistical and machine learning methods to remove noise, outliers, and irrelevant data to ensure data quality and consistency.

[0112] Feature extraction: Use signal processing technology to extract melody, rhythm, chords and other features of music data; for AI digital human voice data, extract key features such as pitch, timbre, and speaking speed.

[0113] Central control module:

[0114] The central control module is a core scheduler that controls the data interaction and process execution between other modules according to preset logic and rules. By receiving input from other modules, the control module can coordinate the operation of the entire system and ensure that each module works according to the predetermined process.

[0115] Feature extraction module:

[0116] This module uses deep learning models (such as recurrent neural networks (RNNs), convolutional neural networks (CNNs), or Transformer models) to extract features from the voice data of AI digital humans. The model learns the potential structure in the voice data through training, and then extracts key features such as pitch, rhythm, and volume.

[0117] Music mapping and generation module:

[0118] Mapping relationship establishment: Based on music theory and rules, a mapping table of speech features to music elements is constructed. For example, pitch can be mapped to notes, and rhythm can be mapped to beats.

[0119] Music generation: Using the generated mapping relationship, the extracted speech features are converted into notes, melodies and rhythms. Music generation algorithms (such as Markov chains, LSTM networks, etc.) are used to generate musical melodies and rhythms, ultimately forming a complete musical work.

[0120] Music interpretation module:

[0121] Sound synthesis: Use waveform synthesis or parameter synthesis technology to simulate the sound of real human singing. According to the generated notes and melodies, the corresponding singing sound is synthesized.

[0122] Synchronize movements and expressions: Use motion capture technology to capture the movement data of the AI ​​digital human, and use expression recognition technology to identify changes in its expression. Synchronize these data with the rhythm of the music, so that the expressions and movements of the AI ​​digital human during the singing process match the rhythm of the music.

[0123] Environmental sound simulation: Use the sound effect library and audio processing technology to simulate the environmental sound effects of different singing scenes, such as indoor, outdoor, concerts, etc., to enhance the immersion and realism of music.

[0124] User feedback and adjustment module:

[0125] This module collects user feedback on music creation and interpretation through the user interface. The feedback text is analyzed using natural language processing technology to extract user preferences and suggestions on music style, rhythm, volume, etc. Based on this feedback, the system adjusts the music creation strategy and interpretation method to meet the user's personalized needs.

[0126] Test optimization module:

[0127] This module uses automated testing tools and algorithms to perform functional tests on each module of the system. By inputting preset test data, we observe whether the system output meets the expected results. At the same time, we collect performance data during system operation, such as response time, resource consumption, etc., to optimize the system performance. Through continuous testing and optimization, we can improve the stability and efficiency of the system.

[0128] The AI ​​digital human-based voice music creation and interpretation system integrates multiple modules and algorithms to realize the entire process from voice data collection to music creation, interpretation and optimization. The system not only improves the efficiency and diversity of music creation, but also provides users with a personalized music experience.

[0129] like Figure 2 As shown, the feature extraction module extraction method provided by the embodiment of the present invention is:

[0130] S101, Deep Learning Model: Design or select a deep learning model suitable for speech analysis, including LSTM or Transformer, to extract pitch, rhythm, intensity, and timbre features from speech;

[0131] S102, preprocessing: including speech noise reduction and normalization processing to ensure the quality of input data.

[0132] like Figure 3 As shown, the test optimization module test optimization method provided by the embodiment of the present invention is:

[0133] S201, functional test: perform functional tests on each module of the system to ensure that it operates normally and meets the design requirements;

[0134] S202, user experience testing: invite users to conduct experience testing, collect user feedback, and optimize and adjust the system based on the feedback;

[0135] S203, performance optimization: optimizing the performance of the system.

[0136] A specific example shows how the AI ​​digital human-based voice music creation and interpretation system works in practical applications:

[0137] Example: Create a personalized birthday song

[0138] A user wants to create a special birthday song for a friend who is about to celebrate his birthday. The user hopes that the song can reflect the personality of his friend and the friendship between them, and he also hopes that the AI ​​digital person can sing the song in a moving way.

[0139] 1. Data Collection Module

[0140] Users upload several voice recordings describing their friends' personalities and their shared memories through the mobile app. At the same time, the system automatically collects music data of different styles from the music database as a reference.

[0141] 2. Data processing module

[0142] The system pre-processes the uploaded voice, cleans it and extracts the basic features of the voice (such as pitch, speaking speed and rhythm).

[0143] The data extracted from the music database is also preprocessed to adapt to subsequent analysis and learning.

[0144] 3. Feature extraction module

[0145] Using deep learning models (such as CNN), the emotional and musical characteristics of the user's voice are analyzed to capture musical elements that can express the user's emotions.

[0146] 4. Music Mapping and Generation Module

[0147] Based on the extracted features, the system automatically converts the user's voice into musical elements (melody, harmony and rhythm).

[0148] Automatically arrange and generate a complete birthday song that incorporates the user's emotions and musical style preferences.

[0149] 5. Music interpretation module

[0150] The AI ​​digital man simulates the human voice through sound synthesis technology and performs this birthday song in a moving way.

[0151] At the same time, the motion capture system captures the performance movements of the AI ​​digital human, ensuring that expressions and movements are synchronized with the rhythm of the music, enhancing the expressiveness of the performance.

[0152] 6. User feedback and adjustment module

[0153] After hearing the AI-created song for the first time, the user provides feedback, hoping to add some personalized lyrics. The system adjusts the song content based on the feedback and adds lyrics automatically generated by AI that are relevant to the characteristics of the user's friend.

[0154] 7. Test optimization module

[0155] The system is tested for functionality to ensure that all elements—melody, lyrics, performance—are seamlessly integrated.

[0156] After confirming that everything is functional, a final version of the song and performance video is created and sent to the user.

[0157] The user received a completely personalized birthday song, performed by an AI digital person in a touching way, which not only contained a specific musical style, but also incorporated the emotions and memories between the user and his friend. The friend was very surprised and touched by this unique birthday gift, and the song became a beautiful commemoration of their friendship.

[0158] This embodiment demonstrates how the AI ​​digital human-based voice music creation and interpretation system can transform human emotions and memories into musical works in practical applications, and bring new artistic experiences through AI interpretation.

[0159] The music mapping and generation module provided by the embodiment of the present invention:

[0160] 1) Music mapping;

[0161] Rule definition: Establishing mapping rules between pitch and notes, speech speed and rhythm; this requires the support of music theory, as well as creative thinking including how to artistically transform language features into musical elements;

[0162] Software implementation: Develop an algorithm to automatically convert the extracted speech features into musical elements; including notes, chords, and rhythms;

[0163] 2) Music generation;

[0164] Music synthesizer: Use MIDI synthesizer or other music software to generate music;

[0165] Innovative guarantee: introducing random algorithms and style diversity.

[0166] The music interpretation module provided by the embodiment of the present invention:

[0167] Voice Synthesis: Apply advanced voice synthesis techniques, including WaveNet, to generate near-realistic human voices;

[0168] Motion synchronization: Use motion capture technology to synchronize the expressions and movements of AI digital people with the music;

[0169] Environmental sound effects: According to the style and scene of the music, simulate the corresponding environmental sound effects, including reverberation, to enhance the sense of presence;

[0170] The user feedback and adjustment module provided by the embodiment of the present invention:

[0171] User Interface: Develop a user-friendly user interface that allows users to upload voice files, select a music style, and receive generated music;

[0172] Feedback mechanism: Establish a feedback system where users can evaluate the generated music and provide suggestions for improvement;

[0173] Data analysis: Analyze user feedback to optimize models and mapping rules.

[0174] like Figure 4 As shown, a method for creating and performing voice music based on AI digital human provided by an embodiment of the present invention includes:

[0175] S301, collecting a large amount of music data and AI digital human voice data through the data collection module; pre-processing the collected data through the data processing module, including data cleaning and feature extraction;

[0176] S302, the central control module uses the deep learning model through the feature extraction module to extract the key features of the AI ​​digital human's voice data, such as pitch, rhythm, and volume;

[0177] S303, establishing a mapping relationship between speech features and music elements through a music mapping and generation module, and converting the extracted speech features into notes, melody and rhythm, generating a musical melody and rhythm, and forming a complete music work;

[0178] S304, using the sound synthesis technology to simulate the sound of real human singing through the music interpretation module, and using the motion capture and expression recognition technology to synchronize the expression and movement of the AI ​​digital human with the music rhythm, while simulating the environmental sound effects of different singing scenes;

[0179] S305, establish a user feedback system through the user feedback and adjustment module, collect user feedback on music creation and interpretation, adjust the music creation strategy and interpretation method according to the feedback to meet the user's personalized needs; perform functional testing and optimization on each module of the system through the test optimization module.

[0180] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method for creating and interpreting voice music based on AI digital humans.

[0181] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the method for creating and interpreting voice music based on AI digital humans.

[0182] Another object of the present invention is to provide an information data processing terminal, which is used to implement the voice music creation and interpretation system based on AI digital human.

[0183] The present invention is specifically implemented:

[0184] The specific method of building a voice music creation and interpretation system based on AI digital human is as follows:

[0185] 1. System architecture design

[0186] 1. Data collection and processing module: collects a large amount of music data and AI digital human voice data, and performs pre-processing, including data cleaning, feature extraction, etc.

[0187] 2. Feature extraction module: Use the deep learning model to extract features from the AI ​​digital person’s voice data, extracting key features such as pitch, rhythm, and volume.

[0188] 3. Music mapping and generation module: Establish a mapping relationship from speech features to musical elements, and convert the extracted speech features into notes, melodies and rhythms, generate musical melodies and rhythms, and form a complete musical work.

[0189] 4. Music interpretation module: Use sound synthesis technology to simulate the sound of real human singing, and use motion capture and expression recognition technology to synchronize the expressions and movements of AI digital people with the rhythm of music. At the same time, simulate the environmental sound effects of different singing scenes to enhance the realism and immersion of music interpretation.

[0190] 5. User feedback and adjustment module: Establish a user feedback system to collect user feedback on music creation and performance, and adjust music creation strategies and performance methods based on feedback to meet users' personalized needs.

[0191] 2. Technical Implementation

[0192] 1. Deep learning model training: Use a large amount of music data and voice data to train the deep learning model so that it can accurately extract voice features and perform music mapping.

[0193] 2. Introduction of randomness and diversity: In the process of music generation, random factors are introduced to create differences and novelty. At the same time, AI digital humans are trained to learn a variety of musical styles and genres to enrich their creative inspiration and expression ability.

[0194] 3. Sound synthesis and expression and movement synchronization: Use advanced sound synthesis technology to simulate the sound of real human singing, and use motion capture and expression recognition technology to synchronize the expression and movement of AI digital people with the rhythm of music.

[0195] 4. Ambient sound processing: Use audio processing technology to simulate the ambient sound effects of different singing scenes, such as reverberation and echo, to enhance the sense of presence and immersion of music performance.

[0196] 3. System Testing and Optimization

[0197] 1. Functional testing: Perform functional testing on each module of the system to ensure that it operates normally and meets the design requirements.

[0198] 2. User experience testing: Invite users to conduct experience testing, collect user feedback, and optimize and adjust the system based on the feedback.

[0199] 3. Performance optimization: Optimize the performance of the system to improve the speed and quality of music creation and performance.

[0200] Through the above methods, a voice music creation and interpretation system based on AI digital humans can be built to enable AI digital humans to independently create and interpret music works, providing users with a brand new music experience.

[0201] To build a system that can convert speech into music, it is necessary to integrate multiple technical components to ensure the efficiency and innovation of the entire process from speech feature extraction to music creation and music interpretation. The following is a detailed introduction on how to implement this system:

[0202] 1. Technical Architecture Overview

[0203] Input layer: receives voice input from the user.

[0204] Processing layer: includes feature extraction, music mapping and music generation modules.

[0205] Output layer: music interpretation and feedback system.

[0206] 2. Detailed technical implementation steps

[0207] 2.1 Feature Extraction

[0208] Deep learning model: Design or select a deep learning model suitable for speech analysis, such as LSTM or Transformer, to extract features such as pitch, rhythm, intensity, and timbre from speech.

[0209] Preprocessing: including speech noise reduction, normalization and other processing to ensure the quality of input data.

[0210] 2.2 Music Mapping

[0211] Rule definition: Establishing mapping rules between pitch and notes, speech speed and rhythm. This requires the support of music theory and creative thinking on how to artistically transform language features into musical elements.

[0212] Software implementation: Develop an algorithm to automatically convert the extracted speech features into musical elements (notes, chords, rhythms).

[0213] 2.3 Music Generation

[0214] Music Synthesizer: Use a MIDI synthesizer or other music software to generate music.

[0215] Innovation guarantee: The introduction of random algorithms and style diversity ensures that the generated music is both individual and novel.

[0216] 2.4 Musical interpretation

[0217] Voice Synthesis: Apply advanced voice synthesis techniques, such as WaveNet, to generate near-realistic human voices.

[0218] Motion synchronization: Use motion capture technology to synchronize the expressions and movements of AI digital people with music.

[0219] Environmental sound effects: According to the style and scene of the music, simulate corresponding environmental sound effects, such as reverberation, to enhance the sense of presence.

[0220] 2.5 User Interaction and Feedback

[0221] User Interface: Develop a friendly user interface that allows users to upload voice files, select a music style, and receive generated music.

[0222] Feedback mechanism: Establish a feedback system where users can evaluate the generated music and provide suggestions for improvements.

[0223] Data analysis: Analyze user feedback to optimize models and mapping rules.

[0224] 3. Technology and resource requirements

[0225] Hardware: Powerful servers and audio processing hardware to support real-time audio processing and large-scale data processing.

[0226] Software: Music production and editing software, AI model development frameworks such as TensorFlow or PyTorch.

[0227] Expertise: A mixed team of music production talents, AI developers, and sound engineers.

[0228] 4. Implementation plan

[0229] Phase 1: Develop and train deep learning models and conduct preliminary feature extraction tests.

[0230] Phase 2: Develop music mapping and generation algorithms and complete the production of the first prototype.

[0231] Phase 3: Improve the music interpretation and user feedback system and conduct comprehensive testing.

[0232] Phase 4: Conduct market testing, collect user feedback and iterate the product.

[0233] Through these steps, a comprehensive system can be built to convert speech into music, which will not only provide new ways of artistic creation, but also promote the integrated development of music and artificial intelligence technologies.

[0234] When developing and testing the AI ​​digital human voice music creation and interpretation system in this example, we conducted a series of detailed experiments and data analysis to verify the performance and effect of the system. The following are some key experimental processes and data results, which demonstrate the significant advantages of this system over existing technologies:

[0235] Experimental Design:

[0236] 1. User experience test:

[0237] Purpose: To evaluate the system's personalized music creation capabilities and user satisfaction.

[0238] Method: 100 trial participants were randomly selected and asked to use the system to create music based on their voice input. After completion, a questionnaire survey was conducted to collect user feedback on the degree of music personalization, satisfaction with emotional expression, and overall experience.

[0239] 2. Music quality assessment test:

[0240] Purpose: To evaluate the music performance quality of AI digital humans.

[0241] Method: Professional musicians evaluated the quality of music created by AI and those created by human musicians using a standardized scoring system that included dimensions such as creativity, technical execution, and emotional expression.

[0242] 3. Performance Benchmarks:

[0243] Purpose: To evaluate the operating efficiency and stability of the system.

[0244] Method: Run the system under high load conditions and monitor processing speed, response time, and system resource consumption.

[0245] Data results and analysis:

[0246] 1. User experience test results:

[0247] Data: In the questionnaire satisfaction rating (out of 10 points), users gave an average rating of 8.7 for the degree of music personalization, an average rating of 9.1 for satisfaction with emotional expression, and an average rating of 8.9 for the overall experience.

[0248] Analysis: The high score indicates that the system performs well in understanding user emotions and personalized music creation, and the user experience is generally positive.

[0249] 2. Music quality assessment test results:

[0250] Data: Compared with the works of human musicians, the music created by AI scored 7.5 in creativity, 8.0 in technical execution, and 8.5 in emotional expression.

[0251] Analysis: Although slightly lower than human musicians in terms of creativity, AI's performance in technical execution and emotional expression is satisfactory, showing the potential of AI in music interpretation.

[0252] 3. Performance benchmark test results:

[0253] Data: When the system handles 100 concurrent requests, the average response time is 2 seconds, and the CPU and memory consumption remain within an acceptable range.

[0254] Analysis: The system demonstrates good performance and efficient resource management, can withstand high usage loads, and is suitable for large-scale deployment.

[0255] Through these experiments and data analysis, we have confirmed that the AI ​​digital human voice music creation and interpretation system has significant advantages over existing technologies in terms of personalized music creation, high-quality music interpretation, and efficient and stable system performance. These advantages enable this system to not only enhance the user's artistic enjoyment, but also have high feasibility and potential market competitiveness in commercial deployment.

[0256] Specific implementation case:

[0257] Project name: AI music idol "Music Dream Factory"

[0258] 1. Project Background

[0259] With the rapid development of artificial intelligence technology, AI is increasingly used in various fields. In the field of music creation and interpretation, the application of AI digital humans is gradually becoming a trend. This project aims to develop a voice music creation and interpretation system based on AI digital humans called "Music Dream Factory" to provide users with a new music creation and appreciation experience.

[0260] 2. System Implementation

[0261] 1. Data collection module:

[0262] A large amount of music data, including melody, rhythm, harmony, etc., is collected through online music platforms, music databases and other channels.

[0263] Use microphones and recording equipment to collect voice data of AI digital people, including singing, reciting, and conversations.

[0264] 2. Data processing module:

[0265] Clean the collected music data to remove noise, redundant information, etc.

[0266] Preprocess the AI ​​digital human’s voice data, including framing, windowing, feature extraction, etc., for subsequent analysis.

[0267] 3. Central control module:

[0268] Responsible for coordinating data interaction between modules to ensure the stability and efficiency of system operation.

[0269] Dynamically adjust the working mode and parameters of each module according to user needs and system status.

[0270] 4. Feature extraction module:

[0271] Use deep learning models (such as convolutional neural networks, recurrent neural networks, etc.) to extract features of AI digital human’s voice data, including timbre, pitch, and length.

[0272] The extracted features are encoded and stored for subsequent music mapping and generation.

[0273] 5. Music mapping and generation module:

[0274] Based on the output of the feature extraction module, a mapping relationship between speech features and musical elements is established, such as timbre mapping to instrument timbre, pitch mapping to melody, etc.

[0275] Use music creation algorithms (such as rule-based creation, generative adversarial network-based creation, etc.) to generate music works with unique style.

[0276] 6. Music interpretation module:

[0277] By using audio synthesis technology to simulate the sound of real human singing, the singing of AI digital people can be made more natural and pleasant.

[0278] Through 3D modeling and animation technology, the expressions and movements of AI digital people can be synchronized with the rhythm of music, enhancing the user's viewing experience.

[0279] 7. User feedback and adjustment module:

[0280] Collect user feedback on music creation and performance through the user interface.

[0281] Adjust music creation and interpretation strategies based on user feedback to continuously optimize the system's performance and effects.

[0282] 8. Test optimization module:

[0283] Perform functional tests on each module of the system to ensure that the system can operate normally and meet user needs.

[0284] Optimize system performance, increase data processing speed, reduce memory usage, etc., to enhance user experience.

[0285] 3. Application Effect

[0286] Through the "Music Dream Factory" system, users can easily create music works with personal characteristics and enjoy the wonderful performances performed by AI digital people. The system not only provides users with a new music creation and appreciation experience, but also promotes the application and development of AI technology in the field of music. At the same time, the system also has certain commercial value and can provide customized music creation and performance services for enterprises and individuals.

[0287] Project name: AI voice music creation platform "Music Weaver"

[0288] 1. Project Background

[0289] "Music Weaver" is a voice music creation and interpretation platform based on AI digital people. It uses deep learning technology to extract features from users' voices and transform them into unique music works. The platform not only provides users with innovative music creation tools, but also provides rich music resources for music lovers and professionals.

[0290] 2. Feature Extraction Module Implementation

[0291] 1. Deep learning model selection: In order to extract features such as pitch, rhythm, intensity, and timbre from speech, we chose the Transformer model. The Transformer model performs well in sequence modeling and can capture long-term dependencies in speech, thereby accurately extracting speech features.

[0292] 2. Preprocessing: Before inputting the speech data into the deep learning model, we first perform noise reduction to eliminate the impact of background noise on feature extraction. Then, we normalize the speech data to ensure the quality and consistency of the input data.

[0293] 3. Music Mapping and Generation Module Implementation

[0294] 1. Music Mapping:

[0295] Rule definition: Based on music theory, we have established mapping rules between pitch and notes, and between speech speed and rhythm. For example, we map the pitch of speech to the notes on the piano keyboard, and map the speed of speech to the rhythm of music. At the same time, we also combine creative thinking to artistically transform language features into musical elements, such as transforming the stress in speech into the emphasized part of music.

[0296] Software implementation: We developed an algorithm that can automatically convert the extracted speech features into musical elements. The algorithm first converts the speech features into musical elements such as notes, chords, and rhythms according to the mapping rules, and then combines and adjusts these elements to generate music works that meet the requirements of music creation.

[0297] 2. Music Generation:

[0298] Music Synthesizer: We use MIDI synthesizers to generate music. After the algorithm generates musical elements such as notes, chords, and rhythms, the MIDI synthesizer converts these elements into actual audio signals to generate a complete musical work.

[0299] Innovation guarantee: In order to introduce innovation and style diversity, we introduced a random algorithm into the music generation process. For example, in terms of chord selection and rhythm change, the algorithm will make random selections based on a certain probability distribution, thereby generating music works with unique styles.

[0300] 4. Test Optimization Module Implementation

[0301] 1. Functional testing: We conducted functional testing on each module of the "Music Weaver" platform, including data collection module, data processing module, feature extraction module, music mapping and generation module, and music interpretation module. Through testing, we ensured that the system can operate normally and meet the design requirements.

[0302] 2. User experience testing: We invited music lovers and professionals to conduct experience testing and collected their feedback. Based on user feedback, we optimized the system and improved the user experience and the quality of the work.

[0303] 3. Performance optimization: We have optimized the performance of the system, including increasing data processing speed, reducing memory usage, etc. Through optimization, we have improved the response speed and stability of the system, providing users with a better user experience.

[0304] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0305] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A voice music creation and interpretation system based on AI digital human, characterized by: include: Data collection module, used to collect music data and AI digital human voice data; A data processing module, connected to the data collection module, is used to clean and extract features from the collected data; The central control module is connected to the data processing module to control the operation of the entire system and coordinate the data interaction between the modules; The feature extraction module is connected to the central control module and uses the deep learning model to extract features from the AI ​​digital human’s voice data; The music mapping and generation module establishes a mapping relationship between speech features and music elements based on the output of the feature extraction module and generates music works; The music interpretation module is used to simulate the sound of real human singing and synchronize the expressions and movements of the AI ​​digital human with the rhythm of the music; User feedback and adjustment module, used to collect user feedback and adjust music creation and interpretation strategies based on the feedback; Test optimization module, used to perform functional testing and performance optimization on each module of the system; Among them, the music mapping and generation module is characterized by converting the voice features of AI digital people into notes, melodies and rhythms through mapping relationships, and using music generation algorithms to generate musical melodies and rhythms to form a complete musical work; the music interpretation module is characterized by using sound synthesis technology to simulate human singing sounds, and using motion capture and expression recognition technology to synchronize the expressions and movements of AI digital people with the rhythm of music, while using sound effect libraries and audio processing technology to simulate the environmental sound effects of different singing scenes.

2. The voice music creation and interpretation system based on AI digital human as claimed in claim 1, characterized in that: The feature extraction module extraction method: 1) Deep learning model: Design or select a deep learning model suitable for speech analysis, including LSTM or Transformer, to extract pitch, rhythm, intensity, and timbre features from speech; 2) Preprocessing: including speech noise reduction and normalization processing to ensure the quality of input data.

3. The voice music creation and interpretation system based on AI digital human as claimed in claim 1, characterized in that: The test optimization module tests the optimization method: (1) Functional testing: Perform functional testing on each module of the system to ensure that it operates normally and meets the design requirements; (2) User experience testing: Invite users to conduct experience testing, collect user feedback, and optimize the system based on the feedback; (3) Performance optimization: Optimize the performance of the system.

4. The AI ​​digital human-based voice music creation and interpretation system as claimed in claim 1, characterized in that: The music mapping and generation module: 1) Music mapping; Rule definition: Establishing mapping rules between pitch and notes, speech speed and rhythm; this requires the support of music theory, as well as creative thinking including how to artistically transform language features into musical elements; Software implementation: Develop an algorithm to automatically convert the extracted speech features into musical elements; including notes, chords, and rhythms; 2) Music generation; Music synthesizer: Use MIDI synthesizer or other music software to generate music; Innovative guarantee: introducing random algorithms and style diversity.

5. The voice music creation and interpretation system based on AI digital human as claimed in claim 1, characterized in that: The music interpretation module: Voice Synthesis: Apply advanced voice synthesis techniques, including WaveNet, to generate near-realistic human voices; Motion synchronization: Use motion capture technology to synchronize the expressions and movements of AI digital people with the music; Environmental sound effects: According to the style and scene of the music, simulate the corresponding environmental sound effects, including reverberation, to enhance the sense of presence.

6. The AI ​​digital human-based voice music creation and interpretation system as claimed in claim 1, characterized in that: The user feedback and adjustment module: User Interface: Develop a user-friendly user interface that allows users to upload voice files, select a music style, and receive generated music; Feedback mechanism: Establish a feedback system where users can evaluate the generated music and provide suggestions for improvement; Data analysis: Analyze user feedback to optimize models and mapping rules.

7. A method for creating and performing voice music based on an AI digital human, which implements the voice music creation and performance system based on an AI digital human as described in any one of claims 1 to 6, characterized in that: The method for creating and performing voice music based on AI digital human includes: Step 1: Collect a large amount of music data and AI digital human voice data through the data collection module; pre-process the collected data through the data processing module, including data cleaning and feature extraction; Step 2: The central control module uses a deep learning model to extract features from the AI ​​digital human’s voice data through a feature extraction module to extract key features such as pitch, rhythm, and volume; Step 3: A mapping relationship between speech features and music elements is established through a music mapping and generation module, and the extracted speech features are converted into notes, melodies and rhythms to generate musical melodies and rhythms to form a complete music work; Step 4: Use sound synthesis technology to simulate the sound of real human singing through the music interpretation module, and use motion capture and expression recognition technology to synchronize the expression and movement of the AI ​​digital human with the rhythm of the music, while simulating the environmental sound effects of different singing scenes; Step 5: Establish a user feedback system through the user feedback and adjustment module to collect user feedback on music creation and interpretation, and adjust the music creation strategy and interpretation method based on the feedback to meet the user's personalized needs; perform functional testing and optimization on each module of the system through the test optimization module.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method for creating and interpreting voice music based on AI digital humans as described in claim 7.

9. A computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to execute the steps of the method for creating and interpreting voice music based on AI digital humans as described in claim 7.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the voice music creation and interpretation system based on AI digital human as described in any one of claims 1-6.

Images