Voice processing method and device based on artificial intelligence, computer equipment and medium

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to an artificial intelligence-based voice processing method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the feature extraction of voice data, and obtaining an acoustic unit sequence; performing rhythm feature extraction on the acoustic unit sequence to obtain rhythm features; encoding the acoustic unit sequence and the rhythm characteristics to obtain a specified acoustic unit sequence; reading historical context information corresponding to the user; fusing the historical context information, the rhythm features and the specified acoustic unit sequence to obtain a target acoustic unit sequence; reasoning the target acoustic unit sequence based on the target large language model to obtain text response data; and performing voice synthesis on the text response data to obtain voice response data and outputting the voice response data. In addition, the voice response data can be stored in the block chain. The method can be applied to voice processing scenes in the financial science and technology field and the digital medical field, and the quality of the generated voice response data is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and can be applied to fields such as fintech and digital healthcare, particularly to artificial intelligence-based speech processing methods, devices, computer equipment, and storage media. Background Technology

[0002] In the field of human-computer interaction, current spoken dialogue products suffer from multi-dimensional defects in practical applications, failing to meet the demands for natural, coherent, and personalized human-computer interaction. Common spoken dialogue systems, such as those based on cascaded ASR-LLM-TTS and traditional end-to-end models, all have certain limitations. In cascaded ASR-LLM-TTS systems, each module is optimized independently, easily leading to error propagation problems. For example, in the speech recognition stage, ASR's accuracy decreases in noisy environments, and erroneous text directly causes LLM to generate dialogue content that deviates from the user's intent. Although subsequent TTS can synthesize clear speech, semantic coherence is compromised. Furthermore, this cascaded system cannot achieve end-to-end collaborative optimization; the modal differences between speech and text are significant, and prosody and emotional information in speech are lost during text conversion, resulting in a substantial reduction in the naturalness of the final response, lower quality of the generated speech response, and decreased user satisfaction.

[0003] In the financial insurance sector, taking insurance consultation as an example, traditional customer service dialogues based on cascading systems may encounter problems when customers in noisy environments inquire about insurance policy details. For instance, ASR (Automatic Speech Recognition) might misinterpret the customer's voice, such as mistaking "the number of payouts for critical illness insurance." This leads to LLM (Limited Language Management) generating an incorrect response, which is then relayed to the customer via TTS (Text-to-Speech) synthesis, resulting in inaccurate information and hindering the customer's understanding of the insurance product, thus reducing their willingness to purchase. Similarly, in the medical field, when patients describe their symptoms to a medical consultation system via voice, ASR may misinterpret the information in noisy environments, such as mistaking "a persistent high fever for three days" as "a persistent high fever for less than three days." LLM may then generate inaccurate diagnostic suggestions based on this erroneous text, which, when relayed to the patient via TTS synthesis, could delay treatment and negatively impact the patient's health.

[0004] Therefore, there is an urgent need for a spoken dialogue system that can effectively solve the above problems and achieve natural, coherent, and personalized human-computer interaction. Summary of the Invention

[0005] The purpose of this application is to provide a speech processing method, apparatus, computer device, and storage medium based on artificial intelligence to solve the technical problems existing in current spoken dialogue systems.

[0006] Firstly, an artificial intelligence-based speech processing method is provided, including: Receive user-input voice data and extract features from the voice data to obtain the corresponding acoustic unit sequence; Prosodic features are extracted from the acoustic unit sequence to obtain the corresponding prosodic features; The acoustic unit sequence and the prosodic features are encoded based on a preset cross-modal prosodic encoder to obtain the corresponding specified acoustic unit sequence. Read the historical context information corresponding to the user from the preset context cache pool; The historical context information, the prosodic features, and the specified acoustic unit sequence are fused to obtain the corresponding target acoustic unit sequence; The target acoustic unit sequence is inferred based on a preset target large language model to obtain the corresponding text response data. The text response data is processed by speech synthesis to obtain the corresponding speech response data; The voice response data is then processed for output.

[0007] Secondly, an artificial intelligence-based voice processing device is provided, comprising: The processing module is used to receive voice data input by the user and extract features from the voice data to obtain the corresponding acoustic unit sequence. The extraction module is used to extract prosodic features from the acoustic unit sequence to obtain the corresponding prosodic features; The encoding module is used to encode the acoustic unit sequence and the prosodic features based on a preset cross-modal prosodic encoder to obtain the corresponding specified acoustic unit sequence. The reading module is used to read the historical context information corresponding to the user from a preset context cache pool; The fusion module is used to fuse the historical context information, the prosodic features, and the specified acoustic unit sequence to obtain the corresponding target acoustic unit sequence. The inference module is used to perform inference processing on the target acoustic unit sequence based on a preset target large language model to obtain the corresponding text response data. The synthesis module is used to perform speech synthesis processing on the text response data to obtain the corresponding speech response data; The output module is used to process the voice response data.

[0008] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described artificial intelligence-based speech processing method.

[0009] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the aforementioned artificial intelligence-based speech processing method.

[0010] In the above-mentioned AI-based speech processing method, device, computer equipment, and storage medium, the following steps are taken: First, user-input speech data is received, and feature extraction is performed on the speech data to obtain a corresponding acoustic unit sequence. Then, prosodic feature extraction is performed on the acoustic unit sequence to obtain corresponding prosodic features. Next, the acoustic unit sequence and the prosodic features are encoded using a preset cross-modal prosodic encoder to obtain a corresponding specified acoustic unit sequence. Subsequently, historical context information corresponding to the user is read from a preset context cache pool. The historical context information, the prosodic features, and the specified acoustic unit sequence are fused to obtain a corresponding target acoustic unit sequence. Further, the target acoustic unit sequence is inferred based on a preset target large language model to obtain corresponding text response data. The text response data is then processed for speech synthesis to obtain corresponding speech response data. Finally, the speech response data is output. Thus, by using a target large language model, this application performs comprehensive reasoning processing on the processed speech data, prosodic features, and historical context information, thereby enabling the automatic and accurate generation of speech response data that has user-personalized features, meets dynamic prosodic requirements, and is coherent with the context, effectively improving the quality of the generated speech response data. Attached Figure Description

[0011] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2 This is a flowchart of an embodiment of the artificial intelligence-based speech processing method according to this application; Figure 3 This is a schematic diagram of the structure of an embodiment of the artificial intelligence-based voice processing device according to this application; Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0014] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0015] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0016] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0017] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0018] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0019] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0020] It should be noted that the AI-based voice processing method provided in this application is generally executed by a server / terminal device, and correspondingly, the AI-based voice processing device is generally located in the server / terminal device.

[0021] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0022] Continue to refer to Figure 2 This document illustrates a flowchart of an embodiment of the AI-based speech processing method according to this application. The order of steps in the flowchart can be changed, and some steps can be omitted, depending on different requirements. The AI-based speech processing method provided in this application can be applied to any scenario requiring speech processing, and therefore can be applied to products in these scenarios, such as speech processing products in the financial insurance field. The AI-based speech processing method includes the following steps: Step S201: Receive voice data input by the user and extract features from the voice data to obtain the corresponding acoustic unit sequence.

[0023] In this embodiment, the artificial intelligence-based voice processing method runs on an electronic device (e.g., Figure 1The server / terminal device shown can acquire user-inputted voice data via wired or wireless connection. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra-wideband) connections, and other currently known or future-developed wireless connection methods. The implementing entity of this application is specifically a voice processing system, also known as a spoken dialogue system, which can be simply referred to as the system. The system can receive new user-inputted voice data through a dedicated input interface. This input interface has multiple data reception capabilities and can process voice and text data simultaneously. For voice data, analog signals are acquired through audio acquisition devices (such as microphones) and then converted into digital signals by an analog-to-digital converter (ADC) for subsequent processing. For text data, it may be directly input by the user through input devices such as keyboards or touchscreens, or it may be text converted from voice input using speech-to-text technology. The aforementioned input interface can also receive supplementary text information actively provided by the user, forming a multimodal input of "voice + optional text". By receiving multimodal input, the system can more comprehensively understand the user's intentions and needs. Voice input can capture emotional information such as the user's tone and intonation, while text input can provide more accurate semantic content. Combining the two can improve the system's understanding accuracy.

[0024] Furthermore, the specific implementation process of extracting features from the aforementioned speech data to obtain the corresponding acoustic unit sequence will be described in more detail in subsequent specific embodiments of this application, and will not be elaborated upon here.

[0025] Furthermore, this application can be applied to voice processing scenarios in the fintech and digital healthcare fields. For example, in an insurance consultation scenario within fintech, the content of the voice data input by the user may include: the user wants to know detailed information about a specific insurance product and asks the system a question via voice, such as: "I'd like to inquire about your company's newly launched health insurance. What critical illnesses does it mainly cover? What is the maximum coverage amount? How is the premium calculated? Is the premium the same for different age groups?" After receiving this voice data, the system will convert it into text for further processing, providing the user with accurate insurance product information. Alternatively, in an insurance claims scenario, the content of the voice data input by the user may include: the user reporting an accident and describing the situation via voice: "Yesterday, I accidentally rear-ended another vehicle. My car's rear was severely damaged, and the other vehicle also sustained some damage. I bought car insurance from your company; how do I apply for a claim now, and what materials do I need to prepare?" The system receives this voice, analyzes the user's needs, and guides the user through the claims process.

[0026] In the field of digital healthcare, in remote medical consultation scenarios, the content of voice data input by users can include: a patient in a remote area who cannot travel to the hospital in time communicates with a remote doctor via voice: "Doctor, I've been coughing a lot lately, especially at night. I can't sleep because of the cough, and I'm coughing up yellow phlegm. My throat is also very sore. I've taken some cough medicine, but it hasn't helped much." Based on the received voice information, the doctor further inquires about the patient's condition, makes a diagnosis, and provides a treatment plan. Alternatively, in health management scenarios, the content of voice data input by users can include: when using a health management app, users record their health data and daily situation via voice, for example: "I weighed myself this morning, and I'm 65 kg, 0.5 kg lighter than yesterday. I ate two eggs, a glass of milk, and a bread for breakfast. For lunch, I ate rice, stir-fried vegetables, and fish. I plan to have some fruit salad for dinner. Also, I jogged for 30 minutes today." The system receives the voice data, analyzes the user's health status, and provides personalized health management suggestions.

[0027] Step S202: Extract prosodic features from the acoustic unit sequence to obtain the corresponding prosodic features.

[0028] In this embodiment, the prosodic feature extraction includes: Fundamental frequency feature (F0) extraction: The YIN algorithm is used to calculate the fundamental frequency variation of the speech. The YIN algorithm determines the fundamental frequency by performing autocorrelation analysis on the speech signal and finding the peak of the autocorrelation function. It can capture the pitch of the speech and reflect the speaker's emotion and tone. Energy feature extraction: The short-time energy of each frame of speech is calculated. Short-time energy refers to the total energy of the speech signal within a short period of time, which can be obtained by integrating the square of the speech signal. Energy features can reflect the volume of the speech; for example, there will be a significant difference in energy values ​​between speaking loudly and speaking softly. Speech rate feature extraction: The number of acoustic units per unit time is counted, and the speech rate is determined by combining the pause interval. For example, a comma corresponds to a 50ms pause, and a period corresponds to a 100ms pause. By analyzing the number of acoustic units and the pause interval, the speech rate can be calculated. Then, the obtained fundamental frequency features, energy features, and speech rate features are integrated to obtain the corresponding prosodic features. Prosodic features are an important attribute of speech, encompassing information such as rhythm, intonation, and speech rate. Extracting these features helps the system better understand the user's speech input and generate speech responses that match the user's input prosody during speech synthesis, making the speech more natural and vivid.

[0029] Step S203: Encode the acoustic unit sequence and the prosodic features based on a preset cross-modal prosodic encoder to obtain the corresponding specified acoustic unit sequence.

[0030] In this embodiment, the specific implementation process of encoding the acoustic unit sequence and the prosodic features based on the preset cross-modal prosodic encoder to obtain the corresponding specified acoustic unit sequence will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0031] Step S204: Read the historical context information corresponding to the user from the preset context cache pool.

[0032] In this embodiment, the aforementioned context cache pool (or multi-turn context cache pool, or simply cache pool) stores keywords of user input and system response, stored dynamic prosodic vectors, user-specific timbre templates, and timbre parameters for each round of synthesized speech. Historical context information related to the user can be retrieved from the cache pool based on the context information of the current dialogue through indexing or querying.

[0033] Multi-turn contextual information is key to maintaining dialogue coherence. By reading historical information, the system can understand the content of previous conversations, user preferences, and system response methods, thus taking the previous context into account when generating new voice responses, making the dialogue more coherent and natural.

[0034] Step S205: The historical context information, the prosodic features, and the specified acoustic unit sequence are fused to obtain the corresponding target acoustic unit sequence.

[0035] In this embodiment, the specific implementation process of fusing the historical context information, the prosodic features, and the specified acoustic unit sequence to obtain the corresponding target acoustic unit sequence will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0036] Step S206: Based on the preset target large language model, the target acoustic unit sequence is subjected to inference processing to obtain the corresponding text response data.

[0037] In this embodiment, based on the aforementioned target large language model, inference calculations are performed using the input target acoustic unit sequence combined with cross-modal semantic associations, personalized adaptation information, and multi-turn contextual coherence processing capabilities learned during the pre-training phase to generate corresponding text response data. During the inference process, the target large language model considers the user's personalized characteristics (such as language style preferences influenced by basic attributes like age and gender), dynamic prosodic requirements, and the coherence of multi-turn dialogues. For example, if the user is young, the model may generate simpler, more colloquial text responses; if the current dialogue has a more lively prosodic style, the model's generated text responses will also try to conform to this prosodic style.

[0038] Furthermore, the specific construction process of the aforementioned target large language model will be described in more detail in subsequent specific embodiments of this application, and will not be elaborated upon here.

[0039] Step S207: Perform speech synthesis processing on the text response data to obtain the corresponding speech response data.

[0040] In this embodiment, the specific implementation process of performing speech synthesis processing on the text response data to obtain the corresponding speech response data will be further described in detail in subsequent specific embodiments of this application, and will not be elaborated on here.

[0041] Step S208: Output the voice response data.

[0042] In this embodiment, the synthesized voice response data can be output to the user through an audio output device (such as a speaker). During the output process, appropriate post-processing can be performed on the voice signal, such as volume adjustment and audio format conversion, to ensure that the output voice quality meets the requirements. The user can then intuitively hear the system's response, thus completing a full interaction.

[0043] After each round of dialogue, the aforementioned context cache pool will be automatically updated. The system will automatically delete expired contexts that have exceeded 10 rounds and update user interaction preferences to ensure the timeliness and accuracy of context modeling.

[0044] Based on the above automated processing flow, this application first receives voice data input by the user and extracts features from the voice data to obtain the corresponding acoustic unit sequence; then, it extracts prosodic features from the acoustic unit sequence to obtain the corresponding prosodic features; subsequently, it encodes the acoustic unit sequence and the prosodic features based on a preset cross-modal prosodic encoder to obtain the corresponding specified acoustic unit sequence; next, it reads the historical context information corresponding to the user from a preset context cache pool; and it fuses the historical context information, the prosodic features, and the specified acoustic unit sequence to obtain the corresponding target acoustic unit sequence; further, it performs inference processing on the target acoustic unit sequence based on a preset target large language model to obtain the corresponding text response data; and it performs speech synthesis processing on the text response data to obtain the corresponding speech response data; finally, it outputs the speech response data. Thus, by using a target large language model, this application performs comprehensive reasoning processing on the processed speech data, prosodic features, and historical context information, thereby enabling the automatic and accurate generation of speech response data that has user-personalized features, meets dynamic prosodic requirements, and is coherent with the context, effectively improving the quality of the generated speech response data.

[0045] In some optional implementations, step S201, which involves extracting features from the speech data to obtain the corresponding acoustic unit sequence, includes the following steps: Obtain the preset noise reduction strategy.

[0046] In this embodiment, the aforementioned denoising strategy refers to a speech denoising strategy, which includes: Short-Time Fourier Transform (SFT): The speech signal is divided into several short-time frames (typically 20-30ms in length), and a Fourier transform is performed on each frame to convert the time-domain signal into a frequency-domain signal. This allows the energy distribution of the speech signal at different frequencies to be obtained. Gaussian Mixture Model (GMM) to distinguish speech from noise: The GMM model is pre-trained using a large number of clean speech and noise samples, enabling it to learn the feature distributions of speech and noise in the frequency domain. When processing a new speech signal, the frequency domain features of each frame are input into the GMM model, and the model's output probability determines whether the frame belongs to speech or noise. Wavelet thresholding denoising: For the frequency domain components corresponding to noise, wavelet transform is used to decompose the signal into wavelet spaces of different scales. Then, based on the difference between noise and speech in the wavelet coefficients, an appropriate threshold is set to perform thresholding on the wavelet coefficients, suppressing the wavelet coefficients corresponding to noise and retaining the wavelet coefficients corresponding to speech. Finally, the processed wavelet coefficients are converted back to the frequency domain signal through inverse wavelet transform, and then inversely transformed back to the time-domain speech.

[0047] The speech data is denoised based on the denoising strategy to obtain the corresponding processed speech.

[0048] In this embodiment, denoising processing of the aforementioned speech data can be performed based on the denoising strategy described above to obtain the corresponding processed speech. This denoising process removes interfering factors, making subsequent acoustic unit extraction more accurate, thereby improving the performance of speech recognition and speech synthesis.

[0049] Based on a pre-defined self-supervised model, features are extracted from the processed speech to obtain corresponding intermediate features.

[0050] In this embodiment, the self-supervised model described above can specifically be the XLS-R self-supervised model. This model can be used to extract features from the denoised speech. Through self-supervised learning, the model learns the feature representation of speech from a large amount of unlabeled speech data. During feature extraction, the speech is processed at a frame rate of 50Hz to extract the intermediate features of the processed speech.

[0051] The intermediate features are clustered based on a preset clustering algorithm to convert them into corresponding discrete acoustic feature sequences.

[0052] In this embodiment, the clustering algorithm described above can specifically employ the k-means clustering algorithm. By performing k-means clustering (k=10000) on the extracted intermediate features, similar feature vectors are grouped into one class, with each class corresponding to one acoustic unit. In this way, continuous speech signals are converted into discrete acoustic unit sequences. Simultaneously, the original unit sequence is retained without deduplication to preserve prosodic details such as pauses and stresses.

[0053] The discrete acoustic feature sequence is used as the acoustic feature sequence.

[0054] Based on the above processing flow, this application obtains a preset denoising strategy; then, based on the denoising strategy, it denoises the speech data to obtain the corresponding processed speech; next, it extracts features from the processed speech using a preset self-supervised model to obtain corresponding intermediate features; then, it clusters the intermediate features using a preset clustering algorithm to convert the intermediate features into corresponding discrete acoustic feature sequences; finally, it uses the discrete acoustic feature sequences as acoustic feature sequences. Feature extraction and acoustic unit generation in this application are key steps in converting speech signals into a computer-processable form. By using a self-supervised model, high-level feature representations of speech can be learned, while clustering discretizes these features into acoustic units, facilitating subsequent speech processing and synthesis. Preserving the original unit sequence can retain the prosodic information of the speech, making the synthesized speech more natural.

[0055] In some optional implementations of this embodiment, step S203 includes the following steps: Based on a preset emotion classifier, the prosodic features are used to perform emotion recognition, and the corresponding emotion recognition results are obtained.

[0056] In this embodiment, a 4-layer Transformer model trained on the CREMA-D + DailyTalk dataset is used as the emotion classifier. This emotion classifier learns from a large amount of speech data with emotion labels, automatically extracting emotional features from the speech and classifying them into different emotion categories, such as happiness, sadness, and anger. Furthermore, by inputting the aforementioned prosodic features into this emotion classifier, the classifier performs emotion recognition on the prosodic features and outputs the corresponding emotion recognition result.

[0057] Scene noise residue detection is performed on the voice data to obtain the corresponding scene detection results.

[0058] In this embodiment, the scene can be determined by combining the residual scene noise in the input speech data, and the corresponding scene detection result can be output. For example, if "traffic noise" is detected, it is determined to be an "outdoor scene". The scene can be determined by analyzing the noise features in the speech signal and matching them with the noise features of different predefined scenes.

[0059] The emotion recognition result and the scene detection result are combined to obtain the corresponding joint label.

[0060] In this embodiment, a joint label of "emotion + scene" can be generated by combining the emotion recognition results output by the emotion classifier with the scene detection results. Emotion and scene information are crucial for speech understanding and generation. Different emotions and scenes affect the prosody, semantics, and expression of speech. Through emotion and scene judgment, the system can better understand the user's voice input and generate appropriate voice responses based on different emotions and scenes during speech synthesis, improving the naturalness and adaptability of the speech.

[0061] The acoustic unit sequence and the joint label are encoded based on the cross-modal prosodic encoder to obtain the corresponding enhanced acoustic unit sequence.

[0062] In this embodiment, the encoding process includes inputting the acoustic unit sequence and the joint label of "emotion + scene" into a cross-modal prosodic encoder. The cross-modal prosodic encoder employs an attention mechanism, which fuses prosodic features with acoustic units by calculating the correlation weights between the acoustic unit sequence and the joint label. The attention mechanism allows the model to focus more on prosodic-related acoustic units, thereby generating an enhanced acoustic unit sequence with dynamic prosodic information.

[0063] The enhanced acoustic unit sequence is used as the designated acoustic unit sequence.

[0064] Based on the above processing flow, this application performs emotion recognition on prosodic features using a pre-defined emotion classifier to obtain corresponding emotion recognition results; then, it performs scene noise residue detection on the speech data to obtain corresponding scene detection results; subsequently, it combines the emotion recognition results and scene detection results to obtain corresponding joint labels; next, it encodes the acoustic unit sequence and joint labels using a cross-modal prosodic encoder to obtain corresponding enhanced acoustic unit sequences; finally, it uses the enhanced acoustic unit sequence as the designated acoustic unit sequence. Thus, the use of cross-modal prosodic encoding in this application is a key step in deeply fusing prosodic information with acoustic units. Through an attention mechanism, the model can automatically learn the correlation between prosodic features and acoustic units, enabling the enhanced acoustic unit sequence to contain rich prosodic information, providing richer cross-modal semantics for subsequent pre-trained models, and helping to improve the quality and naturalness of speech synthesis.

[0065] In some alternative implementations, step S205 includes the following steps: Invoke the preset context attention layer.

[0066] In this embodiment, the aforementioned context attention layer is a pre-built neural network layer used to fuse different information (in this scenario, historical keywords, prosodic vectors, and the enhanced acoustic unit sequence of the current input). Its core function is to calculate the correlation weight between the current input information and the historical information.

[0067] Obtain the preset fusion strategy.

[0068] In this embodiment, the aforementioned fusion strategy includes the following: the context attention layer analyzes the correlation between the current input enhanced acoustic unit sequence and historical information such as historical keywords and prosodic vectors, assigning different weights to different parts of the historical information. These weights reflect the importance of historical information to the current input. Furthermore, based on the calculated relevance weights, the context attention layer performs weighted fusion of the historical information. That is, according to the weights, historical information is selectively integrated into the processing of the current input, enabling the model to comprehensively consider both historical and current information, thereby making more reasonable decisions or generating more suitable results.

[0069] Based on the fusion strategy, the context attention layer is used to fuse the historical context information, the prosodic features, and the specified acoustic unit sequence to obtain the corresponding processed sequence.

[0070] In this embodiment, based on the strategy content of the above fusion strategy, the above context attention layer can be used to fuse historical context information, prosodic features and specified acoustic unit sequences, and the resulting processed sequence can be used as the desired target acoustic unit sequence.

[0071] The processing sequence is used as the target acoustic unit sequence.

[0072] Based on the above processing flow, this application invokes a preset context attention layer; then obtains a preset fusion strategy; subsequently, based on the fusion strategy, the context attention layer is used to fuse historical context information, prosodic features, and a specified acoustic unit sequence to obtain a corresponding processed sequence; this processed sequence is then used as the target acoustic unit sequence. The context attention fusion in this application is a key step in effectively integrating historical information with the current input. Through the attention mechanism, the large language model can automatically learn the relationship between historical information and the current input, making the fused information more consistent with the contextual logic of the dialogue. Ensuring consistent trends in prosodic vector changes can avoid prosodic discontinuity during speech synthesis, thereby improving the naturalness and fluency of the speech.

[0073] In some alternative implementations, step S207 includes the following steps: Obtain the timbre information corresponding to the user.

[0074] In this embodiment, the aforementioned timbre information includes the user-specific timbre template and timbre parameters obtained from the multi-round context cache pool.

[0075] Invoke the preset speech synthesis tool.

[0076] In this embodiment, the selection of the above-mentioned speech synthesis tool is not specifically limited, and can be determined according to the actual business needs. For example, a general speech synthesis module can be selected.

[0077] Based on the timbre information, the speech synthesis tool is used to perform speech synthesis processing on the text response data to obtain the corresponding generated speech.

[0078] In this embodiment, by inputting the aforementioned text response data into a selected speech synthesis tool, the tool synthesizes speech based on the user-specific timbre template and timbre parameters obtained from a multi-turn context cache pool. During the synthesis process, previously extracted and processed dynamic prosodic information, such as fundamental frequency, energy, and speech rate, is incorporated to ensure that the synthesized speech (generated speech) matches the user's input prosody and conforms to the current dialogue scenario and emotional expression. For example, if the current dialogue scenario is sad, the synthesized speech will adjust its fundamental frequency and energy accordingly, making it sound lower and slower.

[0079] The generated speech is used as the speech response data.

[0080] Based on the above processing flow, this application obtains the user's corresponding timbre information; then calls a preset speech synthesis tool; subsequently, based on the timbre information, the speech synthesis tool performs speech synthesis processing on the text response data to obtain the corresponding generated speech; the generated speech is then used as the speech response data. By combining the user's exclusive timbre template and dynamic prosodic information, this application utilizes a speech synthesis tool to generate personalized speech responses that meet dynamic prosodic requirements and are coherent with the context, enabling users to interact with the system more naturally.

[0081] In some optional implementations of this embodiment, before step S206, the electronic device may further perform the following steps: Obtain pre-constructed pre-trained sequence data.

[0082] In this embodiment, a large-scale data collection effort was undertaken. Different types of speech data were acquired from multiple channels to construct a vast and diverse dataset. Specifically, this includes 87,000 hours of English ASR data, with Multilingual LibriSpeech providing 44,500 hours of rich material and People's Speech contributing 30,000 hours. This data covers English speech and corresponding text with different accents, speaking speeds, and language styles; 5,000 hours of multi-scene noisy speech data, simulating various real-world noise conditions such as street noise and indoor echoes, enabling the model to adapt to speech recognition in complex environments; 3,000 hours of emotion-annotated speech data, annotating emotional information such as joy, anger, sorrow, and happiness in the speech, helping the model understand the relationship between speech emotion and text semantics; and 10,000 hours of multilingual speech data, involving multiple languages, broadening the model's language processing capabilities. Integrating this data provides the model with comprehensive and diverse learning materials.

[0083] Furthermore, the construction process of the pre-training sequence data includes: Speech-text alignment: The Montreal ForcedAligner (MFA) tool is used to perform fine alignment between the collected speech and the corresponding text. This tool can accurately identify the start and end times of each word in the speech, and then convert these time intervals into 50Hz acoustic unit index intervals. For example, for the speech and text of the sentence "Hello, world!", MFA can determine that "Hello" in the speech is from 0.5 seconds to 1.2 seconds, and the corresponding acoustic unit index interval is [10, 24] (assuming one unit every 0.02 seconds at 50Hz), thus establishing a precise preliminary correspondence between speech and text. Segmentation and modality selection: Each speech-text pair is segmented into N segments (the value of N can be set according to the actual situation, such as 5 segments). For each segment, a dominant modality is randomly selected, either a speech unit or text. For example, if the first segment selects a speech unit as the primary modality and the second segment selects text as the primary modality, this random selection method increases data diversity, allowing the model to encounter various modality combinations and thus better learn the relationship between speech and text. Inserting submodalities and special tokens: After determining the primary modality, a submodality is inserted with a 50% probability. If the primary modality in a segment is a speech unit, and a submodality is inserted, the submodality might be the corresponding text. Inserting special tokens at modality transitions: If the primary and submodal modalities belong to the same segment, such as "speech unit + text," then <|correspond|> is inserted; if they belong to different segments, such as "text + the next speech unit," then <|continue|> is inserted. Through these special tokens, the model can clearly distinguish the "correspondence" and "continuation" relationships between speech and text. For example, seeing <|correspond|> indicates a direct correspondence between speech and text, while seeing <|continue|> indicates a transition between different segments.

[0084] Invoke a preset initial large language model; wherein, the initial large language model is a model obtained by adding a cross-modal attention layer and expanding the vocabulary to the preset large language model.

[0085] In this embodiment, the aforementioned large language model can specifically use Mistral-7B as the backbone model. The newly added cross-modal attention layer includes introducing a cross-modal attention layer during the construction of the speech-text interleaving sequence. This special layer acts like an intelligent "regulator," dynamically adjusting the correlation strength between speech units and text tokens by calculating attention weights. Taking the processing of a speech containing emotional expression and its corresponding text as an example, when the speech contains an angry emotional prosody, the cross-modal attention layer will increase the attention weight of words related to anger in the text, helping the model more accurately capture the close connection between the emotional prosody in the speech and the semantics of the text, thereby improving the model's ability to understand the comprehensive information of speech and text.

[0086] The expanded vocabulary includes: selecting Mistral-7B as the backbone model and expanding upon its original 10,000 acoustic unit tokens. Two new relation-specific tokens (<|correspond| and <|continue|>) and 100 "emotion-scene" tag tokens were added. For example, a new tag token, "happy-party scene," was added to describe the speech-text features under specific emotions and scenes. During the expansion, only the embedding layers of the newly added tokens were reinitialized, like initializing new "parts" of the model, while retaining the LLM's text-based dialogue capabilities. This ensures that the model enhances its speech-text processing capabilities without losing its original text-based dialogue functionality.

[0087] Based on a preset comprehensive loss function, the initial large language model is trained using the pre-trained sequence data to obtain a trained first generative model.

[0088] In this embodiment, the comprehensive loss function is obtained by combining the prosodic consistency loss, the autoregressive cross-entropy loss, and the original training objective loss. The specific training process for the initial large language model includes: 1. Data preparation and forward propagation. Data input: The interleaving sequence containing information such as text, units, and special tokens is input into the model, and the dynamic prosodic vector of the input is obtained. This vector is used for the subsequent calculation of the prosodic consistency loss. Model prediction: Based on the input data, the model gradually generates the output sequence through autoregression. During the generation process, the model will predict the element (such as speech units) at the next position based on the currently generated partial sequence and input information. Calculate the prediction probability: For each prediction position, the model will output a probability distribution, representing the probability of generating each possible element at that position. 2. Loss calculation. Refer to the calculation process of the comprehensive loss function described later. 3. Backpropagation and parameter update. Backpropagation: Based on the calculated total loss function, the gradient of the model parameters is calculated using the backpropagation algorithm. The gradient represents the sensitivity of the loss function to changes in each parameter, that is, how the parameters should be adjusted to reduce the loss function. Parameter Update: Using optimization algorithms (such as stochastic gradient descent, Adam, etc.), the model parameters are updated based on the calculated gradient. The optimization algorithm adjusts the parameter values ​​in the opposite direction of the gradient according to a certain learning rate, so that the model can generate less loss in the next forward propagation. 4. Iterative Training: Repeat the above process of forward propagation, loss calculation, backpropagation, and parameter update to continuously train the model. In each iteration, the model will adjust according to the new data and the feedback of the loss function, gradually improving its ability to model the speech-text relationship, generating more accurate sequences that meet the requirements, and ensuring the prosodic consistency of the generated speech. As training progresses, the model's performance will gradually improve until it reaches the preset training objective or meets certain convergence conditions, thus obtaining the desired first generative model.

[0089] The first generative model is fine-tuned based on a preset fine-tuning dataset to obtain the corresponding second generative model.

[0090] In this embodiment, a user profile database is first established to comprehensively store users' personalized information. Basic user attributes, such as age and gender, are collected, as these influence speech rate and tone preferences. For example, young people may speak faster and in a higher tone, while older people may speak slower and in a lower tone. Simultaneously, the user's historical dialogues are analyzed in depth to uncover their interaction preferences, such as whether they frequently discuss technology, sports, or entertainment topics, and whether their language style is formal or casual. Furthermore, advanced audio processing technology is used to extract the user's timbre features, essentially creating a "portrait" of the user's voice, so that the generated speech more closely resembles the user's real voice characteristics. The construction process of the aforementioned fine-tuning dataset includes: filtering dialogue data matching the user profile from DailyTalk and Fisher datasets; further filtering dialogue data that matches the user's personality and preferences by combining the user's historical dialogue records to construct a personalized fine-tuning dataset. For example, if the user profile shows that the user is interested in technology topics, then dialogues containing technology-related content will be prioritized when filtering data. The user's unique timbre template is converted into a "timbre constraint label" and embedded into the dialogue template of the fine-tuning data. In this way, during the fine-tuning process, the model can learn the user's vocal characteristics, and can better simulate the user's voice when generating dialogue speech, thus improving the effect of personalized dialogue.

[0091] Subsequently, the first generative model is fine-tuned and trained using the aforementioned fine-tuning dataset. In this way, the model first learns comprehensively on a rich variety of speech-text data, mastering the complex relationship between speech and text. Then, it is fine-tuned in a targeted manner based on the user's personalized information. The final result is a second generative model that can provide users with highly personalized, natural and fluent dialogue services, which serves as the target large language model.

[0092] The second generative model is used as the target large language model.

[0093] Based on the above processing flow, this application obtains pre-constructed pre-trained sequence data; then calls a preset initial large language model; wherein the initial large language model is a model obtained by adding a cross-modal attention layer and expanding the vocabulary to the preset large language model; then, based on a preset comprehensive loss function, the initial large language model is trained using the pre-trained sequence data to obtain a trained first generative model; subsequently, the first generative model is fine-tuned based on a preset fine-tuning dataset to obtain a corresponding second generative model; finally, the second generative model is used as the target large language model, thereby achieving a carefully constructed model through two stages: unified speech-text pre-training enhancement and personalized dialogue fine-tuning. This allows the model to first learn comprehensively on rich and diverse speech-text data, mastering the complex relationship between speech and text, and then perform targeted fine-tuning based on the user's personalized information. Ultimately, it can automatically and accurately construct a target large language model to provide users with highly personalized, natural, and fluent dialogue services, effectively improving the construction efficiency of the target large language model and ensuring the model performance of the obtained target large language model.

[0094] In some optional implementations of this embodiment, before the step of training the initial large language model using the pre-trained sequence data based on a preset comprehensive loss function to obtain the trained first generative model, the electronic device may also perform the following steps: Obtain the preset prosodic consistency loss and autoregressive cross-entropy loss.

[0095] In this embodiment, by incorporating a "prosodic consistency loss" (i.e., the generated speech unit sequence must match the input dynamic prosodic vector), the pre-trained model is ensured not only to learn semantic correspondences but also to maintain prosodic coherence. Specifically, the prosodic consistency loss focuses on the prosodic features of the generated speech. By matching the generated speech unit sequence with the input dynamic prosodic features, this loss prompts the model to maintain consistency with the input speech in terms of pitch, duration, and rate of speech, thereby making the generated speech more natural and fluent, conforming to the prosodic habits of human speech.

[0096] The autoregressive cross-entropy loss described above measures the accuracy of the model in generating interleaving sequences (containing text, units, and specific tokens). It evaluates the difference between the model-generated sequence and the real sequence by calculating the conditional probability of each token in the generated sequence, prompting the model to continuously optimize its parameters and improve its ability to model speech-text relationships. Specifically, the autoregressive cross-entropy loss is as follows: ,in It is the j-th interleaving sequence (containing text, units, and special tokens).

[0097] Obtain the preset combination strategy.

[0098] In this embodiment, the strategy of the above-mentioned combination strategy includes: assuming the original training objective loss is Loriginal, the prosody consistency loss is Lprosody, and the autoregressive cross-entropy loss is Lauto. For regressive scenarios, the corresponding comprehensive pre-training loss function Ltotal can be expressed as: Ltotal = λ1 * Loriginal + λ2 * Lprosody + λ3 * Lauto The regressive function is defined as follows: λ1, λ2, and λ3 are the weight coefficients corresponding to each loss function, and λ1+λ2+λ3=1. These weight coefficients are used to balance the importance of different loss functions in the overall training process.

[0099] The weighting coefficients λ1, λ2, and λ3 need to be adjusted based on the specific task and model requirements. For example, if the core task of the model is to accurately achieve the correspondence and continuation of speech-text, while the requirement for prosody is relatively low, then the value of λ1 can be appropriately increased and the value of λ2 decreased. If it is desired that the generated speech is more natural in prosody and closer to human speech, the value of λ2 can be increased. When emphasizing the accuracy of the model in generating interleaving sequences, the value of λ3 can be increased.

[0100] Based on the aforementioned combination strategy, the prosodic consistency loss, the autoregressive cross-entropy loss, and the original training objective loss are combined to obtain the corresponding combined loss function.

[0101] In this embodiment, the original training objective loss reflects the model's performance on the initially defined core tasks such as speech-text correspondence and continuation. For example, in the speech-text correspondence task, this loss measures the accuracy of the model in determining whether speech segments and text segments correspond; in the continuation task, it measures the accuracy of the model in predicting subsequent speech or text content. This part of the loss ensures that the model can complete basic speech-text association tasks.

[0102] Specifically, the prosodic consistency loss, autoregressive cross-entropy loss, and the original training objective loss can be combined based on the strategy content of the above combination strategy, and the resulting combined loss function can be used as the final comprehensive loss function.

[0103] The combined loss function is used as the comprehensive loss function.

[0104] Based on the above processing flow, this application obtains preset prosodic consistency loss and autoregressive cross-entropy loss; then obtains a preset combination strategy; subsequently, it combines the prosodic consistency loss, autoregressive cross-entropy loss, and the original training objective loss based on the combination strategy to obtain the corresponding combined loss function; finally, it uses the combined loss function as the comprehensive loss function. Thus, this application uses the strategy content based on the combination strategy to combine the prosodic consistency loss, autoregressive cross-entropy loss, and the original training objective loss to automatically and accurately construct the comprehensive loss function, allowing the large language model to adjust its parameters according to the comprehensive loss function during training. By minimizing this comprehensive pre-training loss function, the model can simultaneously learn the semantic correspondence between speech and text, prosodic coherence, and the ability to accurately generate interleaving sequences, thereby comprehensively improving the understanding and processing ability of speech-text relationships.

[0105] In some alternative implementations, the user information obtained is subject to user consent and complies with relevant laws and policies.

[0106] Furthermore, any software tools or components not belonging to our company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0107] Furthermore, the unified spoken dialogue system based on dynamic prosodic adaptation and personalized modeling proposed in this application can solve the problems of error and modal fragmentation in cascaded systems. It reduces ASR errors through a noise filtering module, enhances the semantic prosodic association between speech and text through cross-modal pre-training, and avoids error propagation in cascaded architectures. Simultaneously, dynamic prosodic encoding preserves emotional and contextual information in speech, solving the problem of low naturalness caused by the modal fragmentation between speech and text in traditional systems, resulting in a smoother user interaction experience. Compared to the static prosodic modeling of existing models, this system achieves dynamic matching between user input prosodic and system response prosodic through multi-dimensional prosodic extraction, emotion and scene mapping, and real-time prosodic adjustment. The personalized modeling module customizes interaction styles for different users through a user profile database and exclusive timbre templates, while a multi-turn context cache pool ensures cross-turn coherence of semantics, prosodic, and timbre, avoiding repetitive information from users and significantly improving the efficiency and coherence of multi-turn dialogues, making it particularly suitable for long-term interaction scenarios such as customer service and smart homes. The noise filtering module and multi-scene pre-training data significantly improve the accuracy of speech processing in complex environments such as home, outdoors, and office. The multilingual data extension supports dialogues in multiple languages, including Chinese, English, and Spanish, breaking through the language limitations of the existing English model and making it more widely applicable.

[0108] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0109] It should be emphasized that, to further ensure the privacy and security of the aforementioned voice response data, the voice response data can also be stored in a blockchain node.

[0110] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0111] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0112] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0113] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0114] Further reference Figure 3 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of an artificial intelligence-based speech processing device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0115] like Figure 3 As shown, the AI-based voice processing device 300 described in this embodiment includes: a processing module 301, an extraction module 302, an encoding module 303, a reading module 304, a fusion module 305, an inference module 306, a synthesis module 307, and an output module 308. Wherein: The processing module 301 is used to receive voice data input by the user and extract features from the voice data to obtain the corresponding acoustic unit sequence. Extraction module 302 is used to extract prosodic features from the acoustic unit sequence to obtain the corresponding prosodic features; The encoding module 303 is used to encode the acoustic unit sequence and the prosodic features based on a preset cross-modal prosodic encoder to obtain the corresponding specified acoustic unit sequence. The reading module 304 is used to read the historical context information corresponding to the user from a preset context cache pool; The fusion module 305 is used to fuse the historical context information, the prosodic features, and the specified acoustic unit sequence to obtain the corresponding target acoustic unit sequence. Inference module 306 is used to perform inference processing on the target acoustic unit sequence based on a preset target large language model to obtain the corresponding text response data; Synthesis module 307 is used to perform speech synthesis processing on the text response data to obtain corresponding speech response data; The output module 308 is used to output the voice response data.

[0116] In some optional implementations of this embodiment, the processing module 301 includes: The first acquisition submodule is used to acquire the preset noise reduction strategy; The noise reduction submodule is used to perform noise reduction processing on the speech data based on the noise reduction strategy to obtain the corresponding processed speech. The extraction submodule is used to extract features from the processed speech based on a preset self-supervised model to obtain corresponding intermediate features; The clustering submodule is used to perform clustering processing on the intermediate features based on a preset clustering algorithm, so as to convert the intermediate features into corresponding discrete acoustic feature sequences; The first determining submodule is used to use the discrete acoustic feature sequence as the acoustic feature sequence.

[0117] In some optional implementations of this embodiment, the encoding module 303 includes: The recognition submodule is used to perform emotion recognition on the prosodic features based on a preset emotion classifier to obtain the corresponding emotion recognition result; The detection submodule is used to perform scene noise residual detection on the voice data and obtain the corresponding scene detection results; The combined submodule is used to combine the emotion recognition result and the scene detection result to obtain the corresponding joint label; The encoding submodule is used to encode the acoustic unit sequence and the joint tag based on the cross-modal prosodic encoder to obtain the corresponding enhanced acoustic unit sequence; The second determining submodule is used to use the enhanced acoustic unit sequence as the designated acoustic unit sequence.

[0118] In some optional implementations of this embodiment, the fusion module 305 includes: The first calling submodule is used to invoke the preset context attention layer; The second acquisition submodule is used to acquire the preset fusion strategy; The fusion submodule is used to perform fusion processing on the historical context information, the prosodic features, and the specified acoustic unit sequence using the context attention layer based on the fusion strategy, to obtain a corresponding processing sequence. The third determining submodule is used to use the processing sequence as the target acoustic unit sequence.

[0119] In some optional implementations of this embodiment, the synthesis module 307 includes: The third acquisition submodule is used to acquire the timbre information corresponding to the user; The second submodule is used to invoke preset speech synthesis tools; The synthesis submodule is used to perform speech synthesis processing on the text response data based on the timbre information using the speech synthesis tool to obtain the corresponding generated speech; The fourth determining submodule is used to use the generated speech as the speech response data.

[0120] In some optional implementations of this embodiment, the AI-based voice processing device further includes: The first acquisition module is used to acquire pre-constructed pre-trained sequence data; The calling module is used to call the preset initial large language model; wherein, the initial large language model is a model obtained by adding a cross-modal attention layer and expanding the vocabulary to the preset large language model; The training module is used to train the initial large language model using the pre-trained sequence data based on a preset comprehensive loss function, so as to obtain a trained first generative model. The fine-tuning module is used to fine-tune the first generative model based on a preset fine-tuning dataset to obtain the corresponding second generative model. The first determining module is used to use the second generated model as the target large language model.

[0121] In some optional implementations of this embodiment, the AI-based voice processing device further includes: The second acquisition module is used to acquire the preset prosodic consistency loss and autoregressive cross-entropy loss; The third acquisition module is used to acquire preset combination strategies; The combination module is used to combine the prosodic consistency loss, the autoregressive cross-entropy loss, and the original training target loss based on the combination strategy to obtain the corresponding combined loss function. The second determining module is used to use the combined loss function as the comprehensive loss function.

[0122] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0123] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected via a system bus. It should be noted that only the computer device 4 with components 41-43 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0124] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0125] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 may be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 may also be an external storage device of the computer device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 4. Of course, the memory 41 may also include both the internal storage unit and its external storage device of the computer device 4. In this embodiment, the memory 41 is typically used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for artificial intelligence-based speech processing methods. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or will be output.

[0126] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is typically used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions stored in the memory 41 or to process data, for example, to execute computer-readable instructions of the artificial intelligence-based speech processing method.

[0127] The network interface 43 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 4 and other electronic devices.

[0128] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the artificial intelligence-based speech processing method described above.

[0129] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0130] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A speech processing method based on artificial intelligence, characterized in that, Includes the following steps: Receive user-input voice data and extract features from the voice data to obtain the corresponding acoustic unit sequence; Prosodic features are extracted from the acoustic unit sequence to obtain the corresponding prosodic features; The acoustic unit sequence and the prosodic features are encoded based on a preset cross-modal prosodic encoder to obtain the corresponding specified acoustic unit sequence. Read the historical context information corresponding to the user from the preset context cache pool; The historical context information, the prosodic features, and the specified acoustic unit sequence are fused to obtain the corresponding target acoustic unit sequence; The target acoustic unit sequence is inferred based on a preset target large language model to obtain the corresponding text response data. The text response data is processed by speech synthesis to obtain the corresponding speech response data; The voice response data is then processed for output.

2. The speech processing method based on artificial intelligence according to claim 1, characterized in that, The step of extracting features from the speech data to obtain the corresponding acoustic unit sequence specifically includes: Obtain the preset noise reduction strategy; The speech data is denoised based on the denoising strategy to obtain the corresponding processed speech. Based on a pre-defined self-supervised model, feature extraction is performed on the processed speech to obtain corresponding intermediate features; The intermediate features are clustered based on a preset clustering algorithm to convert them into corresponding discrete acoustic feature sequences. The discrete acoustic feature sequence is used as the acoustic feature sequence.

3. The artificial intelligence-based speech processing method according to claim 1, characterized in that, The step of encoding the acoustic unit sequence and the prosodic features based on a preset cross-modal prosodic encoder to obtain the corresponding specified acoustic unit sequence specifically includes: Based on a preset emotion classifier, the prosodic features are used to perform emotion recognition, and the corresponding emotion recognition results are obtained. Scene noise residue detection is performed on the voice data to obtain the corresponding scene detection results; The emotion recognition result and the scene detection result are combined to obtain the corresponding joint label; The acoustic unit sequence and the joint label are encoded based on the cross-modal prosodic encoder to obtain the corresponding enhanced acoustic unit sequence; The enhanced acoustic unit sequence is used as the designated acoustic unit sequence.

4. The speech processing method based on artificial intelligence according to claim 1, characterized in that, The step of fusing the historical context information, the prosodic features, and the specified acoustic unit sequence to obtain the corresponding target acoustic unit sequence specifically includes: Invoke the preset context attention layer; Obtain the preset fusion strategy; Based on the fusion strategy, the context attention layer is used to fuse the historical context information, the prosodic features, and the specified acoustic unit sequence to obtain the corresponding processing sequence. The processing sequence is used as the target acoustic unit sequence.

5. The artificial intelligence-based speech processing method according to claim 1, characterized in that, The step of performing speech synthesis processing on the text response data to obtain corresponding speech response data specifically includes: Obtain the timbre information corresponding to the user; Invoke the preset speech synthesis tool; Based on the timbre information, the speech synthesis tool is used to perform speech synthesis processing on the text response data to obtain the corresponding generated speech; The generated speech is used as the speech response data.

6. The artificial intelligence-based speech processing method according to claim 1, characterized in that, Before the step of performing inference processing on the target acoustic unit sequence based on a preset target large language model to obtain the corresponding text response data, the method further includes: Obtain pre-constructed pre-trained sequence data; Invoke a preset initial large language model; wherein, the initial large language model is a model obtained by adding a cross-modal attention layer and expanding the vocabulary to the preset large language model; Based on a preset comprehensive loss function, the initial large language model is trained using the pre-trained sequence data to obtain a trained first generative model; The first generative model is fine-tuned based on a preset fine-tuning dataset to obtain the corresponding second generative model. The second generative model is used as the target large language model.

7. The artificial intelligence-based speech processing method according to claim 6, characterized in that, Before the step of training the initial large language model using the pre-trained sequence data based on the preset comprehensive loss function to obtain the trained first generative model, the method further includes: Obtain the preset prosodic consistency loss and autoregressive cross-entropy loss; Obtain the preset combination strategy; Based on the aforementioned combination strategy, the prosodic consistency loss, the autoregressive cross-entropy loss, and the original training objective loss are combined to obtain the corresponding combined loss function. The combined loss function is used as the comprehensive loss function.

8. A voice processing device based on artificial intelligence, characterized in that, include: The processing module is used to receive voice data input by the user and extract features from the voice data to obtain the corresponding acoustic unit sequence. The extraction module is used to extract prosodic features from the acoustic unit sequence to obtain the corresponding prosodic features; The encoding module is used to encode the acoustic unit sequence and the prosodic features based on a preset cross-modal prosodic encoder to obtain the corresponding specified acoustic unit sequence. The reading module is used to read the historical context information corresponding to the user from a preset context cache pool; The fusion module is used to fuse the historical context information, the prosodic features, and the specified acoustic unit sequence to obtain the corresponding target acoustic unit sequence. The inference module is used to perform inference processing on the target acoustic unit sequence based on a preset target large language model to obtain the corresponding text response data. The synthesis module is used to perform speech synthesis processing on the text response data to obtain the corresponding speech response data; The output module is used to process the voice response data.

9. A computer device, characterized in that, It includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the artificial intelligence-based speech processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the artificial intelligence-based speech processing method as described in any one of claims 1 to 7.

Images