Method and device for generating ASR audio corpus based on multi-modal large model
Generating high-quality audio corpus through multimodal large models solves the problem of insufficient corpus diversity in the prior art, and improves the adaptability and recognition accuracy of ASR systems in complex environments.
Patent Information
- Application Number
- CN202510619927.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-14
- Publication Date
- 2025-07-18
- Estimated Expiration
- 2045-05-14
AI Technical Summary
The existing audio corpus generation methods are costly and difficult to cover all scenarios and speaker characteristics, resulting in insufficient corpus diversity. The speech synthesized by simple text-to-speech technology lacks real scene noise, affecting the adaptability of ASR systems in complex environments.
The multimodal large model is used to encode the target domain text into a semantic vector, splicing it into a joint vector, and the speech control parameters and speaker characteristics, to generate noisy speech, and through adversarial noise training, combining noise annotation, text annotation and emotional annotation, high-quality corpus that meets specific needs is selected.
It improves the matching degree between the corpus and actual application scenarios, enhances the robustness and recognition accuracy of the ASR system in complex environments, and reduces training costs.
Smart Images

Figure CN120340506A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the technical field of audio corpora, and specifically relates to a method and device for generating ASR audio corpora based on a multimodal large model. Background Art
[0002] In the field of speech recognition technology, automatic speech recognition (ASR) systems have made significant progress in recent years. With the continuous development of artificial intelligence technology, ASR systems have been widely used in many fields such as intelligent customer service, voice assistants, and smart homes, greatly improving the efficiency of people's information acquisition and interaction, and promoting the transformation of the human-computer interaction method towards a more natural and convenient direction.
[0003] In the prior art, in order to train a high-quality ASR system, a large number of diverse audio corpora are required. There are mainly two traditional ways to generate audio corpora. One is to collect speech data in a real environment, which requires a large amount of manpower and material resources to record, organize, and annotate the speech data. For example, organizing a large-scale recording team to collect speech in different scenarios. The other is to synthesize speech through simple text-to-speech technology, which mainly converts the input text into speech output based on rules or machine learning models, but the synthesized speech often lacks noise and diversity in real scenarios.
[0004] However, these existing methods for generating and annotating audio corpora have obvious defects. The collection cost of real environment speech data is high and it is difficult to cover all scenarios and speaker characteristics, resulting in insufficient diversity of the corpora. The speech synthesized by simple text-to-speech technology cannot simulate complex real scenarios and lacks interference factors such as noise, making the trained ASR system less adaptable to complex environments in practical applications. Summary of the Invention
[0005] This application provides a method and device for generating ASR audio corpora based on a multimodal large model, which can generate high-quality ASR audio corpora that meet specific requirements and have been effectively screened, improving the matching degree and usability of the corpora with actual application scenarios.
[0006] In the first aspect of this application, a method for generating ASR audio corpora based on a multimodal large model is provided, which is applied to a corpus generation platform. The method includes: Using a multimodal large model to encode the target domain text into a semantic vector, encoding the speech control parameters and speaker characteristics into a conditional vector, concatenating the semantic vector and the conditional vector into a joint vector, and decoding the joint vector into a first speech; Select the target noise from the preset noise library according to the scene label, superimpose the target noise on the first speech to generate a noisy speech, and inject adversarial noise into the noisy speech to generate a second speech; Perform noise annotation, text annotation, sentiment annotation, and speaker annotation on the second speech, and align the annotation data to the same time axis to generate a multi-modal annotation file; Set the word error rate threshold and semantic similarity threshold according to the scene label, noise type, and speaker information in the multi-modal annotation file, and screen the target corpus from the multi-modal annotation file according to the word error rate threshold and the semantic similarity threshold.
[0007] Optionally, the encoding the speech control parameters and speaker features into a conditional vector, concatenating the semantic vector and the conditional vector into a joint vector, and decoding the joint vector into the first speech includes: Map the speech rate, pause interval, tone, and emotional intensity to a first vector, map the speaker identifier, dialect, timbre, and age to a second vector, and concatenate the first vector and the second vector to generate a conditional vector; Allocate a first weight of the semantic vector and a second weight of the conditional vector through an attention mechanism, and concatenate the semantic vector and the conditional vector along the channel dimension according to the first weight and the second weight to form a joint vector; Decode the joint vector frame by frame into a Mel spectrogram, and inverse transform the Mel spectrogram into a time-domain waveform to obtain the first speech.
[0008] Optionally, the selecting the target noise from the preset noise library according to the scene label and superimposing the target noise on the first speech to generate a noisy speech includes: Extract the semantic features of the scene label, and match the target noise from the preset noise library according to the semantic features; Calculate the energy weights of the first speech and the target noise according to the signal-to-noise ratio, align the Mel spectrograms of the first speech and the target noise on the time axis, and superimpose them frame by frame according to the energy weights to obtain a first intermediate spectrogram; Convert the first intermediate spectrogram into a time-domain signal through inverse Mel transformation to obtain a noisy speech.
[0009] Optionally, the injecting adversarial noise into the noisy speech to generate a second speech includes: Generate an adversarial perturbation based on the gradient or decision boundary of the target ASR model, and convert the adversarial perturbation into a time-domain waveform or spectral feature to obtain adversarial noise; Locate the injection time window of the adversarial noise according to the speech content, and superimpose the adversarial noise and the noisy speech in the Mel spectrogram domain to obtain a second intermediate spectrogram; Perform an inverse Mel transform on the second intermediate spectrum to generate the second speech with adversarial noise.
[0010] Optionally, the noise annotation, text annotation, emotion annotation, and speaker annotation are performed on the second speech, and the annotation data is aligned to the same time axis to generate a multi-modal annotation file, including: Locate the target frame of the adversarial noise in the second speech according to the injection control parameter of the adversarial noise, and obtain noise annotation data based on the target frame; Generate an initial transcription text according to the speech content of the second speech, locate the first target speech segment masked by ambient noise or adversarial noise according to the noise annotation data, correct the initial transcription text of the first target speech segment through the multi-modal large model in combination with the context semantics, and use the dynamic time warping algorithm to align the timestamp of the corrected initial transcription text with the second speech to obtain text annotation data; Generate an initial emotion result according to the acoustic features of the second speech, locate the second target speech segment not masked by ambient noise or adversarial noise according to the noise annotation data, correct the initial emotion result of the second target recording segment according to the corrected initial transcription text, and use the dynamic time warping algorithm to align the timestamp of the corrected initial emotion result with the second speech to obtain emotion annotation data; Obtain the speaker embedding vector of the second speech, calculate the similarity between the speaker embedding vector and a preset speaker library to match the speaker identifier, and obtain speaker annotation data; Generate a multi-modal annotation file according to the noise annotation data, the text annotation data, the emotion annotation data, and the speaker annotation data.
[0011] Optionally, setting the word error rate threshold and the semantic similarity threshold according to the scene label, noise type, and speaker information of the multi-modal annotation file includes: Match the word error rate baseline threshold and the semantic similarity baseline threshold from a preset mapping table according to the scene label; Parse the noise type in the multi-modal annotation file, map the signal-to-noise ratio value in the multi-modal annotation file to a noise intensity level, and perform a first adjustment on the matched word error rate baseline threshold and the semantic similarity baseline threshold according to the noise type and the noise intensity level; Query the speaker attributes from a preset speaker library according to the speaker identifier, and perform a second adjustment on the matched word error rate baseline threshold and the semantic similarity baseline threshold that have undergone the first adjustment to obtain the word error rate threshold and the semantic similarity threshold.
[0012] Optionally, screening the target corpus from the multimodal annotation file according to the word error rate threshold and the semantic similarity threshold includes: Perform time alignment on the transcribed text of each piece of speech in the multimodal annotation file and a preset reference text, count the number of errors, and calculate the word error rate according to the number of errors. The number of errors includes substitution errors, insertion errors, and deletion errors; Perform semantic encoding on the transcribed text of each piece of speech in the multimodal annotation file and a preset reference text, and calculate the cosine similarity; Determine the target corpus from the speeches whose cosine similarity is greater than or equal to the semantic similarity threshold and whose word error rate is less than or equal to the word error rate threshold.
[0013] In the second aspect of the present application, a generation system for ASR audio corpus based on a multimodal large model is provided, including a vector module, a noise module, an annotation module, and an execution module, where: The vector module is configured to encode the target domain text into a semantic vector using a multimodal large model, encode the speech control parameters and speaker features into conditional vectors, splice the semantic vector and the conditional vector into a joint vector, and decode the joint vector into the first speech; The noise module is configured to select target noise from a preset noise library according to the scene label, superimpose the target noise on the first speech to generate a noisy speech, and inject adversarial noise into the noisy speech to generate a second speech; The annotation module is configured to perform noise annotation, text annotation, emotion annotation, and speaker annotation on the second speech, and align the annotation data to the same time axis to generate a multimodal annotation file; The execution module is configured to set a word error rate threshold and a semantic similarity threshold according to the scene label, noise type, and speaker information in the multimodal annotation file, and screen the target corpus from the multimodal annotation file according to the word error rate threshold and the semantic similarity threshold.
[0014] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface, and a network interface. The memory is used to store instructions. The user interface and the network interface are both used to communicate with other devices. The processor is used to execute the instructions stored in the memory so that the electronic device executes the method described in any one of the above.
[0015] In the fourth aspect of the present application, a computer-readable storage medium is provided. The computer-readable storage medium stores instructions, and when the instructions are executed, the method described in any one of the above is executed.
[0016] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. Encode the text in the target domain into semantic vectors through a multi-modal large model, and jointly generate speech with voice control parameters and speaker features to achieve precise alignment of semantics and acoustic features. By splicing semantic vectors and conditional vectors, multi-dimensional control of speech is supported, enhancing the diversity of the synthesized speech; 2. Generate high-complexity noisy speech by superimposing preset scene noise and adversarial noise (such as random perturbation signals) to simulate interference in real scenarios. Injecting adversarial noise can force the model to learn noise-insensitive features, enhance the robustness to minor perturbations, and reduce the misrecognition rate in actual deployment; 3. Annotate the synthesized speech with noise type, transcribed text, emotion labels, speaker ID, etc. to generate structured metadata, supporting multi-task learning. Align the annotated data precisely with the speech timeline to ensure strict matching of the transcribed text and speech segments, improving the credibility of the data; 4. Dynamically set the word error rate threshold and semantic similarity threshold based on scene labels, noise type, and speaker information, and screen high-quality corpus from multi-modal annotation files. Eliminate low-quality corpus to ensure the accuracy of training data. Prioritize retaining corpus strongly related to the target scene to improve the performance of the model in specific domains. Reduce redundant data and lower the model training cost. Description of the Drawings
[0017] Figure 1 is a schematic flowchart of a method for generating ASR audio corpus based on a multi-modal large model disclosed in an embodiment of the present application; Figure 2 is a schematic block diagram of a system for generating ASR audio corpus based on a multi-modal large model disclosed in an embodiment of the present application; Figure 3 is a schematic structural diagram of an electronic device disclosed in an embodiment of the present application.
[0018] Description of the reference numerals: 201, vector module; 202, noise module; 203, annotation module; 204, execution module; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. Detailed Embodiments
[0019] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments.
[0020] In the description of the embodiments of the present application, words such as "for example" or "for illustration" are used to give examples, illustrations or explanations. Any embodiment or design solution described as "for example" or "for illustration" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Rather, the use of words such as "for example" or "for illustration" is intended to present relevant concepts in a specific manner.
[0021] In the description of the embodiments of the present application, the term "a plurality of" means two or more. For example, a plurality of systems means two or more systems, and a plurality of screen terminals means two or more screen terminals. In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. The terms "include", "comprise", "have" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0022] This embodiment discloses a method for generating ASR audio corpus based on a multimodal large model, which is applied to a corpus generation platform. Figure 1 It is a schematic flowchart of a method for generating an ASR audio corpus based on a multimodal large model disclosed in the embodiments of the present application, as Figure 1 shown. The method includes the following steps: S101: Use a multimodal large model to encode the target domain text into a semantic vector, encode the speech control parameters and speaker features into conditional vectors, splice the semantic vector and the conditional vector into a joint vector, and decode the joint vector into a first speech; S102: Select target noise from a preset noise library according to the scene label, superimpose the target noise on the first speech to generate a noisy speech, and inject adversarial noise into the noisy speech to generate a second speech; S103: Perform noise annotation, text annotation, emotion annotation and speaker annotation on the second speech, and align the annotation data to the same time axis to generate a multimodal annotation file; S104: Set a word error rate threshold and a semantic similarity threshold according to the scene label, noise type and speaker information in the multimodal annotation file, and screen the target corpus from the multimodal annotation file according to the word error rate threshold and the semantic similarity threshold.
[0023] The text in the target domain is encoded using a multimodal large model to convert text information into semantic vectors. This process captures the semantic meaning of the text so that the subsequently generated speech can accurately reflect the text content. The speech control parameters (such as speech rate, intonation, etc.) and speaker characteristics (such as timbre, accent, etc.) are encoded as conditional vectors. These conditional vectors provide additional control information for speech generation, making the generated speech more in line with specific requirements in terms of style and features. The semantic vector and the conditional vector are concatenated into a joint vector, which combines the conditional information of text semantics and speech generation. Then, the multimodal large model decodes the joint vector to generate the first speech. According to the scene label, the target noise matching the specific scene is selected from the preset noise library. For example, if it is a simulated outdoor environment, wind noise, traffic noise, etc. can be selected. The selected target noise is superimposed on the first speech to generate a noisy speech. This step simulates the noise interference in the real environment, making the generated speech closer to the actual application scenario. On the basis of the noisy speech, adversarial noise is further injected. The purpose of adversarial noise is to increase the difficulty of speech recognition and improve the robustness of the ASR model so that it can still accurately recognize speech in the face of noise interference. The second speech is annotated with noise (marking the noise type and intensity), text (recording the text content corresponding to the speech), emotion (recognizing the emotional tendency in the speech) and speaker (determining the speaker's identity). These annotations provide rich information for subsequent corpus screening and application. The above annotation data are aligned to the same time axis to ensure that each annotation information accurately corresponds to a specific time period of the speech, and a multimodal annotation file is generated. This alignment operation facilitates the subsequent comprehensive analysis and processing of the speech and annotation information. According to the scene label, noise type and speaker information in the multimodal annotation file, the word error rate threshold and semantic similarity threshold are set. The word error rate threshold is used to measure the accuracy of the ASR system in speech recognition, and the semantic similarity threshold is used to evaluate the semantic similarity between the recognition result and the original text. According to the set word error rate threshold and semantic similarity threshold, the target corpus that meets the requirements is screened out from the multimodal annotation file. These target corpora can be used to train and evaluate the ASR model to improve the performance of the model in complex scenarios.
[0024] Optionally, encoding the speech control parameter and the speaker feature into a conditional vector, concatenating the semantic vector and the conditional vector into a joint vector, and decoding the joint vector into the first speech comprises: Mapping speech rate, pause interval, intonation and emotional intensity to a first vector, mapping speaker identification, dialect, timbre and age to a second vector, and concatenating the first vector and the second vector to generate a conditional vector; Allocate a first weight for the semantic vector and a second weight for the conditional vector through an attention mechanism, and concatenate the semantic vector and the conditional vector along the channel dimension according to the first weight and the second weight to form a joint vector; Decode the joint vector frame by frame into a Mel spectrogram, and inverse-transform the Mel spectrogram into a time-domain waveform to obtain a first speech.
[0025] Speech rate: Usually, it can be measured by the number of syllables or words uttered per unit time. For example, it is stipulated that 120 words per minute is the medium speech rate, which is mapped to a vector element value; while 80 words per minute is a slower speech rate, mapped to a different value. Pause interval: According to the punctuation marks (such as commas, periods, etc.) in the text and natural semantic pauses, the duration and position of the pauses are counted. For example, the pause after a period may be set to 1 second, mapped to a relatively large vector element value; the pause after a comma is 0.5 seconds, mapped to a relatively small value. Intonation: Intonation reflects the ups and downs of speech and can be quantified by analyzing the fundamental frequency of the speech signal. For example, an increase in the fundamental frequency is represented as a positive number, a decrease as a negative number, and different magnitudes of increase or decrease correspond to different values, which are then mapped into the first vector. Emotional intensity: Based on a pre-set emotional classification model (such as classifying emotions into happy, sad, angry, etc. categories), according to the text content and the expected emotional expression intensity, a numerical value is assigned to each emotional category. For example, a strong happy emotion is mapped to a higher numerical value, and a weak happy emotion is mapped to a lower value. After the above mappings, the numerical values corresponding to all parameters are combined into a vector, namely the first vector. This first vector contains the encoded information of the speech control parameters, providing a control basis for subsequent speech generation in terms of speech rate, pauses, intonation, etc. Speaker identification: A unique identifier is assigned to each speaker and directly mapped to a vector element. For example, speaker A is mapped to 1, speaker B is mapped to 2, etc. Dialect: According to different dialect types (such as Cantonese, Sichuan-Chongqing dialect, Wu dialect, etc.), a dialect coding table is established, and each dialect is mapped to a specific numerical value or vector combination. For example, Cantonese is mapped to the vector [0, 1, 0], and the Sichuan-Chongqing dialect is mapped to [1, 0, 0], etc. Timbre: Timbre is the unique characteristic of a speaker's voice. By analyzing and extracting the spectral features of the speech signal (such as formant frequencies, spectral envelopes, etc.), these features are quantified and mapped into the second vector. For example, prominent high-frequency formants may correspond to relatively high vector element values, and prominent low-frequency formants correspond to lower values. Age: Age is divided into different intervals (such as 0 - 10 years old, 11 - 20 years old, etc.), and each interval corresponds to a numerical value. For example, 0 - 10 years old is mapped to 1, 11 - 20 years old is mapped to 2, etc. The numerical values after mapping the characteristics of each speaker are combined into the second vector, which carries the encoded information of the speaker characteristics and is used to control the speaker style and characteristics of the generated speech. The first vector and the second vector are concatenated to generate a conditional vector. The concatenation method is usually a simple vector connection operation. For example, if the first vector is [a1, a2,......, an] and the second vector is [b1, b2,......, bm], then the concatenated conditional vector is [a1, a2,......, an, b1, b2,......, bm].This conditional vector synthesizes the information of the voice control parameters and the speaker characteristics, providing a basis for the subsequent generation of the joint vector.
[0026] The attention mechanism is used to assign weights to the semantic vector and the conditional vector, and they are concatenated along the channel dimension to form a joint vector. This process realizes the organic integration of semantic information and control information. The attention mechanism is a technique for automatically learning the importance of different information. During the process of generating the joint vector, the attention mechanism analyzes the semantic vector and the conditional vector respectively. According to their relevance to the final voice generation task, it assigns the first weight to the semantic vector and the second weight to the conditional vector. For the semantic vector and the conditional vector, the query (Query), key (Key), and value (Value) vectors can be generated through linear transformations respectively. For example, the semantic vector S undergoes linear transformations W1, W2, W3 to obtain the query vector Qs = S×W1, the key vector Ks = S×W2, and the value vector Vs = S×W3; the conditional vector C undergoes linear transformations to obtain Qa = C×W1, Ka = C×W2, and Va = C×W3. Calculate the similarity between the query vector of the semantic vector and the key vector of the conditional vector, and the similarity between the query vector of the conditional vector and the key vector of the semantic vector. The similarity can be calculated by methods such as dot product, cosine similarity, etc. For example, calculate the dot product of Qs and Ka to obtain the similarity score S1 = Qs·Ka, and calculate the dot product of Qa and Ks to obtain the similarity score S2 = Qa·Ks. Use the softmax function to normalize the similarity scores to obtain the first weight and the second weight. For example, the first weight α = softmax(S1), and the second weight β = softmax(S2). According to the assigned first weight α and second weight β, the semantic vector and the conditional vector are concatenated along the channel dimension. When concatenating, the semantic vector and the conditional vector can be combined in a certain proportion. For example, the joint vector U = α×S + β×C (here the addition operation is a weighted concatenation along the channel dimension), or other concatenation methods can also be used, such as directly connecting the semantic vector and the conditional vector along the channel dimension and then performing a linear transformation according to the weights. Through this concatenation method, the joint vector integrates semantic information and control information, providing a more comprehensive input for the subsequent voice decoding.
[0027] The joint vector is decoded frame by frame into a Mel spectrogram, and then the Mel spectrogram is inversely transformed into a time-domain waveform to finally obtain the first speech. This process realizes the conversion from vector representation to audible speech. A neural network decoder (such as a decoder based on recurrent neural network (RNN), long short-term memory network (LSTM), or Transformer) is used to decode the joint vector frame by frame. The decoder receives the joint vector as input and gradually generates each frame of the Mel spectrogram. During the decoding process, the decoder predicts the Mel spectrogram features of the current frame based on the information in the joint vector and in combination with the previously generated frames. The Mel spectrogram is a representation method that converts a speech signal from the time domain to the frequency domain, and it simulates the human ear's perception characteristics of sound frequencies. Each frame of the Mel spectrogram contains the frequency distribution information of the speech signal within a specific time period. The generated Mel spectrogram is converted back to a time-domain waveform through operations such as an inverse Mel filter bank and inverse short-time Fourier transform (ISTFT). The inverse Mel filter bank converts the Mel spectrogram back to a linear spectrogram, and ISTFT converts the linear spectrogram back to a time-domain signal. After the above inverse transformation, the speech signal in the time domain, that is, the first speech, is obtained. This first speech combines the text semantic information carried by the semantic vector and the speech control parameters and speaker characteristic information contained in the conditional vector, and has characteristics such as a specific speech rate, intonation, emotional intensity, and speaker style.
[0028] By mapping speech control parameters such as speech rate, pause interval, intonation, and emotional intensity into a first vector, and speaker characteristics such as speaker identification, dialect, timbre, and age into a second vector, a refined encoding of the core elements in the speech generation process is achieved, providing rich and controllable feature inputs for subsequent generation of personalized and scenario-based speech. Concatenating the above two types of vectors to generate a conditional vector can flexibly combine different speech control parameters and speaker characteristics, quickly adapt to diverse generation requirements such as different speaker identities, emotional states, and speech styles, and enhance the flexibility and generality of the speech generation system. Using the attention mechanism to dynamically assign weights to the semantic vector and the conditional vector can intelligently judge the relative importance of semantic information and conditional information in the speech generation process according to specific generation tasks and context information, enabling the model to have an adaptive ability when processing different types of texts and scenarios. Based on the weight assignment results, concatenating the semantic vector and the conditional vector along the channel dimension to form a joint vector realizes the deep fusion and joint modeling of semantic information and conditional information at the model level, which helps to generate speech with accurate semantics, distinct features, and meeting expectations. Decoding the joint vector frame by frame into a Mel spectrogram, using this intermediate representation form of the Mel spectrogram effectively retains the spectral features and detailed information of the speech, laying a solid foundation for subsequent waveform generation and ensuring that the generated speech has high quality at the spectral level. Restoring the Mel spectrogram to a time-domain waveform through an inverse transformation, and finally obtaining the first speech. This process can accurately restore the time-domain characteristics of the speech, making the generated speech natural and smooth in auditory perception without obvious distortion, meeting the high requirements for speech quality in practical applications.
[0029] Optionally, the step of selecting a target noise from a preset noise library according to the scene label and superimposing the target noise on the first speech to generate a noisy speech includes: Extracting the semantic features of the scene label and matching a target noise from the preset noise library according to the semantic features; Calculating the energy weights of the first speech and the target noise according to the signal-to-noise ratio, aligning the Mel spectrograms of the first speech and the target noise on the time axis, and superimposing them frame by frame according to the energy weights to obtain a first intermediate spectrogram; Converting the first intermediate spectrogram into a time-domain signal through an inverse Mel transformation to obtain a noisy speech.
[0030] Semantic feature extraction is performed on the scene tags. In this step, with the help of natural language processing techniques, the scene tags (such as "outdoor street", "factory workshop", "noisy restaurant environment", etc.) are converted into semantic vectors that can be understood by a computer. These vectors contain the core information of the scene, such as the environmental type, the tendency of the noise source, etc., providing a semantic basis for subsequent noise matching. According to the extracted semantic features, the target noise is matched from a preset noise library. The preset noise library stores a large number of noise samples of different types and intensities, and each sample is accompanied by a semantic label. By calculating the similarity between the semantic features of the scene tags and the semantic labels of the noise samples, the noise with the highest similarity is selected as the target noise to achieve precise adaptation of the noise to the scene, ensuring that the superimposed noise conforms to the characteristics of the actual scene. The signal-to-noise ratio (SNR) is an index that measures the energy relationship between the speech signal and the noise signal. According to the set target SNR, the energy weights of the first speech and the target noise are calculated. The energy weights reflect the energy proportions of the speech and the noise in the mixed signal. By adjusting the energy weights, the relative intensities of the speech and the noise in the generated noisy speech can be controlled to meet the requirements for the degree of noise interference in different scenarios, such as simulating a slightly noisy background or a strong interference noise environment. The Mel spectrograms of the first speech and the target noise are aligned on the time axis. The Mel spectrogram is a representation method that converts the speech signal into the frequency domain, and it is more in line with the auditory perception characteristics of the human ear. The alignment operation ensures that the speech and the noise are synchronized in the time dimension, facilitating subsequent frame-by-frame superposition. According to the calculated energy weights, the Mel spectrograms of the first speech and the target noise are added frame by frame to obtain the first intermediate spectrogram. The frame-by-frame superposition method can precisely control the mixing ratio of the speech and the noise at each moment, ensuring the accuracy and continuity of the mixed spectrogram in both the time and frequency dimensions. The first intermediate spectrogram is converted into a time-domain signal through the inverse Mel transform. The inverse Mel transform is the inverse process of the Mel spectrogram transform, which restores the Mel spectrogram information in the frequency domain to the waveform signal in the time domain. During the conversion process, the energy distribution and frequency characteristics of the speech and the noise after spectrogram superposition are retained, ensuring that the restored time-domain signal can accurately reflect the sound characteristics of the mixed speech and noise. The time-domain signal obtained through the inverse Mel transform is the noisy speech. Since the energy weights are precisely controlled during the spectrogram superposition process, and accurate alignment and processing are performed in both the time and frequency domains, the generated noisy speech is natural and realistic in terms of audition, and the mixing effect of the noise and the speech meets the requirements of the actual scene, providing high-quality training or test data for subsequent speech recognition, speech enhancement, and other tasks.
[0031] The generated noisy speech can simulate the noise interference in various real scenarios, providing rich noisy training data for speech processing models (such as ASR models). This enables the model to encounter a more diverse noise environment during training, thereby improving the model's robustness and generalization ability to noise in practical applications. Through semantic matching of scene labels and flexible adjustment of signal-to-noise ratio, noisy speech with specific noise characteristics and interference levels can be customized according to different application scenarios and requirements, meeting the requirements for the diversity and authenticity of speech data in different industries (such as intelligent customer service, security monitoring, in-vehicle voice interaction, etc.). Based on a preset noise library and an automated processing flow, a large amount of noisy speech data can be generated quickly and efficiently, avoiding the high cost and low efficiency of manually collecting and annotating noisy speech data in real scenarios, and providing a convenient data generation means for speech technology research and application development.
[0032] Optionally, injecting adversarial noise into the noisy speech to generate the second speech includes: Generating an adversarial perturbation based on the gradient or decision boundary of the target ASR model, converting the adversarial perturbation into a time-domain waveform or spectral feature to obtain adversarial noise; Locating the injection time window of the adversarial noise according to the speech content, and superimposing the adversarial noise and the noisy speech in the Mel spectrogram domain to obtain a second intermediate spectrogram; Performing inverse Mel transformation on the second intermediate spectrogram to generate the second speech containing adversarial noise.
[0033] The core of generating adversarial noise is to utilize the characteristics of the target ASR (automatic speech recognition) model. When the ASR model is trained to a certain stage, there is a mechanism inside it to classify or convert the input speech features, which can abstract the decision boundary (used to distinguish different speech categories) or reflect the impact of input changes on the output through gradient information. Based on gradient: During model training, the gradient represents the rate of change of the loss function with respect to the model parameters. When generating adversarial perturbations, the gradient can be regarded as the direction in the input speech feature space that causes the largest change in the model output. By calculating the gradient of the target ASR model with respect to the input speech and making slight adjustments to the speech features along the gradient direction, adversarial perturbations that cause the model output to produce incorrect recognition can be generated. Based on decision boundaries: The decision boundary is the boundary where the model distinguishes different speech categories. By analyzing the distribution of speech features near the decision boundary, small perturbations that can cross the boundary can be found. When these perturbations are added to the original speech, the speech will "slip" from the originally correctly recognized category into the wrong category, thus forming adversarial perturbations. In specific implementation, iterative optimization algorithms such as the fast gradient sign method (FGSM) and the basic iterative method (BIM) can be used. Taking FGSM as an example, it calculates the gradient of the loss function with respect to the input speech, takes the sign of the gradient, multiplies it by a small step size factor, and adds the result to the original speech feature to obtain the adversarial perturbation. This perturbation is usually small and not easily perceived by humans, but it can have a significant impact on the ASR model. The generated adversarial perturbation may initially exist in the form of speech features (such as Mel frequency cepstral coefficients, etc.). In order to add it to the noisy speech, it needs to be converted into a time domain waveform. This process is achieved through inverse transformation, such as using the inverse process of the Mel filter bank, the inverse discrete cosine transform, etc., to map the perturbation from the feature space back to the time domain to obtain the adversarial noise waveform in the time domain. It is also possible to choose to convert the adversarial perturbation into spectral features, such as the Mel spectrum. Through a specific transformation matrix or algorithm, the perturbation is mapped to the Mel spectrum domain, making it an adversarial noise in the form of spectral features that can be directly superimposed on the speech Mel spectrum. It is crucial to determine the injection time window of the adversarial noise according to the speech content. This requires voice activity detection and semantic analysis of noisy speech. Voice activity detection can identify valid speech segments and silent segments in speech, while semantic analysis further determines the key semantic units (such as words, phrases) in speech and their time positions. Injecting adversarial noise near key semantic units or in time periods that have a greater impact on ASR model recognition can more effectively interfere with the model's recognition process. For example, injecting noise during the pronunciation period of important words in speech may cause the model to misrecognize the word, leading to recognition errors of the entire sentence. Align the generated adversarial noise (whether in the form of spectrum after time domain waveform conversion or directly generated spectrum features) with the Mel spectrum of the noisy speech on the time axis.Ensure that the two are synchronized in the time dimension so that the subsequent superposition operation can accurately add noise to the corresponding positions of the speech. In the Mel spectrogram domain, the adversarial noise is added to the Mel spectrogram of the noisy speech frame by frame according to certain rules (such as linear superposition) to obtain the second intermediate spectrogram. The frame-by-frame superposition method ensures the accuracy of the superposition process, making the mixing effect of the noise and the speech in the spectrogram meet the expectations and preventing abnormal situations caused by time or frequency misalignment. Perform inverse Mel transformation on the second intermediate spectrogram to convert it back from the Mel spectrogram domain to the time-domain signal. The process of inverse Mel transformation is similar to the conversion from the Mel spectrogram to the time-domain waveform when generating the noisy speech. Through operations such as the inverse operation of the Mel filter bank and the inverse discrete cosine transform, the spectral information is restored to the waveform signal in the time domain. The time-domain signal obtained after the inverse Mel transformation is the second speech containing adversarial noise. Since the injection position and intensity of the noise are precisely controlled during the spectral superposition process, the generated second speech may not have obvious abnormalities audibly, but during the recognition of the ASR model, incorrect recognition results will occur due to the presence of adversarial noise, achieving the purpose of testing and enhancing the robustness of the ASR model.
[0034] The generated speech containing adversarial noise can simulate malicious attacks that may occur in real scenarios, helping researchers and developers understand the weaknesses of the ASR model when facing adversarial interference, so as to improve and optimize the model targetedly and enhance the stability and reliability of the model in practical applications. During the training process of the ASR model, introducing speech data containing adversarial noise as training samples can enable the model to learn more robust feature representations and enhance its resistance to noise interference. At the same time, during the model evaluation stage, using such data for testing can more accurately evaluate the performance of the model in complex environments. The injection technology of adversarial noise provides an important means for the research in the field of speech security. Researchers can explore effective attack strategies by analyzing the impact of different types of adversarial noise on the ASR model and further study corresponding defense methods, such as developing adversarial noise detection algorithms and improving the model architecture to enhance robustness, promoting the development of speech security technology. By generating speech data containing adversarial noise, the training dataset of the ASR model can be expanded, increasing the diversity and complexity of the data. This helps the model learn more comprehensive speech features, reduce overfitting, and improve the generalization ability of the model on unseen data.
[0035] Optionally, the noise annotation, text annotation, emotion annotation, and speaker annotation are performed on the second speech, and the annotation data is aligned to the same time axis to generate a multi-modal annotation file, including: Locate the target frame of the adversarial noise in the second speech according to the injection control parameters of the adversarial noise, and obtain the noise annotation data based on the target frame; Generate an initial transcription text based on the speech content of the second speech, locate the first target speech segment masked by scene noise or adversarial noise according to the noise annotation data, correct the initial transcription text of the first target speech segment by combining the context semantics through the multimodal large model, and use the dynamic time warping algorithm to align the time stamps of the corrected initial transcription text with the second speech to obtain text annotation data; Generate an initial emotion result based on the acoustic features of the second speech, locate the second target speech segment not masked by scene noise or adversarial noise according to the noise annotation data, correct the initial emotion result of the second target recording segment according to the corrected initial transcription text, and use the dynamic time warping algorithm to align the time stamps of the corrected initial emotion result with the second speech to obtain emotion annotation data; Obtain the speaker embedding vector of the second speech, calculate the similarity between the speaker embedding vector and a preset speaker library to match the speaker identifier, and obtain speaker annotation data; Generate a multimodal annotation file according to the noise annotation data, the text annotation data, the emotion annotation data, and the speaker annotation data.
[0036] When generating the second voice, the injection of adversarial noise is based on specific control parameters, such as the injection time window, intensity, etc. Using these injection control parameters, the target frames of the adversarial noise in the second voice can be accurately located. For example, if it is specified in the control parameters that the adversarial noise starts to be injected at the 5th second of the voice and lasts for 2 seconds, then according to the sampling rate and frame length of the voice, the frame number range corresponding to these noises can be calculated, thereby determining the target frames. Based on the located target frames, noise annotation data is generated. The annotation data can include information such as the noise type (ambient noise, adversarial noise), the start and end frame numbers of the noise, and the noise intensity. This information provides a basis for subsequent analysis of the impact of noise on tasks such as speech recognition and emotion recognition. Using speech recognition technology, the speech content of the second voice is converted into an initial transcription text. This is usually achieved through an automatic speech recognition model, which converts the speech signal into a corresponding text sequence. However, due to the presence of ambient noise and adversarial noise in the second voice, the initial transcription text may contain errors. According to the noise annotation data, the first target speech segments masked by ambient noise or adversarial noise are located. In these speech segments, the speech signal is severely interfered by noise, resulting in difficult speech recognition, and the errors in the initial transcription text often concentrate here. A multimodal large model is used to correct the initial transcription text by combining the context semantic information of the first target speech segments. The multimodal large model can comprehensively consider multiple factors such as the semantics, grammar, and context of the speech. By analyzing the content of the surrounding normal speech segments, it can infer the possible content of the masked speech segments, thereby optimizing the initial transcription text. The dynamic time warping (DTW) algorithm is used to align the timestamps of the corrected initial transcription text with the second voice. The DTW algorithm can handle the stretching and shrinking changes of the speech signal on the time axis, find the best alignment path between the text and the voice, and ensure that each character in the text annotation data can accurately correspond to the corresponding position in the voice, obtaining accurate text annotation data. According to the acoustic features of the second voice, such as pitch, speech rate, energy, etc., an initial emotion result is generated using an emotion recognition model. These acoustic features are closely related to human emotional states. For example, high pitch and fast speech rate may indicate excited or nervous emotions. The emotion recognition model establishes a mapping relationship between the acoustic features and emotion categories by learning a large amount of speech data with emotion labels, thereby classifying the emotions of new speech. According to the noise annotation data, the second target speech segments not masked by ambient noise or adversarial noise are located. In these speech segments, the speech signal is relatively clear, and the emotion recognition model can more accurately extract acoustic features and judge the emotional state. According to the corrected initial transcription text, the initial emotion result of the second target speech segments is further corrected. The text content can provide additional semantic information for emotion judgment. For example, certain words or sentences themselves have obvious emotional tendencies. By combining the text information, the emotion expressed by the speech can be more comprehensively understood, improving the accuracy of emotion annotation.Similarly, the dynamic time warping algorithm is used to align the timestamps of the corrected initial sentiment results with the second speech. Ensure that the sentiment annotation data can accurately reflect the sentiment changes of the speech at different time periods, and generate sentiment annotation data consistent with the speech timeline. Through speaker recognition technology, obtain the speaker embedding vector of the second speech. The speaker embedding vector is an abstract representation of the speaker's acoustic features, which contains unique information such as the speaker's voice quality and pronunciation habits. Usually, a deep learning model, such as a speaker verification model, is used to extract these embedding vectors from the speech signal. Calculate the similarity between the speaker embedding vector and the preset speaker library. The speaker library stores the embedding vectors of multiple known speakers. By calculating the similarity between the two (such as cosine similarity), find the speaker identifier most similar to the speaker embedding vector of the second speech, so as to determine the speaker of the second speech. Use the matched speaker identifier as the speaker annotation data to annotate the speaker identity information for the second speech, which is convenient for subsequent analysis and processing of the speech of different speakers. According to the noise annotation data, text annotation data, sentiment annotation data, and speaker annotation data, integrate these information of different dimensions. Each annotation data contains timestamp information to ensure that they can be aligned and associated on the same timeline. Construct a multi-modal annotation file according to a specific file format. This file can record information such as the noise type, text content, sentiment state, and speaker identity at each time point in the speech, providing comprehensive and detailed annotation information for speech processing tasks. The multi-modal annotation file can adopt a structured data format, such as JSON, XML, etc., which is convenient for subsequent reading and processing.
[0037] The multi-modal annotation file provides rich annotation information for tasks such as speech recognition, sentiment recognition, and speaker recognition. Information of different modalities can complement and verify each other, thereby improving the accuracy and robustness of these tasks. For example, in speech recognition, combining sentiment annotation information can better understand the semantics and context of the speech and reduce errors caused by noise interference; in sentiment recognition, text annotation information can provide semantic support for sentiment judgment and improve the accuracy of sentiment recognition. It provides detailed data support for academic research in the field of speech. Researchers can use the multi-modal annotation file to analyze the recognition performance of speech in different noise environments, the characteristics of emotional expression, and the variation laws of speaker features, promoting the development and innovation of speech technology. In the actual development of speech applications, such as intelligent customer service, voice assistants, and voice security, the multi-modal annotation file can help developers better understand and process users' speech inputs, and achieve more intelligent and personalized voice interaction functions. For example, in intelligent customer service, the response strategy of the customer service can be adjusted in a timely manner according to the sentiment annotation information to improve user satisfaction.
[0038] Optionally, setting the word error rate threshold and the semantic similarity threshold according to the scene label, noise type, and speaker information in the multimodal annotation file includes: According to the scene label, match the word error rate baseline threshold and the semantic similarity baseline threshold from a preset mapping table; Parse the noise type in the multimodal annotation file, map the signal-to-noise ratio value in the multimodal annotation file to a noise intensity level, and perform a first adjustment on the matched word error rate baseline threshold and the semantic similarity baseline threshold according to the noise type and the noise intensity level; Query the speaker attributes from a preset speaker library according to the speaker identifier, and perform a second adjustment on the matched word error rate baseline threshold and the semantic similarity baseline threshold after the first adjustment according to the speaker attributes to obtain the word error rate threshold and the semantic similarity threshold.
[0039] Scene labels are used to describe the general application scenarios where the speech is located, such as "quiet indoor", "noisy street", "meeting room", "inside a car", etc. Under different scenarios, the interference factors and noise levels that the speech is subjected to vary significantly, which in turn affect the performance of the speech recognition system, such as the word error rate (WER) and the accuracy of semantic understanding (measured by semantic similarity). The preset mapping table stores the baseline thresholds of the word error rate and the baseline thresholds of semantic similarity corresponding to different scene labels. These baseline thresholds reflect the minimum standards that a well-performing speech recognition and semantic understanding system should achieve in the corresponding scenarios. For example, in the quiet indoor scenario, due to less noise interference and high speech clarity, the baseline threshold of the word error rate can be set relatively low (such as 5%), and the baseline threshold of semantic similarity can be set relatively high (such as 0.9); while in the noisy street scenario, due to high background noise and increased difficulty in speech recognition, the baseline threshold of the word error rate will be correspondingly increased (such as 15%), and the baseline threshold of semantic similarity will be decreased (such as 0.7). According to the scene label in the multimodal annotation file, quickly match the corresponding baseline thresholds of the word error rate and the baseline thresholds of semantic similarity from the preset mapping table, providing a basis for subsequent threshold adjustment. The signal-to-noise ratio is the ratio of the power of the speech signal to the power of the noise signal, usually expressed in decibels. It reflects the relative intensity of the effective speech component and the noise component in the speech signal. The higher the signal-to-noise ratio, the better the speech quality and the less affected by noise. Map different signal-to-noise ratio ranges to predefined noise intensity levels. For example, a signal-to-noise ratio greater than 20 dB corresponds to the "low noise level", a signal-to-noise ratio between 10 dB - 20 dB corresponds to the "medium noise level", and a signal-to-noise ratio less than 10 dB corresponds to the "high noise level". This classification method can more intuitively describe the degree of noise interference that the speech is subjected to. Different types of noise have different mechanisms of influencing speech recognition and semantic understanding. For example, white noise is a kind of random noise, evenly distributed across all frequencies, and will uniformly interfere with all frequency bands of the speech signal; while burst noise (such as the sound of closing a door, a car horn) will cause strong interference to the speech at a specific time point, resulting in local distortion of the speech signal. According to the noise type and the noise intensity level, make a first adjustment to the matched baseline thresholds of the word error rate and the baseline thresholds of semantic similarity. Generally speaking, when the noise intensity level is higher or the noise type interferes more severely with the speech, the baseline threshold of the word error rate will be correspondingly increased, and the baseline threshold of semantic similarity will be correspondingly decreased. For example, for the white noise scenario with a high noise level, on the basis of the original baseline threshold, the baseline threshold of the word error rate may be increased by 5%, and the baseline threshold of semantic similarity may be decreased by 0.1; while for the medium noise level scenario with burst noise, the baseline threshold of the word error rate may be increased by 3%, and the baseline threshold of semantic similarity may be decreased by 0.05. This adjustment method can more accurately reflect the difficulty of speech recognition and semantic understanding under different noise conditions. The preset speaker database stores relevant information of multiple known speakers, including attributes such as the age, gender, accent, and speaking speed of the speakers.These attributes can affect the characteristics of the speech signal, thereby influencing the performance of speech recognition and semantic understanding. For example, people of different ages have different pronunciation characteristics. The speech of children and the elderly may have problems such as unclear pronunciation, slower or faster speaking speeds. People from different regions may have different accents, resulting in difficulties for the speech recognition system to recognize certain pronunciations. According to the speaker identification in the multimodal annotation file, query the corresponding speaker attribute information from the preset speaker library. Different attributes of the speaker can affect the accuracy of speech recognition and semantic understanding. Therefore, it is necessary to perform a second adjustment on the word error rate baseline threshold and the semantic similarity baseline threshold after the first adjustment according to the speaker attributes. For example, for a speaker with a strong accent, the difficulty of the speech recognition system in recognizing their speech will increase, so the word error rate baseline threshold needs to be further increased. For a speaker with a faster speaking speed, the system may not have enough time to accurately recognize each word when processing their speech, which will also lead to an increase in the word error rate. At the same time, the difficulty of semantic understanding may also increase, and the semantic similarity baseline threshold needs to be appropriately reduced. The specific adjustment range can be set according to a large amount of experimental data and experience. For example, for a speaker with an obvious accent, the word error rate baseline threshold is increased by 2% on the basis of the first adjustment, and the semantic similarity baseline threshold is reduced by 0.03. After the baseline threshold matching based on the scene label, the first adjustment combining the noise type and intensity level, and the second adjustment according to the speaker attributes, the word error rate threshold and the semantic similarity threshold are finally obtained. These two thresholds comprehensively consider the scene where the speech is located, the noise situation, and the characteristics of the speaker themselves, and can more accurately evaluate the performance of the speech recognition and semantic understanding systems. In practical applications, the word error rate output by the speech recognition system can be compared with the set word error rate threshold, and the semantic similarity output by the semantic understanding system can be compared with the set semantic similarity threshold to determine whether the performance of the system meets the standard under the current speech input. If the word error rate is lower than the threshold and the semantic similarity is higher than the threshold, it indicates that the system performance is good; otherwise, the system needs to be optimized and improved. Setting different thresholds according to different speaker attributes can provide personalized speech services for different users. For example, for users with a heavier accent, appropriately relaxing the word error rate threshold and reducing the semantic similarity threshold at the same time can improve the system's ability to recognize and understand the speech of this user and enhance the user experience.
[0040] Optionally, the screening of the target corpus from the multimodal annotation file according to the word error rate threshold and the semantic similarity threshold includes: Align the transcription text of each speech in the multimodal annotation file with the preset reference text in time, count the number of errors, and calculate the word error rate according to the number of errors. The number of errors includes the number of substitution errors, insertion errors, and deletion errors; Semantically encode the transcribed text of each piece of speech in the multimodal annotation file and a preset reference text, and calculate the cosine similarity; Determine the target corpus from the speech whose cosine similarity is greater than or equal to the semantic similarity threshold and whose word error rate is less than or equal to the word error rate threshold.
[0041] The purpose of time-aligning the transcribed text of each speech in the multimodal annotation file with the preset reference text is to compare the contents of the two under the same semantic and sentence structure framework, so as to accurately evaluate the transcription accuracy of the speech content by the speech recognition system. For example, a speech may contain multiple sentences and words. Time alignment can ensure that each word and sentence in the transcribed text and the reference text corresponds to the corresponding time segment of the speech, avoiding inaccurate error statistics due to time misalignment. A dynamic time warping algorithm can be used to achieve time alignment. Specifically, for speech signals, acoustic features such as Mel-frequency cepstral coefficients can be extracted first, and the transcribed text and the reference text can be aligned according to the pronunciation time point of the speech. The feature sequence is aligned, thus laying the foundation for subsequent error statistics. The number of errors includes substitution errors, insertion errors, and deletion errors. A substitution error refers to a word in the transcribed text being mistakenly identified as another word, such as identifying "apple" as "pingguo"; an insertion error refers to an additional word in the transcribed text that is not in the reference text, such as the reference text is "I like to eat fruit", and the transcribed text is "I like to eat fruit and apples" (with "and apples"); a deletion error refers to the omission of a word in the reference text in the transcribed text, such as the reference text is "I went to the park for a walk today", and the transcribed text is "I went to the park today" (missing "walk"). On the basis of time alignment, the transcribed text and the reference text are compared word by word. When a word mismatch is found, the error type is counted. For example, three counters are set to record the number of substitution errors, insertion errors, and deletion errors respectively, and the entire text sequence is traversed. Each time an error type is encountered, the corresponding counter is increased by 1. The ratio of the sum of the number of substitution errors, insertion errors, and deletion errors to the total number of words in the reference text is used as the word error rate. Semantic encoding of the transcribed text of each speech in the multimodal annotation file and the preset reference text is to convert the text into a vector form that can represent its semantic information. Taking the BERT (Bidirectional Encoder Representations from Transformers) model as an example, the transcribed text and the reference text are respectively input into the BERT model to obtain their corresponding semantic vectors. Before inputting into the model, the text needs to be preprocessed by word segmentation and adding special tags. The semantic vector output by the model can capture the semantic relationship and context information in the text, providing a basis for subsequent similarity calculation. Cosine similarity is an indicator to measure the similarity between two vectors, and its value range is between [-1, 1]. For semantic vectors, the larger the cosine similarity, the closer the semantics of the two texts are. The cosine similarity between two vectors is calculated by calculating the cosine value of their angle. The word error rate threshold and the semantic similarity threshold are pre-set according to the actual application requirements and system performance requirements.The word error rate threshold is used to measure the accuracy of the speech recognition system in transcribing speech content, and the semantic similarity threshold is used to evaluate the semantic consistency between the transcribed text and the reference text. For example, in a voice interaction system with high requirements for speech recognition accuracy, the word error rate threshold may be set to 10%, and the semantic similarity threshold may be set to 0.85. When screening target corpus from the multimodal annotation file, two conditions need to be met simultaneously: the cosine similarity is greater than or equal to the semantic similarity threshold, and the word error rate is less than or equal to the word error rate threshold. Only the speech that meets these two conditions simultaneously is considered high-quality corpus that meets the requirements. The screened target corpus can be used to further optimize the speech recognition and semantic understanding models. Since these corpora have high accuracy and semantic consistency, adding them to the training dataset can improve the model's adaptability to different speech scenarios, speaker characteristics, and noise interference, thereby enhancing the model's performance. In different business scenarios, the target corpus can be used as a basis for evaluating and selecting speech processing systems. For example, in an intelligent customer service system, by analyzing the target corpus, the system's ability to understand and respond to user speech commands can be evaluated, providing a reference for system optimization and upgrade. At the same time, the target corpus can also be used to construct a speech dataset for a specific domain to meet the speech processing needs of different industries.
[0042] This embodiment also discloses a generation system for ASR audio corpus based on a multimodal large model. Figure 2 It is a module schematic diagram of a generation system for ASR audio corpus based on a multimodal large model disclosed in an embodiment of the present application, as Figure 2 shown. The system includes a vector module 201, a noise module 202, an annotation module 203, and an execution module 204, where: The vector module 201 is configured to encode the target domain text into a semantic vector using a multimodal large model, encode the speech control parameters and speaker characteristics into conditional vectors, splice the semantic vector and the conditional vector into a joint vector, and decode the joint vector into the first speech. The noise module 202 is configured to select target noise from a preset noise library according to the scene label, superimpose the target noise on the first speech to generate noisy speech, and inject adversarial noise into the noisy speech to generate the second speech. The annotation module 203 is configured to perform noise annotation, text annotation, emotion annotation, and speaker annotation on the second speech, and align the annotation data to the same time axis to generate a multimodal annotation file. The execution module 204 is configured to set the word error rate threshold and the semantic similarity threshold according to the scene label, noise type, and speaker information in the multimodal annotation file, and screen the target corpus from the multimodal annotation file according to the word error rate threshold and the semantic similarity threshold.
[0043] Optionally, the vector module 201 is configured to: Map the speech rate, pause interval, intonation, and emotional intensity to a first vector, map the speaker identification, dialect, timbre, and age to a second vector, and concatenate the first vector and the second vector to generate a conditional vector; Allocate a first weight of the semantic vector and a second weight of the conditional vector through an attention mechanism, and concatenate the semantic vector and the conditional vector along the channel dimension according to the first weight and the second weight to form a joint vector; Decode the joint vector frame by frame into a Mel spectrogram, and inverse-transform the Mel spectrogram into a time-domain waveform to obtain a first speech.
[0044] Optionally, the noise module 202 is configured to: Extract the semantic features of the scene label, and match the target noise from the preset noise library according to the semantic features; Calculate the energy weights of the first speech and the target noise according to the signal-to-noise ratio, align the Mel spectrograms of the first speech and the target noise on the time axis, and stack them frame by frame according to the energy weights to obtain a first intermediate spectrogram; Convert the first intermediate spectrogram into a time-domain signal through inverse Mel transformation to obtain a noisy speech.
[0045] Optionally, the noise module 202 is configured to: Generate an adversarial perturbation based on the gradient or decision boundary of the target ASR model, and convert the adversarial perturbation into a time-domain waveform or spectral feature to obtain adversarial noise; Locate the injection time window of the adversarial noise according to the speech content, and superimpose the adversarial noise and the noisy speech in the Mel spectrogram domain to obtain a second intermediate spectrogram; Perform inverse Mel transformation on the second intermediate spectrogram to generate the second speech containing adversarial noise.
[0046] Optionally, the annotation module 203 is configured to: Locate the target frame of the adversarial noise in the second speech according to the injection control parameter of the adversarial noise, and obtain noise annotation data according to the target frame; Generate an initial transcription text according to the speech content of the second speech, locate the first target speech segment masked by the scene noise or adversarial noise according to the noise annotation data, correct the initial transcription text of the first target speech segment through the multimodal large model in combination with the context semantics, and use the dynamic time warping algorithm to align the timestamp of the corrected initial transcription text with the second speech to obtain text annotation data; Generate an initial emotion result based on the acoustic features of the second speech, locate the second target speech segment that is not masked by scene noise or adversarial noise according to the noise annotation data, correct the initial emotion result of the second target recording segment according to the corrected initial transcription text, and use the dynamic time warping algorithm to align the timestamp of the corrected initial emotion result with the second speech to obtain emotion annotation data; Obtain the speaker embedding vector of the second speech, calculate the similarity between the speaker embedding vector and a preset speaker library to match the speaker identifier, and obtain speaker annotation data; Generate a multi-modal annotation file according to the noise annotation data, the text annotation data, the emotion annotation data, and the speaker annotation data.
[0047] Optionally, the execution module 204 is configured to: Match the word error rate baseline threshold and the semantic similarity baseline threshold from a preset mapping table according to the scene label; Parse the noise type in the multi-modal annotation file, map the signal-to-noise ratio value in the multi-modal annotation file to a noise intensity level, and perform a first adjustment on the matched word error rate baseline threshold and the semantic similarity baseline threshold according to the noise type and the noise intensity level; Query the speaker attributes from a preset speaker library according to the speaker identifier, and perform a second adjustment on the matched word error rate baseline threshold and the semantic similarity baseline threshold after the first adjustment according to the speaker attributes to obtain the word error rate threshold and the semantic similarity threshold.
[0048] Optionally, the execution module 204 is configured to: Perform time alignment on the transcription text of each speech in the multi-modal annotation file and a preset reference text, count the number of errors, and calculate the word error rate according to the number of errors. The number of errors includes substitution errors, insertion errors, and deletion errors; Perform semantic encoding on the transcription text of each speech in the multi-modal annotation file and a preset reference text, and calculate the cosine similarity; Determine the target corpus from the speeches whose cosine similarity is greater than or equal to the semantic similarity threshold and whose word error rate is less than or equal to the word error rate threshold.
[0049] It should be noted that: when the device provided in the above embodiments realizes its functions, only the division of the above functional modules is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments belong to the same concept. For the specific implementation process, please refer to the method embodiments and will not be elaborated here.
[0050] This embodiment also discloses an electronic device. Referring to Figure 3 , the electronic device may include: at least one processor 301, at least one communication bus 302, a user interface 303, a network interface 304, and at least one memory 305.
[0051] Among them, the communication bus 302 is used to realize the connection and communication between these components.
[0052] Among them, the user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may further include a standard wired interface and a wireless interface.
[0053] Among them, the network interface 304 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).
[0054] Among them, the processor 301 may include one or more processing cores. The processor 301 connects various parts within the entire server through various interfaces and lines, and by running or executing instructions, programs, code sets, or instruction sets stored in the memory 305, as well as calling data stored in the memory 305, it executes various functions of the server and processes data. Optionally, the processor 301 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 301 may integrate a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. in one or several combinations. Among them, the CPU mainly processes the operating system, user interface, and application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above modem may not be integrated into the processor 301 and may be implemented separately by a single chip.
[0055] Among them, the memory 305 may include a Random Access Memory (RAM), or may also include a Read-Only Memory. Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area. Among them, the program storage area can store instructions for implementing the operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above method embodiments, etc.; the data storage area can store the data involved in the above method embodiments. Optionally, the memory 305 may also be at least one storage device located far from the aforementioned processor 301. As Figure 3 shown, in the memory 305 as a computer storage medium, there may be included an operating system, a network communication module, a user interface module, and an application program for a method of generating an ASR audio corpus based on a multi-modal large model.
[0056] In Figure 3 the electronic device shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; while the processor 301 can be used to call the application program for a method of generating an ASR audio corpus based on a multi-modal large model stored in the memory 305. When executed by one or more processors 301, the electronic device is enabled to execute the method of one or more of the above embodiments.
[0057] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.
[0058] In the above embodiments, the descriptions of the respective embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0059] In several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other can be through some service interfaces. The indirect couplings or communication connections of devices or units can be in electrical or other forms.
[0060] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or they can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0061] In addition, each functional unit in various embodiments of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
[0062] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory 305 and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in various embodiments of this application. And the aforementioned memory 305 includes: various media such as USB flash drives, mobile hard disks, magnetic disks, or optical discs that can store program codes.
[0063] The above are only exemplary embodiments of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, all equivalent changes and modifications made according to the teachings of the present disclosure still fall within the scope covered by the present disclosure. After considering the disclosure of the specification, those skilled in the art will easily think of other implementation schemes of the present disclosure. This application aims to cover any variations, uses, or adaptive changes of the present disclosure, and these variations, uses, or adaptive changes follow the general principles of the present disclosure and include common general knowledge or conventional technical means in the technical field not recorded in the present disclosure. The specification and embodiments are only regarded as exemplary, and the scope and spirit of the present disclosure are defined by the claims.
Claims
1. A method for generating ASR audio corpus based on a multimodal large model, characterized in that, Applied to a corpus generation platform, the method includes: Using a multi-modal large model to encode the target domain text into a semantic vector, encoding the voice control parameters and speaker features into conditional vectors, concatenating the semantic vector and the conditional vectors into a joint vector, and decoding the joint vector into a first voice; Selecting target noise from a preset noise library according to the scene label, superimposing the target noise on the first voice to generate a noisy voice, and injecting adversarial noise into the noisy voice to generate a second voice; Performing noise annotation, text annotation, emotion annotation, and speaker annotation on the second voice, and aligning the annotation data to the same time axis to generate a multi-modal annotation file; Setting a word error rate threshold and a semantic similarity threshold according to the scene label, noise type, and speaker information in the multi-modal annotation file, and screening target corpus from the multi-modal annotation file according to the word error rate threshold and the semantic similarity threshold.
2. The method for generating ASR audio corpus based on a multimodal large model according to claim 1, wherein The encoding the voice control parameters and speaker features into conditional vectors, concatenating the semantic vector and the conditional vectors into a joint vector, and decoding the joint vector into a first voice includes: Mapping the speech rate, pause interval, intonation, and emotion intensity to a first vector, mapping the speaker identifier, dialect, voice color, and age to a second vector, and concatenating the first vector and the second vector to generate a conditional vector; Allocating a first weight of the semantic vector and a second weight of the conditional vector through an attention mechanism, and concatenating the semantic vector and the conditional vector along the channel dimension according to the first weight and the second weight to form a joint vector; Decoding the joint vector frame by frame into a Mel spectrogram, and inverse-transforming the Mel spectrogram into a time-domain waveform to obtain a first voice.
3. The method for generating an ASR audio corpus based on a multimodal large model according to claim 1, wherein The selecting target noise from a preset noise library according to the scene label and superimposing the target noise on the first voice to generate a noisy voice includes: Extracting the semantic features of the scene label, and matching the target noise from the preset noise library according to the semantic features; Calculating the energy weights of the first voice and the target noise according to the signal-to-noise ratio, aligning the Mel spectrograms of the first voice and the target noise on the time axis, and superimposing them frame by frame according to the energy weights to obtain a first intermediate spectrogram; Converting the first intermediate spectrogram into a time-domain signal through inverse Mel transformation to obtain a noisy voice.
4. The method for generating ASR audio corpus based on a multimodal large model according to claim 1, wherein The injecting adversarial noise into the noisy voice to generate a second voice includes: Generating an adversarial perturbation based on the gradient or decision boundary of the target ASR model, and converting the adversarial perturbation into a time-domain waveform or spectral feature to obtain adversarial noise; Locating the injection time window of the adversarial noise according to the speech content, and superimposing the adversarial noise and the noisy voice in the Mel spectrogram domain to obtain a second intermediate spectrogram; Performing inverse Mel transformation on the second intermediate spectrogram to generate the second voice containing adversarial noise.
5. The method for generating ASR audio corpus based on a multimodal large model according to claim 4, wherein, The performing noise annotation, text annotation, emotion annotation, and speaker annotation on the second voice, and aligning the annotation data to the same time axis to generate a multi-modal annotation file includes: Locate the target frame of the adversarial noise in the second speech according to the injection control parameter of the adversarial noise, and obtain noise annotation data according to the target frame; Generate an initial transcription text according to the speech content of the second speech, locate the first target speech segment masked by the scene noise or the adversarial noise according to the noise annotation data, correct the initial transcription text of the first target speech segment by combining the context semantics through the multi-modal large model, and use the dynamic time warping algorithm to align the time stamps of the corrected initial transcription text with the second speech to obtain text annotation data; Generate an initial emotion result according to the acoustic features of the second speech, locate the second target speech segment not masked by the scene noise or the adversarial noise according to the noise annotation data, correct the initial emotion result of the second target recording segment according to the corrected initial transcription text, and use the dynamic time warping algorithm to align the time stamps of the corrected initial emotion result with the second speech to obtain emotion annotation data; Obtain the speaker embedding vector of the second speech, calculate the similarity between the speaker embedding vector and a preset speaker library to match the speaker identification, and obtain speaker annotation data; Generate a multi-modal annotation file according to the noise annotation data, the text annotation data, the emotion annotation data and the speaker annotation data.
6. The method for generating ASR audio corpus based on a multimodal large model according to claim 1, wherein The setting of the word error rate threshold and the semantic similarity threshold according to the scene label, noise type and speaker information of the multi-modal annotation file includes: Match the word error rate baseline threshold and the semantic similarity baseline threshold from a preset mapping table according to the scene label; Analyze the noise type in the multi-modal annotation file, map the signal-to-noise ratio value in the multi-modal annotation file to a noise intensity level, and perform a first adjustment on the matched word error rate baseline threshold and the semantic similarity baseline threshold according to the noise type and the noise intensity level; Query the speaker attributes from a preset speaker library according to the speaker identification, and perform a second adjustment on the matched word error rate baseline threshold and the semantic similarity baseline threshold after the first adjustment according to the speaker attributes to obtain the word error rate threshold and the semantic similarity threshold.
7. The method for generating ASR audio corpus based on a multimodal large model according to claim 1, wherein The screening of the target corpus from the multi-modal annotation file according to the word error rate threshold and the semantic similarity threshold includes: Perform time alignment on the transcription text of each speech in the multi-modal annotation file and a preset reference text, count the number of errors, and calculate the word error rate according to the number of errors. The number of errors includes the number of substitution errors, the number of insertion errors and the number of deletion errors; Perform semantic encoding on the transcription text of each speech in the multi-modal annotation file and a preset reference text, and calculate the cosine similarity; Determine the target corpus from the speeches whose cosine similarity is greater than or equal to the semantic similarity threshold and whose word error rate is less than or equal to the word error rate threshold.
8. A generation system for ASR audio corpus based on a multimodal large model, characterized in that, It includes a vector module, a noise module, an annotation module and an execution module, where: A vector module, configured to encode text in a target domain into semantic vectors using a multimodal large model, encode speech control parameters and speaker features into conditional vectors, concatenate the semantic vectors and the conditional vectors into joint vectors, and decode the joint vectors into a first speech; A noise module, configured to select target noise from a preset noise library according to a scene label, superimpose the target noise on the first speech to generate a noisy speech, and inject adversarial noise into the noisy speech to generate a second speech; An annotation module, configured to perform noise annotation, text annotation, emotion annotation and speaker annotation on the second speech, and align the annotation data to the same time axis to generate a multimodal annotation file; An execution module, configured to set a word error rate threshold and a semantic similarity threshold according to the scene label, noise type and speaker information in the multimodal annotation file, and screen target corpus from the multimodal annotation file according to the word error rate threshold and the semantic similarity threshold.
9. An electronic device, characterized in that, It includes a processor, a memory, a user interface and a network interface. The memory is used to store instructions. Both the user interface and the network interface are used to communicate with other devices. The processor is used to execute the instructions stored in the memory, so that the electronic device executes the method according to any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions, which, when executed, execute the method according to any one of claims 1-7.
Citation Information
Patent Citations
Method and device for constructing corpus, computing equipment and storage medium
CN111104546A
Speech recognition system and method for automatic training through speech synthesis method
CN112669825A
Voice data annotation method and device, electronic equipment and storage medium
CN113205814A
Speech corpus generation system training method
CN114120973A
Cross-language corpus synthesis method, speech synthesis model training method and related devices
CN115985283A
Cited By
Data processing method and device and related equipment
CN121278508A
Voice wake-up threshold value self-adaption method and system, electronic equipment and storage medium
CN121331115A
Dynamic noise adaptive speech recognition method and device based on large language model, equipment and medium
CN121983034A
