A deep learning driven partial fake speech data generation method

Abstract

The application discloses a kind of deep learning driven partial counterfeit speech data generation methods, method includes: obtaining real speech data set, and using automatic speech recognition model to the real speech data set is identified, obtains original text information;Using large-scale language model tamperes the original text information, obtains tampered text information;The original text information, the tampered text information and the real speech data set are input into speech editing model and neural network vocoder, generate partial counterfeit speech data;According to the partial counterfeit speech data constructs label information, and outputs the partial counterfeit speech data and the label information;The application proposes new partial counterfeit speech data generation method, can be based on deep speech editing technology, large-scale language model and automatic speech recognition model, generate batch high-quality partial counterfeit speech data.

Description

Technical Field

[0001] This invention relates to the field of multimedia information security and forensics technology, and in particular to a deep learning-driven method for generating partially forged speech data. Background Technology

[0002] Voice signals are a carrier of information and an indispensable medium for communication in human society. Voice signals not only convey content but also contain the unique identity characteristics of the speaker, making them suitable for identity verification in various secure access scenarios and providing strong protection for people's property and privacy. With the rapid development of information technology, the storage, recording, and dissemination of digital voice signals have become increasingly convenient and efficient, allowing people to easily obtain vast amounts of digital voice data from the internet. Simultaneously, various voice editing software programs, such as Adobe Audition, have been designed and developed, enabling anyone to perform various editing operations on voice signals without any barriers. While voice editing software greatly enriches people's entertainment lives, it also allows criminals to easily forge and tamper with digital voice data for nefarious purposes, potentially leading to serious social security problems and personal property losses. However, forged voices generated by voice editing software have identifiable auditory differences from genuine voices and require a large amount of data on the target speaker. With the continuous development of deep learning technology in recent years, deep speech synthesis technology has reached a new level in terms of the naturalness and clarity of synthesized speech. It is now difficult to distinguish between real speech and synthesized speech from the level of human auditory perception alone. Moreover, it only requires a small amount of data to generate speech data with a specific speaker's timbre, further increasing the security risks brought about by deep speech forgery and posing a serious threat to global society, politics and economy.

[0003] Currently, numerous deepfake speech detection methods have been proposed to distinguish between real speech and speech synthesized by deep neural networks; we refer to this synthesized speech as fake speech. These detection methods can effectively differentiate between completely real and completely fake speech to a certain extent, but their performance in detecting partially faked speech is poor. Because partially faked speech contains data from real speech, this makes it even more difficult to detect and poses a greater threat to secure access systems that rely on speech signals for verification, such as automatic speaker verification systems.

[0004] Traditional partial speech forgery involves adding, deleting, or replacing forged speech segments in specific regions of real speech to create partially forged audio. This traditional method requires two spatially independent speech segments from the same speaker to generate a single forged audio segment. Furthermore, because the prosody and tone of the two source speech segments are unrelated, obvious defects such as pauses and abrupt tone changes occur at the splicing points. The shortcomings of traditional speech forgery methods can be summarized in two aspects. First, the forgery generation method itself involves finding two speech segments from the same speaker and then splicing them together to obtain a forged audio segment. However, even when splicing two segments from the same speaker, significant auditory differences still occur at the splicing point, such as sudden pauses, abrupt changes in speech rate, and inconsistent background noise. This is due to the lack of contextual coherence between the two speech segments. Second, the batch data generation method typically uses rule-based methods to modify speech content. The modification operations are often singular and can even lead to semantic logical inconsistencies. For building a partial speech forgery detection system, this type of data does not reflect forged audio in real-world scenarios.

[0005] As the generative capabilities of deep neural networks continue to improve, several speech editing models based on deep neural networks have been proposed. These models can edit speech based on contextual and textual information, directly generating speech waveforms with consistent rhythm and tone, and can edit very small time intervals, such as editing a single word or syllable. While providing convenience, this also provides criminals with a completely new method of forgery and tampering. Compared to traditional partial tampering methods, it offers better consistency in data and auditory perception, and only requires a piece of original speech to generate forged speech data with a very small area of ​​tampering. This forgery method is more flexible and greatly increases the difficulty of detection. Therefore, detecting such forged data is a pressing practical problem, and the batch generation of such forged data is the foundation for subsequent model building and detection methods. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a deep learning-driven method for generating partially forged speech data, in order to address the shortcomings of the existing technology and solve the technical problem that traditional methods cannot generate context-consistent forged speech data in batches.

[0007] The technical solution adopted by this invention to solve the technical problem is as follows:

[0008] In a first aspect, the present invention provides a deep learning-driven method for generating partially forged speech data, comprising:

[0009] Obtain a real speech dataset and use an automatic speech recognition model to recognize the real speech dataset to obtain the original text information;

[0010] The original text information is modified using a large-scale language model to obtain the modified text information.

[0011] The original text information, the tampered text information, and the real speech dataset are input into the speech editing model and the neural network vocoder to generate partial fake speech data;

[0012] Based on the partially forged voice data, label information is constructed, and the partially forged voice data and the label information are output.

[0013] In one implementation, the step of acquiring a real speech dataset and using an automatic speech recognition model to recognize the real speech dataset to obtain raw text information includes:

[0014] Acquire real speech data and standardize and trim the real speech data to obtain the real speech dataset;

[0015] The real speech dataset is input into the automatic speech recognition model to obtain the original text information corresponding to the real speech dataset.

[0016] In one implementation, the step of using a large-scale language model to tamper with the original text information to obtain tampered text information includes:

[0017] The large-scale language model is instructed by a preset prompt to tamper with the original text information;

[0018] The altered text information is obtained by changing the semantics and logic of the original text information through operations such as replacing, deleting, and adding text information.

[0019] In one implementation, the step of using a large-scale language model to tamper with the original text information to obtain tampered text information then includes:

[0020] The original text information and the corresponding tampered text information are stored in a preset file format as a pair of original text information and tampered text information.

[0021] In one implementation, the step of inputting the original text information, the tampered text information, and the real speech dataset into a speech editing model and a neural network vocoder to generate partially forged speech data includes:

[0022] The original text information, the tampered text information, and the real speech dataset are combined into a triplet and input into the speech editing model. The speech editing model then outputs the Mel spectrogram corresponding to the tampered text information.

[0023] The Mel spectrogram of the tampered region is extracted and input into the neural network vocoder to generate a fake speech waveform of the tampered region.

[0024] The real speech in the unaltered area is spliced ​​with the fake speech waveform in the tampered area to obtain the partial fake speech data.

[0025] In one implementation, the step of constructing tag information based on the partially forged speech data and outputting the partially forged speech data and the tag information includes:

[0026] Based on the data sources of each segment in the partially forged speech data, segment-level labels, discourse-level labels, and word-level labels with multiple time resolutions are generated, and the partially forged speech data and the label information are output.

[0027] In one implementation, the step of creating multi-temporal resolution segment-level tags, discourse-level tags, and word-level tags based on the data sources of each segment in the partially forged speech data includes:

[0028] The segment-level labels are assigned to the forged voice data. The forged voice data is divided into segments at preset time intervals. It is determined whether there are tampered areas in the segmented segments. If there are no tampered areas in the segmented segments, a real label is assigned. If there are tampered areas in the segmented segments, a forged label is assigned.

[0029] Assign the speech-level label to the partial forged speech data, determine whether there is a tampered area in the partial forged speech data, if there is a tampered area in the partial forged speech data, then assign the forged label, if there is no tampered area in the partial forged speech data, then assign the real label;

[0030] The forged speech data is assigned word-level labels. It is determined whether there are any traces of tampering in each word of the forged speech data. The words of the forged speech data are labeled with authenticity attributes at the semantic level. If the word has traces of tampering, the forged label is assigned. If the word is an original word, the real label is assigned.

[0031] In one implementation, outputting the partially forged voice data and the tag information includes:

[0032] Output the partially forged speech data, as well as the segment-level tags, discourse-level tags, and word-level tags carried by the partially forged speech data.

[0033] In a second aspect, the present invention also provides a terminal, comprising: a processor and a memory, wherein the memory stores a deep learning-driven partial spoofing speech data generation program, and the deep learning-driven partial spoofing speech data generation program, when executed by the processor, is used to implement the operation of the deep learning-driven partial spoofing speech data generation method as described in the first aspect.

[0034] Thirdly, the present invention also provides a computer-readable storage medium storing a deep learning-driven partial spoofing speech data generation program, which, when executed by a processor, is used to implement the operation of the deep learning-driven partial spoofing speech data generation method as described in the first aspect.

[0035] The present invention, by employing the above technical solution, has the following effects:

[0036] This invention obtains original text information by acquiring a real speech dataset and recognizing it using an automatic speech recognition model. The original text information is then modified using a large-scale language model to obtain modified text information. By inputting the original text information, modified text information, and the real speech dataset into a speech editing model and a neural network vocoder, partially forged speech data can be generated. Label information is constructed based on the partially forged speech data, and the partially forged speech data and corresponding label information are output, thereby achieving batch generation of partially forged speech with high auditory quality. This invention proposes a novel method for generating partially forged speech data, capable of generating batches of context-consistent, diverse, and semantically complete partially forged speech data based on deep speech editing technology, a large-scale language model, and an automatic speech recognition model. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0038] Figure 1 This is a flowchart of a deep learning-driven method for generating partially forged speech data in one implementation of the present invention.

[0039] Figure 2This is a schematic diagram of a partial forged voice data generation process in one implementation of the present invention.

[0040] Figure 3 This is a schematic diagram illustrating the paradigm of speech editing implemented by the deep speech editing module in one implementation of the present invention.

[0041] Figure 4 This is a functional schematic diagram of the terminal in one implementation of the present invention.

[0042] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0043] This invention provides a method, terminal, and storage medium for generating partially forged speech data driven by deep learning. To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0044] Exemplary methods

[0045] Deep speech editing technology can generate partially forged speech data with contextual consistency, significantly increasing the difficulty of detection. Detecting such forged data is a pressing real-world security problem, and the batch generation of this forged speech data forms the basis for subsequent detection methods. Designing a new method to achieve the batch generation of partially forged speech data with contextual consistency is a new requirement in the field of multimedia information security and forensics.

[0046] To address the aforementioned technical problems, embodiments of the present invention provide a deep learning-driven method for generating partially forged speech data. The aim is to propose a novel method for generating partially forged speech data, which can generate batches of context-consistent partially forged speech data based on deep speech editing technology, large-scale language models, and automatic speech recognition models.

[0047] like Figure 1 As shown, this embodiment of the invention provides a deep learning-driven method for generating partially forged speech data, comprising the following steps:

[0048] Step S100: Obtain a real speech dataset and use an automatic speech recognition model to recognize the real speech dataset to obtain the original text information.

[0049] In this embodiment, the deep learning-driven method for generating partially forged speech data is applied in a terminal, which includes, but is not limited to, devices such as computers and mobile terminals; the terminal is equipped with a training and transfer platform for a partially forged speech data generation model based on deep speech editing technology.

[0050] This embodiment proposes a method for generating partially forged speech data. By employing a speech editing model, it addresses the problems of traditional forgery generation methods. Specifically, it modifies the speech content based on context, resulting in samples with no obvious auditory defects, making it impossible for the human ear to discern where the parts were spliced. This makes the forged speech more deceptive. For batch data generation, ChatGPT is used to modify the speech content, ensuring semantic logical integrity. The content modification methods are more flexible and diverse, and better reflect real-world scenarios. Since the benefits of fraud far outweigh the costs, attackers will go to great lengths to create forged speech that can deceive the human ear. Therefore, the data generated in this embodiment better matches real-world scenarios and significantly improves the performance of partially forged speech detection systems. Furthermore, this embodiment addresses the scarcity of existing partially forged speech data, achieving batch generation of context-consistent partially forged speech. This data is more consistent with real-world partially forged speech deception attacks. This greatly assists in building detection systems and protecting social information security.

[0051] like Figure 2 As shown, firstly, this embodiment uses an automatic speech recognition model to recognize real speech samples and obtain original text information; secondly, this embodiment uses a large-scale language model to tamper with the original text information, changing its expression logic and semantics to obtain tampered text information; thirdly, this embodiment sequentially inputs the original text information, tampered text information, and real speech samples into a speech editing model and a neural network vocoder to obtain context-consistent partially forged speech data; finally, this embodiment constructs tag information based on the generated partially forged speech data, including multi-resolution time-resolution segment-level tags, discourse-level tags, and word-level tags.

[0052] Specifically, in one implementation of this embodiment, step S100 includes the following steps:

[0053] Step S101: Obtain real speech data and standardize and crop the real speech data to obtain the real speech dataset.

[0054] In this embodiment, the CSTR VCTK dataset is used as the original real-world speech dataset, which includes speech data from 110 English speakers using different accents. Additionally, to increase the diversity of the original data, publicly available speech data from social media was collected, standardized, and trimmed to an appropriate length to be included as part of the original real-world speech dataset.

[0055] Step S102: Input the real speech dataset into the automatic speech recognition model to obtain the original text information corresponding to the real speech dataset.

[0056] In this embodiment, the original real speech is input into an automatic speech recognition model (such as Whisper) provided by OpenAI to obtain text information corresponding to the real speech content.

[0057] In this embodiment, real speech from any source is input into an automatic speech recognition model (such as Whisper), which then recognizes the real speech samples to obtain the original text information. Only real speech data is needed to generate partially forged speech data with high definition and naturalness, which can be applied to the training and performance evaluation of deepfake speech detection models.

[0058] like Figure 1 As shown, in one implementation of this invention, the deep learning-driven method for generating partially forged speech data further includes the following steps:

[0059] Step S200: Use a large-scale language model to tamper with the original text information to obtain tampered text information.

[0060] In this embodiment, the problem of traditional forgery generation is solved by using a speech editing model. That is, the speech content is modified according to the context, so that the generated sample has no obvious auditory defects and the human ear cannot distinguish where the splicing occurred. Such forged speech is more deceptive.

[0061] Specifically, in one implementation of this embodiment, step S200 includes the following steps:

[0062] Step S201: Instruct the large-scale language model to tamper with the original text information through a preset prompt.

[0063] In this embodiment, a prompt is used to instruct a large language model (such as ChatGPT) to perform a text semantic manipulation task. The large language model is required to modify the input text, altering its semantics and logic to obtain the manipulated text information. The prompt is pre-set and refers to an input text paragraph or phrase, serving as the starting point or guide for the model's output. A descriptive text describing the manipulation operation can be sent to ChatGPT as the prompt, followed by the text information of the spoken content, allowing ChatGPT to manipulate the text content.

[0064] Step S202: By replacing, deleting, and adding text information, the semantic and logical expression of the original text information is changed to obtain the tampered text information.

[0065] In this embodiment, a large-scale language model and an automatic speech recognition model are used to provide the materials needed for speech editing to the deep speech editing model, namely, the modified text information and the original text information, and this process is fully automated in the form of a program script. The modification of text information can be achieved by replacing, deleting, and adding text information to change the expressive semantics and logic of the original text information.

[0066] Step S203: Store the original text information and the corresponding tampered text information in a preset file format as a pair of original text information and tampered text information.

[0067] In this embodiment, the text information and tampered text information of a single voice sample obtained in the above process can be stored in a preset file format, which can be a CSV file format. A program is designed to process a batch of voice samples to extract text information and tampered text information, thereby obtaining a large number of original text information and tampered text information pairs.

[0068] Table 1 shows examples of storing original and tampered text information in CSV files.

[0069]

[0070] The numbers in parentheses in Table 1 for the original text tampered area and the tampered text tampered area are used for code calls. Specifically, they indicate which part of the text was tampered with. For example, index numbers starting from 1, i.e., [4,5], indicate that, for the original text information, the part between the 4th and 5th words was tampered with (excluding the 4th word, including the 5th word). For deletion cases, such as the tampered text [4,4] here, it means that a semantically neutral transition segment was generated between the 4th word and the 5th word, smoothing the pronunciation before and after the deleted segment, making the pronunciation of the deleted word natural and continuous.

[0071] like Figure 1 As shown, in one implementation of this invention, the deep learning-driven method for generating partially forged speech data further includes the following steps:

[0072] Step S300: Input the original text information, the tampered text information, and the real speech dataset into the speech editing model and the neural network vocoder to generate partial fake speech data.

[0073] In this embodiment, after obtaining the original text information and the tampered text information in steps S100 and S200, the original text information, the tampered text information, and the real speech sample are sequentially input into the speech editing model and the neural network vocoder to obtain context-consistent forged speech data. This embodiment utilizes a large-scale language model and an automatic speech recognition model to provide the deep speech editing model with the materials needed for speech editing, namely the tampered text information and the original text information, and automates this process entirely in the form of a program script.

[0074] Specifically, in one implementation of this embodiment, step S300 includes the following steps:

[0075] Step S301: Input the original text information, the tampered text information, and the real speech dataset into the speech editing model as a triple, and output the Mel spectrogram corresponding to the tampered text information through the speech editing model.

[0076] Step S302: Extract the Mel spectrogram of the tampered area, input it into the neural network vocoder, and generate a fake speech waveform of the tampered area.

[0077] Step S303: The real speech in the unaltered area is spliced ​​with the fake speech waveform in the tampered area to obtain the partial fake speech data.

[0078] like Figure 3 As shown, in this embodiment, the acquired original text information and the tampered text information are combined with real speech to form a triple, which is then input into deep speech editing models with various implementation schemes. The deep speech editing models output a Mel spectrogram that matches the tampered text information. By extracting the Mel spectrogram of the tampered portion, it is input into neural network vocoders with various implementation schemes to generate a forged speech waveform for the tampered region. Subsequently, the real speech in the untampered region is concatenated with the forged speech waveform in the tampered region to obtain the final partially forged speech data.

[0079] Specifically, to increase the diversity of the generated forged data, attackers may use various vocoders to generate speech in the tampered areas. Therefore, this embodiment increases data diversity by introducing multiple deep speech editing models and neural network vocoders. Specifically, this embodiment uses four deep speech editing methods and six common neural network vocoders, which can form up to 24 data combinations, ensuring the diversity of the generated speech data.

[0080] Table 2 shows the deep speech editing model and neural network vocoder used.

[0081] Deep speech editing model Neural network vocoder FluentSpeech HiFiGAN EditSpeech MelGAN CampNet Parallel WaveGAN A3T BigVGAN - Diffwav - Style-Melgan

[0082] In this embodiment, the combination of multiple deep speech editing methods and multiple neural network vocoders ensures the data diversity of the generated speech data: multiple latest deep speech editing schemes are used to modify the speech content according to the tampered text information, thereby increasing the data diversity of the generated tampered region Mel spectrogram; multiple neural network vocoders are used to generate waveforms from the generated tampered region Mel spectrogram, thereby increasing the data diversity of the speech waveform.

[0083] like Figure 1 As shown, in one implementation of this invention, the deep learning-driven method for generating partially forged speech data further includes the following steps:

[0084] Step S400: Construct tag information based on the partial forged voice data, and output the partial forged voice data and the tag information.

[0085] In this embodiment, after obtaining the original text information and the tampered text information in steps S100 and S200, the original text information, the tampered text information, and the real speech sample can be sequentially input into the speech editing model and the neural network vocoder to obtain high-quality partial forged speech data.

[0086] Specifically, in one implementation of this embodiment, step S400 includes the following steps:

[0087] Step S401: Based on the data sources of each segment in the partially forged speech data, create segment-level tags, discourse-level tags, and word-level tags with multiple time resolutions, and output the partially forged speech data and the tag information.

[0088] Step S402: Output the partially forged speech data and the segment-level tags, discourse-level tags, and word-level tags carried by the partially forged speech data.

[0089] In this embodiment, since the deep learning-based forgery detection model requires both speech and labels for training, after obtaining some forged speech data, this embodiment also needs to generate corresponding label information based on the forged speech. The purpose of the labels is to construct the deep learning-based forgery detection system; the labels record which part of the speech sample is tampered with and which part is original.

[0090] Specifically, in one implementation of this embodiment, step S401 includes the following steps:

[0091] Step S401a: Assign the segment-level label to the partial forged voice data, divide the partial forged voice data into segments according to a preset time interval, determine whether there is a tampered area in the segmented segments, if there is no tampered area in the segmented segments, assign a real label, if there is a tampered area in the segmented segments, assign a forged label.

[0092] Step S401b: Assign the speech-level label to the partial forged speech data, determine whether there is a tampered area in the partial forged speech data, if there is a tampered area in the partial forged speech data, then assign the forged label, if there is no tampered area in the partial forged speech data, then assign the real label.

[0093] Step S401c: Assign word-level labels to the partial forged speech data, determine whether there are any tampering traces in each word of the partial forged speech data, and label the authenticity attributes of the words of the partial forged speech data from a semantic level. If the word has tampering traces, assign the forged label; if the word is an original word, assign the real label.

[0094] In this embodiment, multi-temporal resolution segment-level labels, discourse-level labels, and word-level labels are created based on the data sources of each segment in the partially forged speech data. Multi-temporal resolution labels assign true / false labels to time frames at different temporal resolutions in the partially forged speech samples. Multi-temporal resolution segment-level labels involve dividing a generated single speech sample into segments at certain time intervals (e.g., 20ms, 40ms, etc.). If the segment has not been tampered with, it is given a true label (represented by 1); if there is a tampered area within the segment, it is given a forged label (represented by 0). Multi-temporal resolution word-level labels, from a semantic perspective, label the true / false attributes of words within a single speech sample. If a word has been tampered with, it is given a forged label (represented by 0); if it is the original word, it is given a true label (represented by 1). Multi-temporal resolution discourse-level labels indicate that if any part of the entire speech sample has been tampered with, the sample is considered a forged sample (represented by 0); otherwise, it is a true sample (represented by 1). All of these labels can be used in the training process of the detection model.

[0095] In this embodiment, a complete partial forged speech generation process was designed using a deep speech editing model, a large-scale language model, and an automatic speech recognition model. This process can generate a large amount of context-consistent partial forged speech data. The partially forged speech data generated in this embodiment has a smaller tampering area and better coherence. It can be used for training and performance evaluation of deep forged speech detection models, solving the problem of the scarcity of existing partial forged speech data. It achieves the batch generation of context-consistent partial forged speech, and this data is more consistent with real-world partial forged speech spoofing attacks, which can greatly help build detection systems and protect social information security.

[0096] This embodiment achieves the following technical effects through the above technical solution:

[0097] This embodiment acquires a real speech dataset and uses an automatic speech recognition model to recognize the real speech dataset to obtain original text information. It then uses a large-scale language model to tamper with the original text information to obtain tampered text information. The original text information, the tampered text information, and the real speech dataset are input into a speech editing model and a neural network vocoder to generate partially forged speech data. Label information is constructed based on the partially forged speech data, and the partially forged speech data and the label information are output. Deep speech editing technology can generate context-consistent partially forged speech data, which greatly increases the difficulty of detection. Detecting such forged speech data is a pressing real-world security problem, and the batch generation of such forged speech data is the foundation for building models and detection methods. This embodiment designs a deep learning-driven method for generating partially forged speech data, constructing a complete process from a single original speech data point to a single partially forged speech data point. The generated partially forged speech data has the characteristics of context consistency, rich diversity, and complete semantic logic, and can be applied to the training and performance evaluation of deep partially forged speech detection models, thereby helping to improve the detection performance of such partially forged data. This embodiment can play an important role in designing and building deep forged speech detection models and other practical applications.

[0098] Exemplary device

[0099] Based on the above embodiments, the present invention also provides a terminal, comprising: a processor, a memory, an interface, a display screen, and a communication module connected via a system bus; wherein, the processor is used to provide computing and control capabilities; the memory includes a storage medium and internal memory; the storage medium stores an operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the storage medium; the interface is used to connect to external devices, such as mobile terminals and computers; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or a mobile terminal.

[0100] When the computer program is executed by the processor, it is used to implement a deep learning-driven method for generating partially forged speech data.

[0101] It will be understood by those skilled in the art that Figure 4 The schematic diagram shown is only a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. The specific terminal may include more or fewer components than shown in the figure, or combine some components, or have different component arrangements.

[0102] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing a deep learning-driven partial spoofing speech data generation program, which, when executed by the processor, is used to implement the operation of the deep learning-driven partial spoofing speech data generation method as described above.

[0103] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a deep learning-driven partially spoofed speech data generation program, which, when executed by the processor, is used to implement the operation of the deep learning-driven partially spoofed speech data generation method as described above.

[0104] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory.

[0105] In summary, this invention provides a deep learning-driven method, terminal, and storage medium for generating partially forged speech data. The method includes: acquiring a real speech dataset and using an automatic speech recognition model to recognize the real speech dataset to obtain original text information; using a large-scale language model to tamper with the original text information to obtain tampered text information; inputting the original text information, the tampered text information, and the real speech dataset into a speech editing model and a neural network vocoder to generate partially forged speech data; constructing tag information based on the partially forged speech data and outputting the partially forged speech data and the tag information. This invention proposes a novel method for generating partially forged speech data, which can generate batches of high-quality partially forged speech data based on deep speech editing technology, a large-scale language model, and an automatic speech recognition model.

[0106] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A deep learning-driven method for generating partially forged speech data, characterized in that, include: Obtain a real speech dataset and use an automatic speech recognition model to recognize the real speech dataset to obtain the original text information; The original text information is modified using a large-scale language model to obtain the modified text information. The original text information, the tampered text information, and the real speech dataset are input into the speech editing model and the neural network vocoder to generate partial fake speech data; Based on the partially forged voice data, a tag information is constructed, and the partially forged voice data and the tag information are output; The step of constructing tag information based on the partially forged voice data and outputting the partially forged voice data and the tag information includes: Based on the data sources of each segment in the forged speech data, multi-temporal resolution segment-level labels, discourse-level labels, and word-level labels are generated, and the forged speech data and the label information are output. The process of creating multi-temporal resolution segment-level tags, discourse-level tags, and word-level tags based on the data sources of each segment in the partially forged speech data includes: The segment-level labels are assigned to the forged voice data. The forged voice data is divided into segments at preset time intervals. It is determined whether there are tampered areas in the segmented segments. If there are no tampered areas in the segmented segments, a real label is assigned. If there are tampered areas in the segmented segments, a forged label is assigned. Assign the speech-level label to the partial forged speech data, determine whether there is a tampered area in the partial forged speech data, if there is a tampered area in the partial forged speech data, then assign the forged label, if there is no tampered area in the partial forged speech data, then assign the real label; The forged speech data is assigned word-level labels. It is determined whether there are any traces of tampering in each word of the forged speech data. The words of the forged speech data are labeled with authenticity attributes at the semantic level. If the word has traces of tampering, the forged label is assigned. If the word is an original word, the real label is assigned.

2. The deep learning-driven method for generating partially forged speech data according to claim 1, characterized in that, The process of acquiring a real speech dataset and using an automatic speech recognition model to recognize the real speech dataset to obtain raw text information includes: Acquire real speech data and standardize and trim the real speech data to obtain the real speech dataset; The real speech dataset is input into the automatic speech recognition model to obtain the original text information corresponding to the real speech dataset.

3. The deep learning-driven method for generating partially forged speech data according to claim 1, characterized in that, The process of using a large-scale language model to tamper with the original text information to obtain tampered text information includes: The large-scale language model is instructed by a preset prompt to tamper with the original text information; The altered text information is obtained by changing the semantics and logic of the original text information through operations such as replacing, deleting, and adding text information.

4. The deep learning-driven method for generating partially forged speech data according to claim 1, characterized in that, The process of using a large-scale language model to tamper with the original text information to obtain tampered text information then includes: The original text information and the corresponding tampered text information are stored in a preset file format as a pair of original text information and tampered text information.

5. The deep learning-driven method for generating partially forged speech data according to claim 1, characterized in that, The step of inputting the original text information, the tampered text information, and the real speech dataset into the speech editing model and the neural network vocoder to generate partially forged speech data includes: The original text information, the tampered text information, and the real speech dataset are combined into a triplet and input into the speech editing model. The speech editing model then outputs the Mel spectrogram corresponding to the tampered text information. The Mel spectrogram of the tampered region is extracted and input into the neural network vocoder to generate a fake speech waveform of the tampered region. The real speech in the unaltered area is spliced ​​with the fake speech waveform in the tampered area to obtain the partial fake speech data.

6. The deep learning-driven method for generating partially forged speech data according to claim 1, characterized in that, The output of the partial forged voice data and the tag information includes: Output the partially forged speech data, as well as the segment-level tags, discourse-level tags, and word-level tags carried by the partially forged speech data.

7. A terminal, characterized in that, include: The processor and memory, wherein the memory stores a deep learning-driven partial spoofing speech data generation program, which, when executed by the processor, is used to implement the operation of the deep learning-driven partial spoofing speech data generation method as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a deep learning-driven partial spoofing speech data generation program, which, when executed by a processor, is used to implement the operation of the deep learning-driven partial spoofing speech data generation method as described in any one of claims 1-6.

Images