Obstacle speech recognition and reconstruction method based on speech data retrieval enhancement technology

Abstract

The invention discloses a barrier speech recognition and reconstruction method based on a speech data retrieval enhancement technology. The method comprises the following steps: segmenting a to-be-recognized obstacle speech into a plurality of segments; for each segment, obtaining a plurality of similar voice segments and corresponding text contents by retrieving a barrier voice database; respectively coding the similar voice segments and the to-be-recognized obstacle voice to obtain similar voice codes and to-be-recognized obstacle voice codes; respectively converting the similar speech codes and the to-be-recognized obstacle speech codes into first codes and second codes understood by a large language model; the first code and the second code are input into a large language model, and text information corresponding to the to-be-recognized obstacle voice is obtained by using set cue words, and the cue words comprise historical dialogue content of the to-be-recognized obstacle voice and text content corresponding to similar voice segments; and performing voice reconstruction on the recognized text information. According to the invention, the accuracy and generalization ability of obstacle speech recognition can be improved.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and more specifically, to an obstacle speech recognition and reconstruction method based on speech data retrieval enhancement technology. Background Art

[0002] Dysarthria refers to a speech disorder caused by paralysis, weakened contraction or uncoordinated movement of speech-related muscles due to neurological diseases. The pronunciation of patients with dysarthria is generally intermittent and unclear, making it difficult for them to communicate with others. In recent years, with the development of deep learning, speech recognition (ASR) technology has made great progress, but the recognition of disordered speech for patients with dysarthria is still a challenging task. In order to better recognize disordered speech, researchers have designed methods based on speech recognition and combined the characteristics of disordered speech to improve model performance. For example, artificial neural networks (ANN) are used to analyze the experimental results of ASR for dysarthric speech. For another example, hidden Markov models (HMM) are used to design ASR for dysarthric speech, and the results of different acoustic features such as fast Fourier transform, linear prediction and cepstral coefficients are compared. Due to the limitation of equipment computing power, these studies have not achieved particularly significant results. With the development of ASR technology using deep learning methods and the substantial improvement of computing power, a large number of studies have been carried out to improve the performance of ASR for dysarthric speech. For example, Takashima et al. proposed an end-to-end ASR framework that jointly encapsulates acoustic and language models, where the acoustic model part of the framework is shared across speakers with dysarthria and the language model part of the framework is assigned to each language regardless of dysarthria.

[0003] Large language models (LLMs) are highly complex and powerful machine learning models in the field of artificial intelligence, which are specifically designed to understand and generate human language. These models learn language patterns, grammar, semantics, and contextual relationships by training on large amounts of text data. At present, large language models such as Chat GPT, Wenxin Yiyan, Tongyi Qianwen, Kimi, etc., are able to follow longer human language text instructions (i.e., prompt words) and give expected responses, showing language capabilities close to humans. Thanks to the context understanding ability of large language models, some researchers have integrated large language models into speech recognition of normal speech and achieved good results. For example, Zheng et al. proposed a method to convert the output probability of a bidirectional language model into an effective language prior probability combined with large language models such as GPT, and obtained better recognition results than the ASR system using traditional neural network language models.

[0004] Retrieval-Augmented Generation (RAG) is a technology that combines information retrieval and large language model (LLM) generation capabilities. It enhances the results of AI-generated content by retrieving relevant content from data stores, improving accuracy and robustness. RAG technology is particularly suitable for tasks that require domain-specific knowledge, and can effectively avoid the hallucination problem of large models, provide authoritative knowledge, and ensure data security.

[0005] Text-to-speech technology (TTS) is a technology that converts text into speech, which enables computers to simulate human speech. TTS technology plays an important role in human-computer interaction and is widely used in smart assistants, smart customer service, audiobooks, call centers, and in-vehicle entertainment equipment. The development of TTS technology has gone through the process from early mechanical speech synthesis to the current application of natural language processing and deep learning technology. Modern TTS systems not only focus on the naturalness and fluency of the sound, but also strive to personalize and emotionalize it to provide a richer and more realistic interactive experience.

[0006] In the prior art, patent application CN202211156788.6 discloses a feature-enhanced method for processing dysarthric speech. The method comprises the following steps: S1, performing a fast Fourier transform on the original signal to calculate its spectrum signal; S2, performing an empirical mode decomposition on the spectrum signal to obtain each eigenmode decomposition component; S3, calculating the power spectrum density of the first m eigenmode decomposition components to obtain a power spectrum feature vector; S4, performing a fast Walsh-Hadamard transform on the first m eigenmode decomposition components to obtain Walsh transform coefficients, and then extracting the statistical features of each Walsh transform coefficient to obtain a statistical feature vector; S5, combining the power spectrum feature vector and the statistical feature vector to obtain a combined feature vector. This scheme improves the accuracy of dysarthric speech recognition. Patent application CN202110512052.7 discloses a method and system for dysarthric speech recognition based on visual facial contour movement. The system includes multimodal data acquisition, multimodal fusion feature calculation, multimodal speech recognition calculation and language model calculation modules; the multimodal data acquisition calculation module is used to obtain the facial contour motion video data of the dysarthric person and the speech data synchronized with the video; the multimodal fusion feature calculation module is used to fuse the facial contour motion features and the speech acoustic features; the multimodal speech recognition calculation module is used to obtain the mapping relationship from the multimodal features to the phoneme characters; the language model calculation module is used to obtain the mapping relationship from the phoneme characters to the Chinese sentences. This scheme obtains the fused multimodal features by fusing the speech acoustic feature parameters with the pronunciation movements of the dysarthric person, and uses the fused multimodal features for dysarthric speech recognition, thereby effectively improving the accuracy of dysarthric speech recognition.

[0007] After analysis, the existing speech recognition solutions have the following main defects:

[0008] 1) Existing speech recognition technology cannot retrieve the content of speech database for reference, so the accuracy is low. At the same time, most traditional obstacle speech recognition methods use N-gram language models, which cannot integrate long-length context for recognition.

[0009] 2) Existing speech recognition methods based on large language models are difficult to apply to impaired speech. Due to the ambiguity of impaired speech, its pronunciation is difficult to directly establish a correspondence with the input text of the large language model, so the speech must be converted into coded content that the large model can understand. Summary of the invention

[0010] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for obstacle speech recognition and reconstruction based on speech data retrieval enhancement technology. The method comprises the following steps:

[0011] Based on the length of the data in the disordered speech database, the disordered speech to be recognized is divided into multiple segments;

[0012] For each of the multiple segments, encoding is performed using a first speech encoder, and a plurality of similar speech segments and corresponding text contents are obtained by searching a disordered speech database;

[0013] For the multiple similar speech segments and the to-be-recognized impaired speech, respectively, the same second speech encoder is used to encode them to obtain corresponding similar speech codes and to-be-recognized impaired speech codes;

[0014] Using a trained speech code converter, convert the similar speech code and the to-be-identified obstacle speech code into a first code and a second code that can be understood by a large language model respectively;

[0015] Inputting the first code and the second code into the large language model, and using the set prompt words, obtaining text information corresponding to the to-be-recognized disordered speech, wherein the prompt words include the historical conversation content of the to-be-recognized disordered speech and the text content corresponding to the multiple similar speech segments;

[0016] For the recognized text information, speech synthesis technology is used to reconstruct the speech.

[0017] Compared with the prior art, the advantages of the present invention are that, in view of the problems of relatively insufficient data of disordered speech, large differences between disordered speech and normal speech, and difficulty in using conversation history information in existing speech recognition schemes, the disordered speech recognition and reconstruction method based on speech data retrieval enhancement technology proposed by the present invention uses a large pre-trained language model obtained from a large amount of data to make up for the problem of insufficient data of disordered speech, and uses historical conversation content to prompt learning of the large model, and finally recognizes disordered speech in a targeted manner through contextual learning capabilities and enhanced retrieval technology, and then uses existing speech synthesis technology (TTS) to generate correct speech based on accurate recognition results. The present invention reduces the impact of speech differences on recognition results, improves recognition accuracy, and can convert disordered speech into easy-to-understand text and normal speech, which is conducive to normal communication between articulators and the outside world.

[0018] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0020] Figure 1 is a flow chart of a method for obstacle speech recognition and reconstruction based on speech data retrieval enhancement technology according to an embodiment of the present invention;

[0021] Figure 2 is a process diagram of a method for identifying and reconstructing an obstructed speech based on a speech data retrieval enhancement technique according to an embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of retrieval enhancement of barrier speech according to an embodiment of the present invention;

[0023] Figure 4 is a schematic diagram of speech recognition using a large language model according to an embodiment of the present invention;

[0024] Figure 5 It is a schematic diagram of speech reconstruction based on speech synthesis technology according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0026] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0027] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0028] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0029] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0030] The present invention aims to utilize the contextual learning ability of a large language model (or simply the large model) and the retrieval enhancement technology to achieve high-precision recognition of the speech of patients with dysarthria. In general, the present invention conducts targeted and efficient retrieval of similar content of the speech of patients with dysarthria in the disordered speech database, and submits the retrieval results to the large language model, which then accurately recognizes the speech with its good contextual learning ability combined with the context. After recognition, real-time speech reconstruction is completed using existing speech synthesis technology so that patients with dysarthria can communicate normally with the outside world.

[0031] Specifically, combined Figure 1 and Figure 2 As shown, the provided obstacle speech recognition and reconstruction method based on speech data retrieval enhancement technology includes the following steps:

[0032] Step S1, based on the length of the data in the obstacle speech database, the obstacle speech to be recognized is divided into multiple segments.

[0033] Combination Figure 3 As shown, for the content of the disordered speech to be recognized, a segmenter is first used to segment it into multiple small speech segments, so that the length of each segment is close to the length of the data in the disordered speech database.

[0034] Specifically, the length L of the shortest speech in the disordered speech database (or database for short) is recorded. min , the length of the longest speech is L max , the average silence length M of all silent segments of all voices in the database mean , and the standard deviation of silence length M devPerform voice endpoint detection (VAD) on the speech to be recognized from the beginning, and whenever a silent segment appears, record its length as M. In one embodiment, the segmenter considers whether to perform speech segmentation in this silent segment according to the following strategy:

[0035] 1. If the length of the speech segment obtained by segmentation here is L <L min , no segmentation;

[0036] 2. If the length of the speech segment obtained by segmentation here is L ≥ L min And L <L max , while M≤M mean +2*M dev , no segmentation;

[0037] 3. If the length of the speech segment obtained by segmentation here is L ≥ L min And L <L max , while M>M mean +2*M dev , cut in this silent segment;

[0038] 4. If the length of the speech segment obtained by segmentation here is L ≥ L max , split at the longest silent segment between the previous split point and the current split point.

[0039] In summary, step S1 divides the obstacle speech to be recognized into multiple speech segments with a length similar to that of the data in the obstacle speech database, which are recorded as n segments.

[0040] Step S2: For each segmented segment, a speech encoder is used to encode the segment, and a plurality of similar speech segments of the segment and corresponding text content are obtained by searching the obstacle speech database.

[0041] Still combined Figure 3 As shown, for each segmented segment, a speech encoder (such as HuBERT encoder, Whisper encoder or Wav2Vec encoder, etc.) is used to encode the corresponding codes of multiple segments. Then, based on similarity (such as cosine similarity or Euclidean distance as a measurement method), k disordered speech with the corresponding codes closest to the segment code and the text content corresponding to the disordered speech are found in the disordered speech database.

[0042] For example, a total of n*k similar disordered speech is obtained, and then they are sorted from high to low according to the similarity metric, and the top k most similar disordered speech and their corresponding text content are selected as reference content for subsequent large-model speech recognition.

[0043] In summary, step 2 performs a database search on the disordered speech to be recognized, and based on certain rules (such as the similarity of the encoded content of different speech after encoding using a speech encoder), searches the disordered speech database for content similar to the speech to be recognized, and obtains the k most similar disordered speech segments and their corresponding text contents.

[0044] Step S3: Encode the retrieved multiple similar speech segments and the to-be-recognized obstacle speech using the same encoder to obtain corresponding speech encoding vectors.

[0045] The k speech segments obtained from the database (referred to as sample speech) and the obstacle speech to be recognized (referred to as test speech) are respectively encoded into vectors using the same speech encoder. The speech encoder is, for example, a HuBERT encoder, a Whisper encoder, or a Wav2Vec encoder. The speech encoder here may be the same as or different from the speech encoder in step S2, and the present invention is not limited to this.

[0046] Step S4, using the trained speech code converter, converts the speech code vector into a code that can be understood by the large language model.

[0047] In step S4, the impaired speech to be recognized and the similar impaired speech detected by the database are converted to generate codes that can be understood by the large model. For example, a speech encoder is used to generate codes for the speech content, and then the code content is converted by specific rules (such as a trained code converter) to generate codes that can be understood by the large model, that is, the speech code vector is converted into codes that can be understood by the large language model by using a speech code converter.

[0048] Step S5, based on the prompt words, the conversation history information, and the codes and corresponding text contents of the retrieved similar speech segments, the content information corresponding to the obstacle speech to be recognized is obtained using the large language model.

[0049] For example, the encoded vector is converted into a code that can be understood by the large model through a trained encoder converter, and then input into the large model. Figure 4 This is a schematic diagram of speech recognition based on a large model, taking k = 2 as an example. For the text input (prompt word) of the large model, for example, it is written in the following format:

[0050] ***********************************************

[0051] Please refer to the following sample speech and text of a patient with dysarthria, as well as the text of the historical conversation content before the test speech, to generate the text corresponding to the test speech:

[0052] <Sample Voice 1>, Sample 1 corresponds to the text: Sample text 1

[0053] <Sample Voice 2>, Sample 2 corresponding text: Sample text 2

[0054] …

[0055] <Sample speech k>, sample k corresponds to text: Sample text k

[0056] Historical dialogue text: Historical dialogue content

[0057] <Test voice>, corresponding text:

[0058] *******************************************************

[0059] The large model automatically generates the text corresponding to the test speech according to the above prompt words. The angle brackets indicate the encoding and converted content of the above speech, and the underline indicates the text obtained from the database and the historical conversation content.

[0060] It should be noted that for the training problem of the encoder converter, the encoder converter can be trained by changing different voice data and corresponding texts through the same speech encoder, large model, and prompt words. It is worth noting that the encoder converter is only responsible for converting speech content into text content, and the obstacle speech data is very scarce, so normal speech data can be used for training. The encoder converter can be of various types. For example, a multi-layer neural network can be used to convert speech coding from the input space to the input space of the large model, or a structure that allows speech coding to cross-fuse information within the large model.

[0061] In summary, in step S5, the prompt word, the conversation history information, the code of the obstacle speech to be recognized, the code of the retrieved obstacle speech segment and the text content therein are input into the large language model, and the obstacle speech content is accurately recognized after being processed by the large language model. This speech recognition solution that uses a large language model combined with context can obtain more accurate recognition results.

[0062] Step S6: reconstruct the recognized voice content information based on speech synthesis technology.

[0063] Figure 5 It is an illustration of speech reconstruction based on a speech synthesis model. After correctly identifying the disordered speech content using a large model, normal speech can be reconstructed from the text content through a speech synthesis model or other existing speech synthesis technologies, thereby enabling real-time communication between people with articulation disorders and others.

[0064] In one embodiment, the speech synthesis technology is a speech synthesis model of VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech). The core of VITS is to combine variational autoencoder (VAE) with Transformer to achieve seamless conversion from text to speech. This design enables the speech synthesis model to learn the continuous distribution of audio features during training, thereby generating more fluent and realistic speech samples.

[0065] In summary, compared with the prior art, the present invention has the following advantages:

[0066] 1) The present invention obtains content similar to the speech to be recognized for reference by retrieving the database, so that the large model can perform speech recognition by referring to the retrieved speech data. This retrieval enhancement technology improves the accuracy of language recognition and reduces the demand for obstacle speech sample data in the model training process.

[0067] 2) The present invention integrates obstructive speech into the large model for context learning and historical dialogue prompt learning, and performs speech recognition by integrating long-length context, thereby further improving the accuracy and generalization ability of speech recognition.

[0068] 3) The present invention takes into account the ambiguity of obstructive speech and the problem that its pronunciation is difficult to directly correspond to the input text of the large language model. The speech is converted into encoded content that can be understood by the large model, making full use of the context understanding ability of the large language model.

[0069] 4) It has been proven that more fluent and realistic speech can be generated for speech disorders with different characteristics (such as different lengths, different ambiguity, etc.).

[0070] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0071] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0072] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0073] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0074] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0075] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0076] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0077] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0078] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for identifying and reconstructing obstructed speech based on speech data retrieval enhancement technology, comprising the following steps: Based on the length of the data in the disordered speech database, the disordered speech to be recognized is divided into multiple segments; For each of the multiple segments, encoding is performed using a first speech encoder, and a plurality of similar speech segments and corresponding text contents are obtained by searching a disordered speech database; For the multiple similar speech segments and the to-be-recognized impaired speech, respectively, the same second speech encoder is used to encode them to obtain corresponding similar speech codes and to-be-recognized impaired speech codes; Using a trained speech code converter, convert the similar speech code and the to-be-identified obstacle speech code into a first code and a second code that can be understood by a large language model respectively; Inputting the first code and the second code into the large language model, and using the set prompt words, obtaining text information corresponding to the to-be-recognized disordered speech, wherein the prompt words include the historical conversation content of the to-be-recognized disordered speech and the text content corresponding to the multiple similar speech segments; For the recognized text information, speech synthesis technology is used to reconstruct the speech.

2. The method according to claim 1, characterized in that Divide the obstacle speech to be recognized into multiple segments according to the following steps: Determine the shortest speech length L in the disordered speech database min , the longest speech length L max , and the average silence length M of all silent segments of speech mean , and the standard deviation of silence length M dev ; For the obstructed speech to be recognized, speech endpoint detection is performed. Whenever a silent segment appears, its length is recorded as M, and whether to segment the silent segment is determined according to the following strategy: If the length of the speech segment obtained by segmentation is L <L min , no segmentation; If the length of the speech segment obtained by segmentation here is L ≥ L min And L <L max , and M≤M mean +2*M dev , no segmentation; If the length of the speech segment obtained by segmentation here is L ≥ L min And L <L max , and M>M mean +2*M dev , cut in this silent segment; If the length of the speech segment obtained by segmentation here is L ≥ L max , then the segmentation will be performed at the longest silent segment between the previous segmentation point and the current segmentation point.

3. The method according to claim 1, characterized in that The first speech encoder and the second speech encoder may be the same or different types of encoders.

4. The method according to claim 3, characterized in that The first speech encoder and the second speech encoder are selected from a HuBERT encoder, a Whisper encoder, or a Wav2Vec encoder.

5. The method according to claim 1, characterized in that The similarity between the multiple similar speech segments is measured based on cosine similarity or Euclidean distance.

6. The method according to claim 1, characterized in that The speech synthesis technology is the VITS speech synthesis model.

7. The method according to claim 1, characterized in that The speech coder is a multi-layer neural network.

8. The method according to claim 1, characterized in that The speech coder is trained using normal speech data and is used to convert speech content into text content.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images