A speech-to-text method, apparatus, computer device, medium, and program product

Abstract

This application discloses a speech-to-text method, apparatus, computer device, medium, and program product. It acquires multiple types of speech, extracts features from these speech types to obtain their acoustic features, performs type recognition on the speech types, and obtains a first sub-text as a prompt word. Finally, a large language model analyzes the acoustic features of the multiple speech types and the first sub-text. The large language model itself has the ability to learn multiple language types, and based on this ability, it can understand the meaning of each type of sub-speech in the multiple speech types based on their acoustic features. Furthermore, the large language model also has context learning capabilities. Based on this capability, in the process of predicting the text corresponding to each sub-speech, the large language model matches prompt words for the first sub-speech belonging to the first type, which is more difficult to convert, to reduce the recognition difficulty, thereby obtaining the converted text corresponding to the multiple speech types.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a speech-to-text method, apparatus, computer equipment, medium, and program product. Background Technology

[0002] Speech-to-text is a technology that converts speech into text, and it is frequently used in scenarios such as smart homes and transcribing meeting content.

[0003] In related technologies, a model with speech-to-text functionality is typically pre-trained, and then the speech is input into the model, which then converts the speech to obtain the corresponding text.

[0004] However, in some scenarios, speech generally includes multiple types (hereinafter referred to as multi-type speech), such as Chinese speech mixed with a small amount of English speech (e.g., the speech form of "I have a meeting today, I may be home late"), etc. The aforementioned model generally achieves speech-to-text function through contextual understanding, resulting in a low accuracy of the model for multi-type speech conversion. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a speech-to-text method, apparatus, computer device, medium, and program product for improving the accuracy of converting various types of speech into text.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] On one hand, embodiments of this application provide a method for speech-to-text conversion, the method comprising:

[0008] Acquire multiple types of speech, the multiple types of speech including a first sub-speech belonging to a first type and a second sub-speech belonging to a second type, wherein in the multiple types of speech, the first number of the first sub-speech belonging to the first type is less than the second number of the second sub-speech belonging to the second type, and the first type and the second type are different types;

[0009] Feature extraction is performed on the multiple types of speech to obtain the acoustic features of the multiple types of speech;

[0010] Type recognition is performed on the multiple types of speech to obtain a first sub-text belonging to the first type. The first sub-text is the text corresponding to the first sub-speech of the first type in the multiple types of speech.

[0011] The acoustic features of the multiple speech types and the first sub-text are analyzed using a large language model to obtain the converted text corresponding to the multiple speech types. The converted text includes the text corresponding to the first sub-speech and the text corresponding to the second sub-speech.

[0012] On the other hand, embodiments of this application provide a speech-to-text apparatus, the apparatus comprising: an acquisition unit, a feature extraction unit, a recognition unit, and an analysis unit;

[0013] The acquisition unit is used to acquire multiple types of speech, which include a first sub-speech belonging to a first type and a second sub-speech belonging to a second type. In the multiple types of speech, the first number of first sub-speech belonging to the first type is less than the second number of second sub-speech belonging to the second type. The first type and the second type are different types.

[0014] The feature extraction unit is used to extract features from the multiple types of speech to obtain the acoustic features of the multiple types of speech.

[0015] The recognition unit is used to perform type recognition on the multiple types of speech to obtain a first sub-text belonging to the first type, wherein the first sub-text is the text corresponding to the first sub-speech of the first type in the multiple types of speech;

[0016] The analysis unit is used to analyze the acoustic features of the multiple speech types and the first sub-text using a large language model to obtain the converted text corresponding to the multiple speech types. The converted text includes the text corresponding to the first sub-speech and the text corresponding to the second sub-speech.

[0017] On the other hand, embodiments of this application provide a computer device, the computer device including a processor and a memory:

[0018] The memory is used to store computer programs and to transfer the computer programs to the processor;

[0019] The processor is configured to execute the methods described above according to instructions in the computer program.

[0020] On the other hand, embodiments of this application provide a computer-readable storage medium for storing a computer program for performing the methods described above.

[0021] On the other hand, embodiments of this application provide a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the methods described above.

[0022] As can be seen from the above technical solution, multiple types of speech are acquired, including first sub-speech belonging to the first type and second sub-speech belonging to the second type. The number of first sub-speech belonging to the first type is less than the number of second sub-speech belonging to the second type, meaning that the first sub-speech belonging to the first type is sparse data in the multiple speech types, making its conversion more difficult. Feature extraction is performed on the multiple speech types to obtain their acoustic features. These acoustic features reflect the characteristics of the speech, thus helping the large language model understand the multiple speech types. Type recognition is performed on the multiple speech types to obtain the first sub-text. This first sub-text not only belongs to the first type but is also the text corresponding to the first sub-speech belonging to the first type in the multiple speech types. Using this as a cue word helps the large language model understand the more difficult-to-convert first sub-speech belonging to the first type. Finally, the large language model analyzes the acoustic features and the first sub-text of the multiple speech types. The large language model itself has the ability to learn multiple language types, and based on this ability, it can understand the meaning of each type of sub-speech in the multiple speech types based on the acoustic features. Furthermore, the large language model also has context learning capabilities. Based on this capability, in the process of predicting the text corresponding to each sub-speech, the large language model matches the first sub-text of the first sub-speech belonging to the first type, which is more difficult to convert among multiple types of speech, as the prompt word, so as to reduce the recognition difficulty and thus obtain the converted text corresponding to multiple types of speech.

[0023] Therefore, by using a large language model with context learning and multi-type language learning capabilities, the acoustic features of multi-type speech that have a representational function, as well as the first sub-text that has a cueing function, can be analyzed to obtain the converted text corresponding to multi-type speech. This improves the large language model's ability to recognize the relatively sparse first-type sub-speech, and thus improves the accuracy of the large language model in converting multi-type speech. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A schematic diagram of an intelligent driving scenario provided in an embodiment of this application;

[0026] Figure 2 A schematic diagram illustrating an intelligent customer service scenario provided in an embodiment of this application;

[0027] Figure 3A schematic diagram of a digital human live streaming scenario provided in an embodiment of this application;

[0028] Figure 4 This is a schematic diagram illustrating an application scenario of a speech-to-text method provided in an embodiment of this application.

[0029] Figure 5 A flowchart illustrating the speech-to-text method provided in an embodiment of this application;

[0030] Figure 6 This is a schematic diagram illustrating the application of a speech-to-text method provided in an embodiment of this application;

[0031] Figure 7 A training diagram illustrating a speech-to-text method provided in an embodiment of this application;

[0032] Figure 8 This is an application diagram illustrating another speech-to-text method provided in an embodiment of this application;

[0033] Figure 9 A training diagram illustrating another speech-to-text method provided in an embodiment of this application;

[0034] Figure 10 This is an application diagram illustrating yet another speech-to-text method provided in an embodiment of this application;

[0035] Figure 11 A training diagram illustrating yet another speech-to-text method provided in this application embodiment;

[0036] Figure 12 A schematic diagram of the structure of a speech-to-text device provided in an embodiment of this application;

[0037] Figure 13 This application provides a schematic diagram of the structure of a server according to an embodiment of the present application.

[0038] Figure 14 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0039] The embodiments of this application will now be described with reference to the accompanying drawings.

[0040] In related technologies, for speech that includes only a single type (hereinafter referred to as single-type speech), such as all-Chinese speech or all-English speech, a model with speech-to-text functionality is used to convert single-type speech. Generally, this model utilizes its contextual understanding capability to continuously predict the text corresponding to the next sub-speech in the speech, thereby achieving the conversion of speech into text. Here, a sub-speech refers to each small segment of speech after it has been divided.

[0041] If this model is used to convert multiple types of speech, the resulting text may all belong to the same type, or produce similar-sounding but incorrect expressions, leading to low conversion accuracy. For example, using the sentence "I have a meeting today, I might be home late" as an example, the model might result in the text "I have a meeting today, I might be home late" or "I have a boat trip today, I might be home late" instead of the original sentence.

[0042] Furthermore, the imbalance in the number of different types of sub-speech within multiple types leads to data sparsity for a particular type. For example, the first sub-speech belonging to the first type is far less than the second sub-speech belonging to the second type, resulting in sparse data for the first sub-speech belonging to the first type. Consequently, the trained model exhibits poor generalization ability and has a low accuracy rate in recognizing the sub-speech of the sparse data type during the conversion process, leading to a low accuracy rate in the conversion of multiple types of speech.

[0043] Based on this, this application provides a speech-to-text method. By using a large language model with context learning and multi-type language learning capabilities, the acoustic features of multi-type speech with representational function and the first sub-text with prompting function are analyzed to obtain the converted text corresponding to multi-type speech, thereby improving the recognition ability of the large language model for the relatively sparse first type of sub-speech, and thus improving the accuracy of the large language model for multi-type speech conversion.

[0044] The speech-to-text method provided in this application can be applied to computer devices with speech-to-text capabilities, such as terminal devices and servers.

[0045] Specifically, terminal devices can be desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Smart in-vehicle devices can be in-vehicle navigation terminals and in-vehicle computers, etc. Portable wearable devices can be smartwatches, smart bracelets, head-mounted devices, etc., but are not limited to these.

[0046] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal devices and servers can be connected directly or indirectly via wired or wireless communication; this application does not impose any restrictions on this.

[0047] The speech-to-text method provided in this application can be applied to various scenarios, including but not limited to the functions of converting user (device user) speech into text for display in smart devices such as smart homes, smart cockpits, smart speakers, in-vehicle devices, and smartphones; meeting content transcription functions; automatic reply functions for smart customer service; automatic video subtitle generation; and digital human live streaming. Three scenarios are given below as examples.

[0048] Scenario 1: The function of converting the user's voice into text and displaying it in the vehicle's equipment.

[0049] See Figure 1 This figure is a schematic diagram of an intelligent driving scenario provided in an embodiment of this application. Figure 1 In the middle, the display screen of the vehicle-mounted equipment is like Figure 1 As shown in 101.

[0050] Users can issue voice commands to the in-vehicle device, such as the voice command "Please turn on music". This voice command belongs to multiple voice types, where "music" belongs to the first type and "Please turn on" belongs to the second type. The in-vehicle device can execute the speech-to-text method provided in this application, thereby, after acquiring multiple voice types, the in-vehicle device extracts features from the multiple voice types to obtain their acoustic features, performs type recognition on the multiple voice types, and obtains the first sub-text corresponding to "music".

[0051] A pre-trained large language model is installed in the in-vehicle device. Based on the acoustic features of the various speech types obtained above and the first sub-text, the large language model analyzes the data to obtain the converted text corresponding to each speech type, namely, the text form "Please turn on music". The in-vehicle device can display the text form "Please turn on music" on its display screen 101 so that the user can confirm whether the voice command received by the in-vehicle device is correct. In addition, the in-vehicle device can also play music based on this voice command, such as displaying and playing the voice "Okay, music has been turned on for you".

[0052] The speech-to-text method provided in this application enables in-vehicle devices to convert user-issued voice commands that can be represented as various types of speech with high accuracy, thereby improving the user experience when using in-vehicle devices.

[0053] Scenario 2: Automatic reply function in intelligent customer service.

[0054] See Figure 2 This figure is a schematic diagram of an intelligent customer service scenario provided in an embodiment of this application. Figure 2In China, users log in to platforms that offer functions such as shopping via smartphones and other terminal devices. These platforms typically have intelligent customer service functions that automatically answer users' questions, thereby reducing the workload of human customer service representatives.

[0055] During the process of asking questions, users can ask questions via voice (hereinafter referred to as voice questions), such as the voice form "What is the shelf life of ABC (the abbreviation for the product)?". Voice questions are classified into multiple voice types, such as "ABC" belonging to the first type and "What is the shelf life of..." belonging to the second type. The intelligent customer service can execute the voice-to-text method provided in the embodiments of this application. After acquiring multiple types of voice, the intelligent customer service performs feature extraction on the multiple types of voice to obtain the acoustic features of the multiple types of voice, performs type recognition on the multiple types of voice, and obtains the first sub-text corresponding to "ABC".

[0056] The intelligent customer service function can call upon a large language model to analyze the acoustic features of the aforementioned multiple speech types and the first sub-text, obtaining the converted text corresponding to each speech type, namely, the text form "What is the shelf life of ABC?". The intelligent customer service can display this text form, "What is the shelf life of ABC?", on the display screen 202 of the terminal device 201 equipped with the intelligent customer service, so that the user can confirm whether the intelligent customer service has correctly understood their question. Furthermore, the intelligent customer service can also provide a response based on this question.

[0057] The speech-to-text method provided in this application enables intelligent customer service to convert user-generated questions into various speech types with high accuracy, thereby improving the user experience.

[0058] Scenario 3: Digital Human Live Streaming Function.

[0059] See Figure 3 This figure is a schematic diagram of a digital human live streaming scenario provided in an embodiment of this application. Figure 3 In this context, digital humans can be used as virtual anchors for live streaming. Taking Chinese-language e-commerce live streaming as an example, the digital human's script can be displayed as subtitles in the live stream. For instance, if the digital human says, "This is newly produced ABC, which can be used for entertainment activities such as playing games and watching live streams," it can be displayed as subtitles in the live stream, as shown in 301. Furthermore, the product names read by the digital human can be converted into text and displayed on the live stream screen, as shown in 302.

[0060] When the digital human reads product names or live-stream scripts in both Chinese and English, the speech-to-text method provided in this application can be used to acquire multiple speech types, extract their acoustic features, and then identify the first sub-text corresponding to the Chinese speech mixed in with the English speech. Based on the acoustic features of the multiple speech types and the first sub-text, a large language model is used to analyze the data and obtain the converted text corresponding to each speech type. This converted text is then displayed on the live-stream screen for viewers to understand.

[0061] The speech-to-text method provided in this application can convert various types of speech emitted by digital humans into converted text and display it on the live broadcast screen during digital human live streaming. Moreover, the conversion accuracy is high, thereby improving the user's experience of watching digital human live streams.

[0062] It should be noted that the above application scenarios are merely examples. The speech-to-text method provided in this embodiment can also be applied to other scenarios, which are not limited here.

[0063] To facilitate understanding of the speech-to-text method provided in this application embodiment, the following describes the application scenario of the speech-to-text method by taking the server as the execution subject of the speech-to-text method and automatic video subtitle generation as an application scenario.

[0064] See Figure 4 This figure is a schematic diagram illustrating an application scenario of a speech-to-text method provided in an embodiment of this application. For example... Figure 4 As shown, this application scenario includes a terminal device 410 and a server 420, which can communicate with each other via a communication network. The communication network uses standard communication technologies and / or protocols, typically the Internet, but can also be any network, including but not limited to Bluetooth, a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), mobile, private networks, or any combination of virtual private networks. In some embodiments, customized or dedicated data communication technologies may be used to replace or supplement the aforementioned data communication technologies.

[0065] Terminal device 410 has a client installed, which is used to play videos. During video playback, terminal device 410 emits audio, which typically has multiple types, such as Chinese audio mixed with English audio. Terminal device 410 sends this audio to server 420.

[0066] Server 420 corresponds to the client's server and is used to generate corresponding subtitles for the video and send the subtitles to terminal device 410, so that the subtitles are displayed on terminal device 410. Server 420 acquires multiple types of audio, which include a first sub-audio belonging to a first type (such as an audio segment corresponding to English) and a second sub-audio belonging to a second type (such as an audio segment corresponding to Chinese). Among them, the first sub-audio belonging to the first type has fewer first sub-audio belonging to the second type than the second sub-audio belonging to the second type, that is, the first sub-audio belonging to the first type is sparse data in the multiple types of audio, and its conversion is more difficult.

[0067] Server 420 performs feature extraction on multiple speech types to obtain acoustic features of each type. These acoustic features reflect the characteristics of the speech, thus helping the large language model understand the various speech types. Type recognition is then performed on the multiple speech types to obtain the first sub-text. This first sub-text not only belongs to the first type but also corresponds to the first sub-speech of the first type among the multiple speech types. Using this as a cue word helps the large language model understand the first sub-speech of the first type, which is more difficult to convert.

[0068] Server 420 analyzes the acoustic features and first subtext of multiple speech types using a large language model. This large language model possesses multi-type language learning capabilities, enabling it to understand the meaning of each type of sub-speech based on its acoustic features. Furthermore, the large language model has context learning capabilities. In predicting the text corresponding to each sub-speech, it matches the first subtext of the first type (which is more difficult to convert) as a prompt word to reduce recognition difficulty, thus obtaining the converted text corresponding to the multiple speech types—i.e., the subtitles for the audio. Server 420 sends these subtitles to terminal device 410, allowing terminal device 410 to display the corresponding subtitles while playing the video, thereby achieving real-time or pre-conversion of the letters.

[0069] Therefore, by using a large language model with context learning and multi-type language learning capabilities, the acoustic features of multi-type speech that have a representational function, as well as the first sub-text that has a cueing function, can be analyzed to obtain the converted text corresponding to multi-type speech. This improves the large language model's ability to recognize the relatively sparse first-type sub-speech, and thus improves the accuracy of the large language model in converting multi-type speech.

[0070] The speech-to-text method provided in this application embodiment can be executed by a server. However, in other embodiments of this application, the terminal device may also have similar functions to the server to execute the speech-to-text method provided in this application embodiment. For example, the terminal device 410 may convert the speech in a video while playing the video to obtain converted text, and then display the converted text as subtitles in the video. Alternatively, the terminal device and the server may jointly execute the speech-to-text method provided in this application embodiment; this embodiment does not limit this approach.

[0071] The following describes in detail a speech-to-text method provided in this application through method embodiments.

[0072] See Figure 5 This figure is a flowchart illustrating the speech-to-text method provided in an embodiment of this application. For ease of description, the following embodiments will still use a server as the execution subject of the speech-to-text method. Figure 5 As shown, the speech-to-text method includes S501-S504.

[0073] This application scenario includes servers, which can be standalone servers used for training models (such as acoustic feature extraction models for feature extraction, classification models for recognition, and speech-to-text models for conversion). After the large language model has been trained (such as full training or fine-tuning), the trained large language model can be deployed on the server or terminal device corresponding to the product to provide speech-to-text services; alternatively, it can be a server that has already deployed and trained a model to provide speech-to-text services. The following explanation uses a server to provide speech-to-text services as an example.

[0074] As mentioned above, multi-type speech encompasses multiple types, and the number of sub-speech words corresponding to these types is uneven. In related technologies, models are generally trained to convert only a single type of speech. The purpose of this training is typically to extract embedding features representing multiple speech types and then learn these features to achieve conversion. However, when using such models to convert multi-type speech, their ability to convert sub-speech words belonging to another type within the multi-type speech is insufficient.

[0075] Alternatively, a model could be trained using a large amount of training data on various speech types to suit those types. However, currently, there is a limited amount of open-source data on various speech types, and manual annotation is very costly, making it difficult to acquire such data. Furthermore, in constructing various speech types, the position of the first sub-speech of the first type can vary, potentially resulting in different overall speech patterns. This necessitates even more training data, further increasing the difficulty of acquiring it. Insufficient training data can lead to lower generalization and accuracy in the subsequently trained model.

[0076] In other words, in the process of converting multiple types of speech, the conversion of sparser types of sub-speech is more difficult. For example, the conversion of the first sub-speech belonging to the first type mentioned above is more difficult, which leads to a higher difficulty in converting multiple types of speech.

[0077] Based on this, the embodiments of this application no longer train the model to extract the embedding features of multiple types of speech, but instead perform conversion based on acoustic features. During the conversion process, the text corresponding to the first sub-speech of the first type, i.e., the first sub-text, is used as a prompt word to improve the conversion capability of the large language model. The specific training process will be explained later. The application process will be explained in detail below.

[0078] S501: Acquire multiple types of voice.

[0079] Multi-type speech includes a first sub-speech belonging to a first type and a second sub-speech belonging to a second type. The first and second types are distinct categories. In the multiple speech segments obtained by dividing multi-type speech, each segment is a sub-speech. For example, multi-type speech can be divided into segments by frames (e.g., 10-30 milliseconds) or other durations, with each segment being a sub-speech. Alternatively, multi-type speech can be divided by bit depth, resulting in multiple sub-speech. Bit depth, also known as bit count, determines the precision and dynamic range of speech. A higher bit count allows for a more accurate representation of speech amplitude, providing richer sound detail and a wider dynamic range. The first sub-speech is the sub-speech belonging to the first type within the multi-type speech, and the second sub-speech is the sub-speech belonging to the second type within the multi-type speech.

[0080] The embodiments of this application do not limit the type. For example, the type is used to describe the language type (such as Chinese, English, etc.), the degree of information compression (such as whether it is an abbreviation, shorthand, etc.), or the importance of information (such as whether it is a keyword, etc.). For example, when the first type is English and the second type is Chinese, the corresponding multi-type speech is Chinese mixed with English. As another example, when the first type is an abbreviation for normal language and the second type is normal language, the corresponding multi-type speech is normal language mixed with abbreviation, etc.

[0081] The first quantity refers to the number of first sub-speech items in a multi-type speech pattern, and the second quantity refers to the number of second sub-speech items in the same pattern. In a multi-type speech pattern, the first quantity is less than the second quantity. For example, the proportion of first sub-speech items belonging to the first type in the multi-type speech pattern is less than or significantly less than the proportion of second sub-speech items belonging to the second type. For instance, if the duration of a multi-type speech pattern is 10 seconds, the duration of the first sub-speech item belonging to the first type is 1 second, and the duration of the second sub-speech item belonging to the second type is 9 seconds, then the proportion of the first sub-speech item belonging to the first type in the multi-type speech pattern is 0.1, and the proportion of the second sub-speech item belonging to the second type in the multi-type speech pattern is 0.9. Furthermore, if the multi-type speech pattern is divided into multiple speech frames, the number of speech frames corresponding to the first sub-speech item belonging to the first type is less than the number of speech frames corresponding to the second sub-speech item belonging to the second type; that is, the first quantity is less than the second quantity.

[0082] All data collected in this application (such as various types of voice) is collected with the consent and authorization of the data subject (such as user, organization or enterprise), and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0083] S502: Extract features from multiple speech types to obtain the acoustic features of multiple speech types.

[0084] Feature extraction is performed on multiple speech types to obtain acoustic features that characterize these speech types. These acoustic features are used to describe the properties of audio in different aspects, such as energy, pitch, voiced sounds, fricatives, pitch, intensity, and frequency.

[0085] Acoustic features of multi-type speech are features extracted directly from multi-type speech and can reflect the characteristics of multi-type speech. They include most of the key information in multi-type speech. Based on acoustic features, not only can the accuracy of subsequent conversion be improved, but the process of obtaining acoustic features is also relatively basic. Feature extraction can be performed directly based on unsupervised models such as Unsupervised Pre-Training for Speech Recognition (wav2vec). The unsupervised approach reduces the requirements for training data of multi-type speech, and even single-type speech can be used during the training process. This avoids the influence of sparse sub-speech on the conversion process and improves the robustness of speech-to-text conversion.

[0086] S503: Perform type recognition on multiple types of speech to obtain the first sub-text belonging to the first type.

[0087] Type recognition is performed on multiple types of speech to obtain the text corresponding to the first sub-speech belonging to the first type among the multiple types of speech, i.e., the first sub-text, and the first sub-text belongs to the first type.

[0088] The embodiments of this application do not specifically limit the recognition method. For example, the type of each sub-speech in multiple types of speech can be identified first to obtain the type of each sub-speech. Then, the first sub-speech belonging to the first type can be converted to obtain the first sub-text belonging to the first type. Alternatively, a model that can only recognize speech belonging to the first type can be trained, and then the type of multiple types of speech can be identified based on this model to obtain the first sub-text belonging to the first type. Specific examples will be given later, and will not be elaborated further here.

[0089] As mentioned above, the low accuracy of multi-type speech conversion is generally due to insufficient conversion capability for the first sub-speech belonging to the first type. By specifically identifying the first sub-speech belonging to the first type in multi-type speech, the accuracy of converting sparse sub-speech is improved. This sub-text can then be used as a prompt word to assist the large language model in subsequent conversions, thus improving the accuracy of multi-type speech conversion. Furthermore, using the first sub-text as a prompt word avoids the problem of inconsistent training data quality and insufficient diversity caused by data augmentation during training, further improving the accuracy and robustness of speech-to-text conversion. The training process will be explained in detail later and will not be repeated here.

[0090] S504: Based on the acoustic features of multiple speech types and the first sub-text, the converted text corresponding to the multiple speech types is obtained through analysis using a large language model.

[0091] The converted text is the text obtained by converting speech. The converted text corresponding to multiple speech types is the text obtained by converting multiple speech types. The converted text corresponding to multiple speech types includes the subtext corresponding to the first sub-speech of the first type and the subtext corresponding to the second sub-speech of the second type.

[0092] Large Language Models (LLMs) are deep learning-based natural language processing models trained on large-scale data. They possess the ability to understand and generate language, such as the ability to understand multiple types of languages ​​(referred to as multi-type language learning ability) and contextual understanding. As a possible implementation method, LLMs can reduce the difficulty of implementing speech-to-text functionality for pre-trained models.

[0093] Taking the context learning and multi-type language learning capabilities of large language models as an example, the multi-type language learning capabilities of large language models can be used to understand the meaning of each type of sub-speech in multi-type speech based on the acoustic features of multi-type speech. That is, large language models can understand the first sub-speech belonging to the first type and the second sub-speech belonging to the second type. For example, large language models can understand Chinese and English, and thus can understand Chinese speech mixed with English.

[0094] By leveraging the contextual learning capabilities of large language models, the text corresponding to the next sub-speech can be predicted based on the previous sub-speech, thus enabling the understanding of the semantics of multiple speech types. Since multiple speech types contain various sub-speech categories, two sub-speechs with adjacent pronunciation sequences may belong to different categories. For example, in multiple speech types, the (i-1)th sub-speech may belong to the second category, while the ith sub-speech may belong to the first category. Therefore, in addition to the high difficulty in converting the first sub-speech belonging to the first category, predicting sub-speech across categories also leads to significant conversion challenges.

[0095] Based on this, in order to reduce the difficulty of converting the first sub-speech belonging to the first type, the embodiments of this application add a first sub-text as a prompt word, that is, the text corresponding to the first sub-speech belonging to the first type among multiple types of speech. Thus, when the large language model converts sub-speech, it uses the first sub-text to guide the conversion. For example, the similarity between the sub-speech and the prompt word is calculated. If the similarity is high, the sub-speech is more likely to be the first sub-text, and thus the sub-speech can be converted into the first sub-text. This means that the reliability of the first sub-text is higher than the reliability of the result predicted based on the previous sub-speech, avoiding the problem of low accuracy in predicting the first sub-speech belonging to the first type caused by cross-type and sparsity, improving the accuracy of converting the first sub-speech belonging to the first type, and thus improving the accuracy of converting multiple types of speech.

[0096] As can be seen from the above technical solution, multiple types of speech are acquired, including first sub-speech belonging to the first type and second sub-speech belonging to the second type. The number of first sub-speech belonging to the first type is less than the number of second sub-speech belonging to the second type, meaning that the first sub-speech belonging to the first type is sparse data in the multiple speech types, making its conversion more difficult. Feature extraction is performed on the multiple speech types to obtain their acoustic features. These acoustic features reflect the characteristics of the speech, thus helping the large language model understand the multiple speech types. Type recognition is performed on the multiple speech types to obtain the first sub-text. This first sub-text not only belongs to the first type but is also the text corresponding to the first sub-speech belonging to the first type in the multiple speech types. Using this as a cue word helps the large language model understand the more difficult-to-convert first sub-speech belonging to the first type. Finally, the large language model analyzes the acoustic features and the first sub-text of the multiple speech types. The large language model itself has the ability to learn multiple language types, and based on this ability, it can understand the meaning of each type of sub-speech in the multiple speech types based on the acoustic features. Furthermore, the large language model also has context learning capabilities. Based on this capability, in the process of predicting the text corresponding to each sub-speech, the large language model matches the first sub-text of the first sub-speech belonging to the first type, which is more difficult to convert among multiple types of speech, as the prompt word, so as to reduce the recognition difficulty and thus obtain the converted text corresponding to multiple types of speech.

[0097] Therefore, by using a large language model with context learning and multi-type language learning capabilities, the acoustic features of multi-type speech that have a representational function, as well as the first sub-text that has a cueing function, can be analyzed to obtain the converted text corresponding to multi-type speech. This improves the large language model's ability to recognize the relatively sparse first-type sub-speech, and thus improves the accuracy of the large language model in converting multi-type speech.

[0098] This application does not specifically limit the feature extraction method for obtaining acoustic features, nor does it limit the recognition method for obtaining the first sub-text. The speech-to-text process is explained below using three methods as examples.

[0099] Method 1 involves feature extraction using the first acoustic feature, followed by recognition using the first classification model and the speech-to-text model. See A1-A5 for details (not shown in the figure).

[0100] See Figure 6 The figure is a schematic diagram illustrating the application of a speech-to-text method provided in an embodiment of this application.

[0101] A1: Acquire multiple types of voice.

[0102] A2: The acoustic features of multiple speech types are obtained by extracting features from multiple speech types using the first acoustic feature extraction model.

[0103] The first acoustic feature extraction model is used to extract features from speech, obtaining the corresponding acoustic features. This model can be used to extract features from multiple types of speech, resulting in acoustic features for each type. The first acoustic feature extraction model can be the encoding part of wav2vec or the Whisper model, etc., and this application does not specifically limit its application to this.

[0104] A3: By using the first classification model to identify the type of multiple speech types, the first sub-speech belonging to the first type of multiple speech types is obtained.

[0105] The first classification model is used to identify speech belonging to the first type. By using the first classification model to perform type recognition on multiple types of speech, it is possible to obtain the sub-speech belonging to the first classification model among multiple types of speech.

[0106] As one possible implementation, the first classification model is also used to identify other types of speech. For example, if the first classification model is used not only to identify speech belonging to the first type but also to identify speech belonging to the second type, the first classification model can be used to perform type identification on multiple types of speech to obtain the type corresponding to each sub-speech in the multiple types of speech. Thus, based on the type corresponding to each sub-speech in the multiple types of speech, the first sub-speech belonging to the first type in the multiple types of speech can be obtained.

[0107] Therefore, compared to models that can only recognize a single type (such as type 1), models that can recognize all types of speech (such as type 1 and type 2) have higher recognition accuracy. This is because by learning the characteristics of multiple speech types, their generalization ability is higher, and their recognition accuracy is also higher. Furthermore, models that can only recognize a single type not only need type recognition ability but also the ability to recognize speech of other types (such as type 2), while models that can recognize multiple speech types only need type recognition ability, making them easier to train.

[0108] A4: The first sub-speech belonging to the first type is converted into the first sub-text by using a speech-to-text model.

[0109] Speech-to-text models are used to convert speech belonging to the first type into text. By converting the first sub-speech of the first type using this model, the corresponding text (the first sub-text) can be obtained, which also belongs to the first type. It should be noted that compared to large language models, speech-to-text models are more suitable for converting speech belonging to the first type. They are more targeted and have higher conversion accuracy, thus making the first sub-text more reliable as a prompt word, thereby improving the accuracy of the subsequent converted text obtained based on the prompt word.

[0110] A5: Based on the acoustic features of multiple speech types and the first sub-text, the converted text corresponding to the multiple speech types is obtained through analysis using a large language model.

[0111] Therefore, the acoustic features of multiple speech types are extracted by the first acoustic feature extraction model. The acoustic features of multiple speech types are obtained by the first classification model and the speech-to-text model. The first sub-text corresponding to the first sub-speech of the first type in the multiple speech types is obtained. Then, the acoustic features of the first sub-text and multiple speech types are analyzed by the large language model to obtain the converted text of multiple speech types.

[0112] This method requires fewer models, is easier to train, and trains quickly, making it convenient and efficient. Furthermore, each function corresponds to a specific model, making it more targeted and resulting in more accurate results from each model, thus improving conversion accuracy.

[0113] The training process of the first acoustic feature extraction model, the first classification model, and the speech-to-text model is explained below. See B1-B6 for details (not shown in the figures).

[0114] See Figure 7 The figure is a training diagram of a speech-to-text method provided in an embodiment of this application.

[0115] B1: Obtain the first sample speech.

[0116] The first sample speech is the sample speech used for training. Compared to multi-type speech, it also includes the required labels. Specifically, the first sample speech includes sub-sample speech belonging to the first type and sub-sample speech belonging to the second type. Sub-sample speech is a speech segment within the sample speech. In the first sample speech, the number of sub-sample speech belonging to the first type is less than the number of sub-sample speech belonging to the second type, that is, the sub-sample speech belonging to the first type is relatively sparse in the first sample speech.

[0117] The first sample speech includes a first label and a second label. The first label is used to indicate which sub-sample speech in the first sample speech belongs to the first type. That is, the first label is used to identify which sub-sample speech in the first sample speech belongs to the first type. The second label is the real text corresponding to the first sample speech, that is, the correct text corresponding to the first sample speech after it has been converted into text.

[0118] B2: The acoustic features of the first sample speech are obtained by extracting features from the first sample speech using the initial first acoustic feature extraction model.

[0119] Compared to the first acoustic feature extraction model, the initial first acoustic feature extraction model has not yet been trained. That is, the accuracy of the initial first acoustic feature extraction model is generally lower than that of the first acoustic feature extraction model. The two models have the same structure and function. Therefore, the initial first acoustic feature extraction model can be used to extract features from the first sample speech and obtain the acoustic features of the first sample speech.

[0120] B3: The first sample speech is identified by the initial first classification model to obtain the subsample speech belonging to the first type in the first sample speech.

[0121] Compared to the first classification model, the initial first classification model has not yet been fully trained, meaning its accuracy is generally lower than that of the first classification model. Both models have the same structure and function, thus the initial first classification model can be used to identify the type of the first sample speech, obtaining sub-sample speech belonging to the first type. Sub-sample speech refers to speech segments within the sample speech.

[0122] B4: The speech of the first type of sub-sample is converted into the first sample text by using a speech-to-text model.

[0123] The first sample subtext is the text corresponding to the first type of sub-sample speech in the first sample speech, and the first sample subtext belongs to the first type.

[0124] The speech-to-text model is a pre-trained model, meaning that a pre-trained model for converting speech of type 1 into text can be used directly without the need to train the speech-to-text model again, thus reducing the training difficulty.

[0125] It should be noted that the speech-to-text model is used to convert speech belonging to the first type into text. That is, the speech-to-text model only converts speech of a single type; it does not handle speech of multiple types (such as multiple types of speech, the first sample speech, etc.). The conversion difficulty is lower, and it is more targeted, resulting in higher accuracy of the obtained first sample sub-text. In other words, compared to directly converting multiple types of speech into text and then obtaining the sub-text corresponding to the first sub-speech of the first type from the converted text, the speech-to-text model using methods B3 and B4 is more accurate.

[0126] B5: Based on the acoustic features of the first sample speech and the first sample subtext, the converted text corresponding to the first sample speech is obtained through analysis using a large language model.

[0127] The large language model is a pre-trained model, meaning that the already trained model can be used directly without further training, which reduces the training difficulty and does not affect the accuracy of subsequent conversions.

[0128] B6: Based on the differences between the first label and the first sample speech belonging to the first type, and the differences between the second label and the converted text corresponding to the first sample speech, adjust the model parameters of the initial first acoustic feature extraction model and the model parameters of the initial first classification model to obtain the first acoustic feature extraction model and the first classification model.

[0129] Because the initial first acoustic feature extraction model has low accuracy in feature extraction, and the initial first classification model has low accuracy in recognition, the accuracy of the first sample speech belonging to the first type in the first sample speech obtained based on the initial first classification model, and the accuracy of the converted text corresponding to the first sample speech obtained based on the initial first classification model and the initial first acoustic feature extraction model are low. Therefore, based on the differences between the first label and the first sample speech belonging to the first type, and the differences between the second label and the converted text corresponding to the first sample speech, while keeping the model parameters of the speech-to-text model and the large language model unchanged, the model parameters of the initial first acoustic feature extraction model and the initial first classification model are adjusted. Thus, the first acoustic feature extraction model is obtained based on the initial first acoustic feature extraction model, and the first classification model is obtained based on the initial first classification model.

[0130] Therefore, since the speech-to-text model only converts sub-speech including the first type, and the large language model has the ability to learn multiple language types, the speech-to-text model and the large language model are not affected by the inclusion of multiple speech types. Thus, pre-trained speech-to-text models and large language models can be used directly. During training, the model parameters of the speech-to-text model and the large language model can be frozen, preventing updates during training. The first acoustic feature extraction model and the first classification model, which are affected by multiple speech types, are then trained. Specifically, the model parameters of the initial first acoustic feature extraction model and the initial first classification model are adjusted to obtain the corresponding first acoustic feature extraction model and first classification model.

[0131] This approach reduces training difficulty while enabling joint training of four models. This makes the first acoustic feature extraction model and the first classification model more suitable for converting various types of speech, thereby improving the accuracy of speech-to-text conversion. Furthermore, the training process of the first acoustic feature extraction model is essentially unsupervised training, reducing the training data and costs required for its development.

[0132] As one possible implementation, as mentioned above, the first classification model can identify not only the first type but also the second type. Therefore, adaptive adjustments are needed during training to make the first classification model more accurate.

[0133] Since the first classification model and the initial first classification model are basically the same in structure and function, the initial first classification model can also identify the first type and the second type. At this time, the first label corresponding to the first sample speech is not only used to indicate the sub-sample speech belonging to the first type in the first sample speech, but also to indicate the sub-sample speech belonging to the second type in the first sample speech. That is, the first label is used to indicate the type of each sub-sample speech in the first sample speech.

[0134] Based on this, the aforementioned step B3, which involves identifying the first sample speech using the initial first classification model to obtain the sub-sample speech belonging to the first type in the first sample speech, can be fine-tuned so that the first sample speech can be identified using the initial first classification model to obtain the type corresponding to each sub-sample speech in the first sample speech.

[0135] Correspondingly, step B6 above, which involves adjusting the model parameters of the initial first acoustic feature extraction model and the initial first classification model based on the differences between the first label and the sub-sample speech belonging to the first type in the first sample speech, and the differences between the second label and the converted text corresponding to the first sample speech, can be fine-tuned as follows: based on the differences between the types corresponding to each sub-sample speech in the first label and the first sample speech, and the differences between the second label and the converted text corresponding to the first sample speech, the model parameters of the initial first acoustic feature extraction model and the initial first classification model are adjusted, thus obtaining the first acoustic feature extraction model and the first classification model.

[0136] Therefore, compared to the B1-B6 method, which only recognizes a single type of speech, the first classification model that can recognize all types of speech in multiple types has higher recognition accuracy. This is because the first classification model that can recognize multiple types of speech has higher generalization ability and recognition accuracy due to having learned from multiple speech types, thus resulting in higher accuracy for subsequent speech-to-text conversion. Furthermore, the first classification model that can only recognize a single type needs not only type recognition ability but also the ability to recognize only that specific type, while the first classification model that can recognize multiple types of speech only needs type recognition ability, making its training more difficult.

[0137] Method 2 involves feature extraction using the second acoustic feature, followed by identification using a feature transformation model and similarity calculation. See C1-C7 for details (not shown in the figure).

[0138] See Figure 8 The figure is a schematic diagram illustrating the application of another speech-to-text method provided in an embodiment of this application.

[0139] C1: Acquire multiple types of speech.

[0140] C2: The second acoustic feature extraction model is used to extract features from multiple types of speech to obtain the acoustic features of multiple types of speech.

[0141] The second acoustic feature extraction model is used to extract features from speech, obtaining the corresponding acoustic features. This second acoustic feature extraction model can be used to extract features from multiple types of speech, thus obtaining the acoustic features of various speech types. The second acoustic feature extraction model can be the same as the first acoustic feature extraction model, or it can be a different model; this application does not specifically limit this.

[0142] C3: The acoustic features of multiple speech types are transformed through a feature transformation model to obtain the initial embedding features.

[0143] Feature transformation models are used to convert acoustic features describing speech into embedding features describing text. This allows for the transformation of acoustic features from multiple speech types to obtain initial embedding features. In other words, the feature transformation model converts acoustic features describing the characteristics of multiple speech types from a speech dimension into initial embedding features describing the characteristics of multiple speech types from a text dimension. The candidate embedding features used for subsequent similarity calculations also describe the characteristics of candidate sub-texts from a text dimension, thereby improving the accuracy of similarity calculations and ultimately improving the accuracy of subsequent speech-to-text conversion.

[0144] Compared to directly converting multiple speech types into embedded features that describe the characteristics of multiple speech types from the text dimension using a single model, first converting multiple speech types into acoustic features that describe the characteristics of multiple speech types from the speech dimension, and then converting them into initial embedded features that describe the characteristics of multiple speech types from the text dimension, can not only improve the accuracy of the initial embedded features, but also reduce the dependence on the second acoustic feature extraction model and feature transformation model, thereby further improving the accuracy of the initial embedded features.

[0145] C4: Obtain candidate embedding features of candidate subtexts.

[0146] Candidate subtexts are subtexts belonging to the first type, and candidate embedding features are the embedding features corresponding to the candidate subtexts. Both candidate embedding features and initial embedding features are features that describe data from the text dimension. They belong to the same feature space and are more similar in expression, so as to improve the accuracy of their similarity calculation.

[0147] This application does not specifically limit the method of obtaining candidate sub-texts. For example, the sub-text corresponding to the first sub-speech of the first type that appears frequently in multiple speech scenarios is determined as candidate sub-text.

[0148] C5: Based on the similarity between the initial embedding features and each candidate embedding feature, the target candidate subtext is obtained from multiple candidate subtexts.

[0149] The similarity between the initial embedding feature and each candidate embedding feature is calculated, thereby identifying the candidate embedding feature whose similarity with the initial embedding feature satisfies the first similarity condition from multiple candidate embedding features. The candidate subtext corresponding to the candidate embedding feature that satisfies the first similarity condition is determined as the target candidate subtext. The target candidate subtext can be one or multiple, and this application does not make a specific limitation on this.

[0150] The embodiments of this application do not specifically limit the first similarity condition. For example, the first similarity condition is the top K similarity rankings, that is, the candidate subtexts corresponding to the top K candidate embedding features in terms of similarity with the initial embedding features are determined as the target candidate subtexts. Alternatively, the first similarity condition is the highest similarity, that is, the candidate embedding feature corresponding to the highest similarity among the similarities with the initial embedding features is determined, and the candidate subtext corresponding to that candidate embedding feature is determined as the target candidate subtext, and so on.

[0151] C6: Obtain the first subtext based on the target candidate subtext.

[0152] The similarity between the candidate embedding features of the target candidate subtext and the initial embedding features obtained based on multiple speech types is high. Therefore, the target candidate subtext is likely to be the text corresponding to the first subtext of the first type among multiple speech types. Thus, the target candidate subtext is used as the first subtext to guide the large language model to perform the conversion, thereby improving the accuracy of the large language model conversion.

[0153] The embodiments of this application do not specifically limit the method of obtaining the first sub-text, such as concatenating multiple target candidate sub-texts to obtain the first sub-text.

[0154] C7: Based on the acoustic features of multiple speech types and the first sub-text, the converted text corresponding to the multiple speech types is obtained through analysis using a large language model.

[0155] Therefore, a second acoustic feature extraction model is used to extract features from multiple speech types, resulting in acoustic features of each speech type. A feature transformation model is then used to convert these acoustic features into initial embedding features more suitable for similarity calculation. This ensures that both the initial and candidate embedding features describe data characteristics from a textual perspective, improving the accuracy of similarity calculation. Candidate sub-texts that are highly similar to the initial embedding features are obtained through similarity calculation and used as the first sub-text. Finally, a large language model is used to analyze the acoustic features of the first sub-text and the multiple speech types, resulting in the transformed text of the multiple speech types.

[0156] Compared to the first method mentioned above, this method requires fewer models, trains faster, and is more convenient and efficient. Furthermore, each function corresponds to a separate model, making it more targeted and resulting in more accurate results from each model, thus improving conversion accuracy. In addition, compared to the first method which uses a combination of a first classification model and a speech-to-text model to obtain the first sub-text, this method uses a feature conversion model and similarity calculation to obtain the first sub-text, reducing reliance on the accuracy of the first classification model. This not only improves conversion accuracy but also reduces the requirements for training data and training accuracy.

[0157] The training process of the second acoustic feature extraction model and the feature transformation model is described below. See D1-D7 for details (not shown in the figures).

[0158] See Figure 9 The figure is a training diagram of another speech-to-text method provided in an embodiment of this application.

[0159] D1: Obtain the second sample speech.

[0160] The second sample speech is used for training and, compared to multi-type speech, it also includes the required labels. Specifically, the second sample speech includes sub-sample speech belonging to the first type and sub-sample speech belonging to the second type. In the second sample speech, the number of sub-sample speech belonging to the first type is less than the number of sub-sample speech belonging to the second type, that is, the sub-sample speech belonging to the first type is relatively sparse in the second sample speech.

[0161] The second sample speech includes a third label, where the third label is the corresponding real text of the second sample speech, that is, the correct text corresponding to the second sample speech after it has been converted into text. As one possible implementation, the second sample speech may also include a fourth label, where the fourth label is the corresponding real candidate subtext of the second sample speech. The real candidate subtext refers to the subtext corresponding to the first subtext of the second sample speech that belongs to the first type.

[0162] As one possible implementation, the second sample speech can be the same as the first sample speech, the difference being that the second sample speech includes a third label, while the first sample speech includes a first label and a second label.

[0163] D2: Extract features from the second sample speech using the initial second acoustic feature extraction model to obtain the acoustic features of the second sample speech.

[0164] Compared to the second acoustic feature extraction model, the initial second acoustic feature extraction model has not yet been trained. That is, the accuracy of the initial second acoustic feature extraction model is generally lower than that of the second acoustic feature extraction model. The two models have the same structure and function, so the initial second acoustic feature extraction model can be used to extract features from the second sample speech to obtain the acoustic features of the second sample speech.

[0165] D3: The acoustic features belonging to the second sample speech are transformed by the initial feature transformation model to obtain the initial sample embedding features of the second sample speech.

[0166] Compared to the feature transformation model, the initial feature transformation model is not yet fully trained. This means the accuracy of the initial feature transformation model is generally lower than that of the standard feature transformation model. Both models have the same structure and function, allowing the acoustic features of the second sample speech to be transformed using the initial feature transformation model, thus obtaining the initial sample embedding features. The initial sample embedding features are the embedding features obtained after transforming the acoustic features of the sample speech; for example, the initial sample embedding features of the second sample speech are the embedding features obtained after transforming the acoustic features of the second sample speech.

[0167] D4: Based on the similarity between the initial sample embedding features of the second sample speech and each candidate embedding feature, sample candidate subtexts are obtained from multiple candidate subtexts.

[0168] The similarity between the initial sample embedding feature and each candidate embedding feature is calculated, thereby identifying from multiple candidate embedding features the candidate embedding feature whose similarity with the initial sample embedding feature of the second sample speech satisfies the first similarity condition. The candidate subtext corresponding to the candidate embedding feature that satisfies the first similarity condition is determined as the sample candidate subtext. The sample candidate subtext can be one or multiple, and this application does not make a specific limitation on this.

[0169] D5: Obtain the second sample subtext based on the sample candidate subtext.

[0170] The candidate embedding features of the sample candidate subtext have a high similarity with the initial sample embedding features obtained based on the second sample speech. Therefore, the sample candidate subtext is likely to be the text corresponding to the first subtext of the first type in the second sample speech. Thus, the sample candidate subtext is used as the second sample subtext to guide the large language model to perform the conversion, thereby improving the accuracy of the large language model conversion.

[0171] D6: Based on the acoustic features of the second sample speech and the second sample subtext, the converted text corresponding to the second sample speech is obtained through analysis using a large language model.

[0172] The large language model is a pre-trained model, meaning that the already trained model can be used directly without further training, which reduces the training difficulty and does not affect the accuracy of subsequent conversions.

[0173] D7: Based on the difference between the converted text corresponding to the third label and the second sample speech, adjust the model parameters of the initial second acoustic feature extraction model and the initial feature conversion model to obtain the second acoustic feature extraction model and the feature conversion model.

[0174] Based on the difference between the converted text corresponding to the third label and the second sample speech, the model parameters of the initial second acoustic feature extraction model are adjusted to obtain the second acoustic feature extraction model, and the model parameters of the initial feature conversion model are adjusted to obtain the feature conversion model.

[0175] As one possible implementation, if the second sample speech includes a fourth label, then step D7, which involves adjusting the model parameters of the initial second acoustic feature extraction model and the initial feature conversion model based on the difference between the third label and the converted text corresponding to the second sample speech, can be further refined into adjusting the model parameters of the initial second acoustic feature extraction model and the initial feature conversion model based on the difference between the third label and the converted text corresponding to the second sample speech, as well as the difference between the fourth label and the second sample sub-text, to obtain the second acoustic feature extraction model and the feature conversion model. This refines the loss function and improves the accuracy of the trained second acoustic feature extraction model and feature conversion model.

[0176] Therefore, since the large language model is unaffected by the inclusion of multiple speech types, its model parameters can be frozen during training, preventing updates during the training process. In contrast, training the second acoustic feature extraction model and feature transformation model, which are affected by multiple speech types, involves adjusting the initial model parameters of the second acoustic feature extraction model and the initial feature transformation model, thus obtaining the corresponding second acoustic feature extraction model and feature transformation model.

[0177] This approach enables joint training of three models while reducing training difficulty. This makes the second acoustic feature extraction model and the feature conversion model more suitable for converting various types of speech, thereby improving the accuracy of speech-to-text conversion. Furthermore, the training process of the second acoustic feature extraction model is essentially unsupervised training, reducing the training data and costs required. In addition, compared to the first sample speech, the second sample speech requires fewer labels, thus reducing the cost and difficulty of obtaining training data.

[0178] Method 3 involves feature extraction using the second acoustic feature, followed by identification using a second classification model, feature transformation model, prediction model, and similarity calculation. See E1-E11 for details (not shown in the figure).

[0179] See Figure 10 This figure is a schematic diagram illustrating the application of another speech-to-text method provided in the embodiments of this application.

[0180] E1: Acquire multiple types of voice.

[0181] E2: The second acoustic feature extraction model is used to extract features from multiple types of speech to obtain the acoustic features of multiple types of speech.

[0182] E3: By using a second classification model to identify the type of speech in multiple speech types, the position of the first sub-speech belonging to the first type is obtained in the multiple speech types.

[0183] The second classification model is used to identify speech belonging to the first type. This allows for type identification of multiple speech types, obtaining the position of the first sub-speech belonging to the first type within the multiple speech types. For example, if multiple speech types can be divided into 5 sub-speech units, arranged in speech order as [A, B, B, B, A], where A represents the first sub-speech belonging to the first type and B represents the second sub-speech belonging to the second type, then the position of the first sub-speech belonging to the first type within the multiple speech types, obtained through the second classification model, can be represented as [0, 1, 1, 1, 0]. Here, 0 indicates that the feature corresponding to that position belongs to the first type, and 1 indicates that the feature corresponding to that position does not belong to the first type.

[0184] As one possible implementation, the second classification model can also be used to identify the second type of speech. By using the second classification model to identify the types of multiple types of speech, the type of each sub-speech in the multiple types of speech can be obtained, thereby obtaining the position of the first sub-speech belonging to the first type in the multiple types of speech.

[0185] E4: The acoustic features of multiple speech types are transformed through a feature transformation model to obtain the initial embedding features.

[0186] E5: Based on the position of the first sub-speech belonging to the first type in the multi-type speech, the initial embedding features are transformed to obtain masked embedding features including semantic sub-features.

[0187] Semantic-free sub-features are sub-features in masked embedding features that lack actual semantic meaning. These semantic-free sub-features are generally obtained based on the position of the first sub-speech belonging to the first type within multiple speech types. In masked embedding features, the feature corresponding to the position of the first sub-speech belonging to the first type within multiple speech types has no actual semantic meaning. That is, the masked embedding feature only includes the sub-feature corresponding to the second sub-speech belonging to the second type. Positions where the sub-feature should correspond to the first sub-speech belonging to the first type are replaced with semantically meaningless content, such as 0.

[0188] For example, after obtaining the position of the first sub-speech belonging to the first type in the multi-type speech, which can be represented as [0, 1, 1, 1, 0], it is multiplied with the initial embedding feature to obtain the mask embedding feature, so that in the mask embedding feature, the feature corresponding to the position of the first sub-speech belonging to the first type in the multi-type speech is represented as 0.

[0189] As one possible implementation, the length of the multi-type speech (i.e., the number of sub-speech) can be controlled to be equal to the length of the initial embedding feature and the length representing the position of the first sub-speech belonging to the first type in the multi-type speech. This can improve the efficiency of the step of transforming the initial embedding feature according to the position of the first sub-speech belonging to the first type in the multi-type speech to obtain the mask embedding feature.

[0190] E6: Based on the mask embedding features, predictive embedding features are obtained by predicting the semantically unrelated sub-features in the mask embedding features using a prediction model.

[0191] In other words, based on the mask embedding features, a prediction model is used to predict the predicted embedding features. The prediction model is used to predict features belonging to the second type, such as predicting the (i+1)th sub-feature based on the i-th sub-feature in the mask embedding features through contextual understanding, thereby supplementing the semantics that the semantically null sub-features in the mask embedding features should express. That is, the semantically null sub-features in the mask embedding features obtain their intended meaning through prediction, thus obtaining the predicted embedding features.

[0192] As mentioned above, since the initial embedding features include not only the sub-features corresponding to the first sub-speech belonging to the first type, but also the sub-features corresponding to the second sub-speech belonging to the second type, if the feature transformation model has low accuracy in transforming multiple types, the initial embedding features may have low accuracy in representing multiple types of speech. To reduce the dependence on the feature transformation model, the sub-features corresponding to the first sub-speech belonging to the first type can be masked from the initial embedding features, and then prediction can be performed by a prediction model to obtain predicted embedding features that only include the second type, with higher reliability.

[0193] The embodiments of this application do not specifically limit the prediction model. For example, the prediction model can be a Bidirectional Encoder Representations from Transformers (BERT) model, a Robustly Optimized BERT Pretraining Approach (RoBERTa) model, a Text-To-Text Transfer Transformer (T5) model, etc. It should be noted that using a prediction model for text prediction has higher accuracy for masked embedding features described from the text dimension.

[0194] As a possible implementation, multiple sub-speeches in multi-type speech may correspond to the same semantics. For example, the pronunciation of "I" occupies 1 second, but when the multi-type speech is divided based on 20 milliseconds, multiple sub-speeches are obtained, so "I" corresponds to 3 sub-speeches, that is, 3 sub-speeches correspond to the same semantics. At this time, if the prediction continues based on the masked embedding features, unwanted content may be obtained. Based on this, the masked embedding features can be merged, that is, the sub-features corresponding to multiple sub-speeches with the same semantics in the masked embedding features are merged respectively to obtain the merged embedding features. It can be understood that the merging operation cannot fill semantics for the sub-features without semantics. Then, according to the merged embedding features, the prediction model is used to predict the sub-features without semantics in the merged embedding features to obtain the predicted embedding features.

[0195] Therefore, if multiple sub-speeches in multi-type speech correspond to the same semantics, in order to improve the prediction accuracy, after obtaining the masked embedding features, the sub-features belonging to the same semantics in the masked embedding features can be merged and then predicted by the prediction model to improve the accuracy of the predicted embedding features obtained.

[0196] E7: Obtain the candidate embedding features of the candidate sub-text, the correspondence between the candidate sub-text and the mapped sub-text, and the mapped embedding features corresponding to multiple mapped sub-texts respectively.

[0197] The candidate sub-text is a sub-text belonging to the first type, and the mapped sub-text is a sub-text belonging to the second type. The candidate sub-text and the mapped sub-text with a corresponding relationship express the same semantics. For example, in the scenario of converting speech with mixed Chinese and English, the first type is English, the second type is Chinese, the candidate sub-texts are the possible mixed English, such as meeting, deadline, etc., the mapped sub-text corresponding to the candidate sub-text meeting is "会议" (meeting in Chinese), and the mapped sub-text corresponding to the candidate sub-text deadline is "截止日期" (deadline in Chinese), etc.

[0198] The embodiments of this application do not specifically limit the method of obtaining the correspondence. For example, if candidate subtexts that may be needed are obtained, the candidate subtexts belonging to the first type are converted into mapping subtexts belonging to the second type by means of a conversion model. The mapping subtexts and the candidate subtexts generally express the same semantics, thereby establishing the correspondence between the mapping subtexts and the candidate subtexts.

[0199] The mapping embedding feature is the embedding feature corresponding to the mapping subtext. The embodiments of this application do not specifically limit the way of obtaining the mapping embedding feature. For example, the mapping embedding feature is obtained by extracting features from the mapping subtext through a prediction model. Thus, the extraction logic of the mapping embedding feature and the prediction embedding feature is similar. Moreover, the prediction embedding feature is a feature that only represents the second type, and the mapping embedding feature is also a feature that only represents the second type. The subsequent calculation of the similarity between the two is more reliable.

[0200] As one possible approach, feature extraction can be performed on the mapped sub-texts using a prediction model to obtain the mapping embedding features corresponding to each mapped sub-text. This makes the extraction logic of the mapping embedding features and the predicted embedding features more consistent, thereby improving the reliability of subsequent similarity calculations.

[0201] E8: Obtain the target mapped subtext based on the similarity between multiple mapped embedded features and the predicted embedded features.

[0202] The similarity between the predicted embedding feature and each mapped embedding feature is calculated, thereby identifying the mapped embedding feature whose similarity with the predicted embedding feature satisfies the second similarity condition from multiple mapped embedding features. The mapped subtext corresponding to the mapped embedding feature that satisfies the second similarity condition is determined as the target mapped subtext. The target mapped subtext can be one or more, and this application does not make a specific limitation on this.

[0203] The embodiments of this application do not specifically limit the first similarity condition and the second similarity condition; the two can be the same or different.

[0204] E9: Based on the correspondence and target mapping subtext, obtain the target candidate subtext.

[0205] The target candidate subtext is a candidate subtext that corresponds to the target mapping subtext. The target candidate subtext and the target mapping subtext are subtexts that belong to different types but have the same semantics.

[0206] E10: Obtain the first subtext based on the target candidate subtext.

[0207] E11: Based on the acoustic features of multiple speech types and the first sub-text, the converted text corresponding to the multiple speech types is obtained through analysis using a large language model.

[0208] Therefore, the second acoustic feature extraction model is used to extract features from multiple types of speech to obtain the acoustic features of multiple types of speech. The feature transformation model is used to convert the acoustic features of multiple types of speech into initial embedding features that are more suitable for similarity calculation. Then, combined with the second classification model and the prediction model, the initial embedding features including the first type and the second type are converted into predicted embedding features that only include the second type. Thus, the first sub-text is obtained through similarity calculation. Finally, the acoustic features of the first sub-text and multiple types of speech are analyzed by the large language model to obtain the converted text of multiple types of speech.

[0209] Compared to the second method mentioned above, this method requires more models, but the extraction logic of mapping and predicting embedding features is similar. Furthermore, both methods only represent the second type of embedding features, leading to higher reliability in subsequent similarity calculations. Additionally, it reduces the reliance on the feature transformation model, resulting in more accurate first sub-text used as prompt words, thus improving the accuracy of speech-to-text conversion. Moreover, it achieves high accuracy in first-type conversion without the need for a feature transformation model, reducing the training difficulty of the feature transformation model.

[0210] The training process of the second acoustic feature extraction model and the feature transformation model is described below. See F1-F13 for details (not shown in the figures).

[0211] See Figure 11 The figure is a training diagram of another speech-to-text method provided in the embodiments of this application.

[0212] F1: Get the third sample speech.

[0213] The third sample speech is the sample speech used for training. Compared to multi-type speech, it also includes the required labels. Specifically, the third sample speech includes sub-sample speech belonging to the first type and sub-sample speech belonging to the second type. In the third sample speech, the number of sub-sample speech belonging to the first type is less than the number of sub-sample speech belonging to the second type, that is, the sub-sample speech belonging to the first type is relatively sparse in the third sample speech.

[0214] The third sample speech includes a fifth label and a sixth label. The fifth label indicates the true type of each sub-sample speech in the third sample speech, which refers to the correct type corresponding to the sub-speech. The sixth label is the actual text corresponding to the third sample speech, that is, the correct text corresponding to the third sample speech after it has been converted into text.

[0215] As one possible implementation, the second sample speech can be the same as the first sample speech, the difference being that the second sample speech includes the third label, while the third sample speech includes the fifth and sixth labels.

[0216] For example, in this embodiment, taking the third sample speech as Chinese mixed with English as an example, in this scenario, Chinese belongs to the second type, equivalent to the mother language, that is, the main language of the sentence, that is, the language used for most of the content of the sentence; English belongs to the first type, equivalent to embedded speech, that is, the language to which the non-mother language part of the sentence belongs. Converting this type of third sample speech can also be called code-switching.

[0217] One possible implementation involves obtaining an initial text, which includes multiple sub-initial texts, each belonging to the second type; that is, the initial text consists only of text of the second type. The target sub-initial texts within the initial text are translated, changing them from the second type to the first type, resulting in a target translated text belonging to the first type. The target sub-initial texts can be one or more. The target sub-initial texts in the initial text are then replaced with the target translated text to obtain sample text, which includes both the first and second types. Based on this sample text, a third sample speech is obtained, such as through speech reading from the sample text. The sixth label can be directly taken from the sample text, and the fifth label can be obtained based on the target translated text. In other words, in the third sample speech, the sub-speech corresponding to the target translated text is of the first type, and the sub-speech corresponding to the remaining texts is of the second type. Therefore, the third sample speech can be obtained using data from a single language, and the labels are easy to obtain, thus reducing the difficulty and cost of acquiring training data.

[0218] One possible implementation involves obtaining an initial audio file, which includes multiple sub-initial audio files, each belonging to a second type. That is, the initial audio file consists only of audio files of the second type. A target sub-initial audio file is then replaced, changing it from the second type to the first type, resulting in a target replacement audio file belonging to the first type. This target sub-initial audio file can be one or more sub-initial audio files. Replacing the target sub-initial audio file with the target replacement audio file yields a third sample speech, which includes both the first and second types. The sixth label can be obtained based on the text corresponding to the initial audio file, and the fifth label can be obtained based on the target replacement audio file. In the third sample speech, the target replacement audio file is of the first type, and the remaining audio files are of the second type. This allows the third sample speech to be obtained from single-speech data, reducing the difficulty of acquiring and constructing training data.

[0219] F2: Extract features from the third sample speech using the initial second acoustic feature extraction model to obtain the acoustic features of the third sample speech.

[0220] Continuing with the example of the third sample speech being Chinese speech interspersed with English, if the third sample speech includes T sub-sample speech, where T is an integer greater than 1, these sub-sample speech are obtained by dividing the third sample speech into frames. The third sample speech can be represented as X = [x1, x2, ..., x...]. T ], where X is the third sample speech, x1 is the first sub-sample speech in the third sample speech, x2 is the second sub-sample speech in the third sample speech, and x T Let T be the Tth subsample speech in the third sample speech, where T represents the length of the third sample speech, i.e., the number of frames of the third sample speech.

[0221] If the initial second acoustic feature extraction model is the aforementioned wav2vec, then the third sample speech X = [x1, x2, ..., x T The input is fed into the wav2vec model, which performs feature extraction to obtain the acoustic features of the third sample speech, which can be represented as H = [h1, h2, ..., h...]. T ], where H is the acoustic feature of the third sample speech, h1 is the acoustic feature based on the first sub-sample speech in the third sample speech, h2 is the acoustic feature based on the second sub-sample speech in the third sample speech, h T The acoustic features are based on the Tth subsample of the third sample speech.

[0222] F3: The acoustic features of the third sample speech are identified by the initial second classification model to obtain the predicted type of each subsample speech in the third sample speech.

[0223] In this embodiment, the second classification model is used not only to identify the first type of sub-speech but also to identify the second type of sub-speech. Correspondingly, the initial second classification model can also identify the first type of sub-speech and the second type of sub-speech. Thus, the initial second classification model is used to identify the third sample speech, obtaining the predicted type corresponding to each sub-sample speech in the third sample speech. The predicted type is the type obtained through prediction, which may be correct or incorrect.

[0224] Continuing with the example of the third sample speech being Chinese speech mixed with English, the structure of the initial second classification model can consist of a linear layer (i.e., a fully connected layer, used to perform linear transformation on the input data) and an activation function layer (i.e., a softmax layer, used to output the probability of each element, with the sum of the probabilities of all elements being 1). The initial second classification model has 3 output nodes, each corresponding to a type. The three types are the first type, the second type, and the silence type. Among them, speech belonging to the silence type is speech without any speech signal in the third sample speech.

[0225] Therefore, by using the acoustic features H = [h1, h2, ..., h] of the third sample speech... T The input is fed into the initial second classification model, which performs recognition to obtain the predicted type corresponding to each sub-sample of speech in the third sample speech. This can be represented as D = [d0, d1, d1, d2, ...], which is equivalent to obtaining the frame-level recognition result. The first position in D represents the predicted type of the first sub-sample of speech in the third sample speech, and so on. d0 represents the silence type, d1 represents the second type, and d2 represents the first type. For ease of understanding, assume T = 4 and D = [d0, d1, d1, d2].

[0226] As one possible approach, beam search can be performed on the output of the initial second classification model to obtain the prediction type corresponding to each subsample speech in the third sample speech, thus avoiding getting trapped in local optima.

[0227] F4: Based on the prediction type corresponding to each sub-sample speech in the third sample speech, obtain the position of the sub-sample speech belonging to the first type in the third sample speech.

[0228] Based on the prediction type corresponding to each sub-sample speech in the third sample speech, determine the location of the sub-sample speech belonging to the first type in the third sample speech.

[0229] Continuing with the example of the third sample speech being Chinese mixed with English, assuming T = 4 and D = [d0, d1, d1, d2], the first subsample speech in the third sample speech is of the silent type, the second and third subsamples are of the second type, and the fourth subsample speech is of the first type. The position of the subsample speech belonging to the first type in the third sample speech can be represented by the mask vector, i.e., mask = [1, 1, 1, 0], where mask is the mask vector, 1 indicates that the position is not of the first type, and 0 indicates that the position belongs to the first type.

[0230] F5: The acoustic features belonging to the third sample speech are transformed by the initial feature transformation model to obtain the initial sample embedding features of the third sample speech.

[0231] The initial sample embedding features of the third sample speech are the embedding features obtained after transforming the acoustic features of the third sample speech.

[0232] Continuing with the example of the third sample speech being Chinese speech mixed with English, the structure of the initial feature transformation model can consist of a linear layer and an activation function layer. The number of output nodes of the initial feature transformation model is the sum of the modeling units of the native language and the modeling units of the embedded language plus 1 (representing the blank symbol). The modeling units can be words (such as English words), characters (such as Chinese characters and English letters), byte-pair encoding (BPE), etc.

[0233] Therefore, by using the acoustic features H = [h1, h2, ..., h] of the third sample speech... T The input is fed into the initial feature transformation model, which transforms the input to obtain the initial sample embedding features of the third sample speech, which can be represented as hyp = [u1, u2, u3, u4]. Here, hyp is the initial sample embedding feature of the third sample speech, u1 is the initial sample embedding feature corresponding to the first sub-sample speech in the third sample speech, u2 is the initial sample embedding feature corresponding to the second sub-sample speech in the third sample speech, and so on.

[0234] As one possible approach, beam search can be performed on the output of the initial feature transformation model to obtain the initial sample embedding features of the third sample speech, thus avoiding getting trapped in local optima.

[0235] F6: Based on the initial sample embedding features of the third sample speech, obtain the predicted text corresponding to the third sample speech.

[0236] The initial sample embedding features of the third sample speech are used to describe the text corresponding to the third sample speech. Therefore, based on the initial sample embedding features of the third sample speech, the text corresponding to the third sample speech can be obtained, that is, the predicted text corresponding to the third sample speech.

[0237] The predicted text is based on the third sample speech. It may be a text that correctly represents the third sample speech or a text that incorrectly represents the third sample speech.

[0238] As one possible implementation, the initial feature transformation model includes multiple neural network layers. One of its neural network layers can output the initial sample embedding features of the third sample speech, and another neural network layer can output the predicted text corresponding to the third sample speech. That is, the initial feature transformation model outputs the initial sample embedding features and predicted text of the third sample speech, thereby reducing the complexity of the operation.

[0239] F7: Based on the position of the sub-sample speech belonging to the first type in the third sample speech, the initial sample embedding features of the third sample speech are transformed to obtain the sample mask embedding features of the third sample speech.

[0240] The sample mask embedding feature includes semantically meaningless sample sub-features, which are obtained based on the positions of first-type sub-sample speech within the third-type sample speech. In the sample mask embedding feature, the features corresponding to the positions of first-type sub-sample speech within the third-type sample speech have no practical meaning. That is, the sample mask embedding feature only includes sub-features corresponding to second-type sub-sample speech; positions that should correspond to first-type sub-sample speech are replaced with semantically meaningless content, such as 0.

[0241] Continuing with the example of the third sample speech being Chinese speech interspersed with English, the masking vector mask = [1,1,1,0], used to represent the position of the first type of sub-sample speech in the third sample speech, has the same length as the initial sample embedding feature hyp = [u1,u2,u3,u4]. Therefore, the masking vector can be applied to the initial sample embedding feature of the third sample speech, such as replacing the element at position 0 with... <mask>The symbol, the sample mask embedding feature of the third sample speech can be represented as hyp ′ =[u1,u2,u3, <mask>].

[0242] Furthermore, if multiple sub-samples in the third sample speech correspond to the same semantic meaning, then sub-features belonging to the same semantic meaning in the sample mask embedding features can be merged. Continuing with the example of the third sample speech being Chinese speech interspersed with English, we will further analyze the hypothallic language... ′ Perform a merge operation to obtain the hyp. ″ Assume u1 = <blank>If u2 = u3 = A, then hypo ″ =[A, <mask>]. Among them, hyp ″ This refers to the aforementioned merged embedding feature.

[0243] F8: Based on the sample mask embedding features, the prediction model is used to predict the semantically unrelated sample sub-features in the sample mask embedding features to obtain the sample prediction embedding features.

[0244] Based on the sample mask embedding features, a prediction model is used to predict the sample predicted embedding features. The prediction model is used to predict features belonging to the second type, such as predicting the (i+1)th sub-feature based on the i-th sub-feature in the mask embedding features through contextual understanding. This completes the supplementation of sub-features in the mask embedding features that have no actual semantic meaning. That is, the meaning that the sub-features in the mask embedding features that have no actual semantic meaning are predicted to express their meaning, thus obtaining the sample predicted embedding features.

[0245] Continuing with the example of the third sample speech being Chinese mixed with English, the prediction model can be the BERT model, thus rendering the hyphenated speech. ″ When inputting into the BERT model, select the output of the last hidden layer of the BERT model, which corresponds to... <mask>The elements at the position are used as semantic features embedded in the word u4 of the embedded language, thereby enabling the prediction of sub-features without semantic meaning in the sample mask embedding features and obtaining the sample prediction embedding features.

[0246] F9: Based on the similarity between multiple mapping embedding features and the sample prediction embedding features, obtain the sample mapping subtext.

[0247] The similarity between the sample prediction embedding feature and each mapping embedding feature is calculated, thereby identifying the mapping embedding feature from multiple mapping embedding features that satisfies the second similarity condition with the sample prediction embedding feature. The mapping subtext corresponding to the mapping embedding feature that satisfies the second similarity condition is determined as the target mapping subtext. The target mapping subtext can be one or multiple, and this application does not make a specific limitation on this.

[0248] Continuing with the example of the third sample speech being Chinese mixed with English, we calculate the similarity between multiple mapped embedding features and the predicted embedding features of the sample, such as through an attention mechanism, and select the top K most similar mapped embedding features. Here, K is a hyperparameter; the larger K is, the greater the probability that the correct embedded language words will be included in the subsequent third sample sub-text, resulting in higher conversion accuracy. However, this also increases the computational cost of the large language model. If K is too large, it may cause confusion when matching text to sub-speech belonging to the first type, leading to a decrease in recognition accuracy. As a possible implementation, K can be set to 128.

[0249] It should be noted that, before step F9, candidate embedding features of candidate subtexts can be obtained, as well as the correspondence between candidate subtexts and mapped subtexts and the mapping embedding features corresponding to multiple mapped subtexts.

[0250] For example, obtain a list of commonly used nouns and verbs in the embedded language (nouns and verbs usually appear as embedded languages ​​in code-switching phenomena), then use a dictionary or model to translate these words into their corresponding words in the native language, and then use the corresponding vectors in the feature extraction layer (such as the embedding layer) of the BERT model as the vectors of the original embedded language words, that is, the mapping embedding vectors corresponding to the mapped subtext.

[0251] F10: Based on the correspondence and sample mapping subtext, obtain the sample candidate subtext.

[0252] Obtain candidate subtexts that correspond to the sample mapping subtexts, i.e., sample candidate subtexts. The sample candidate subtexts and the sample mapping subtexts are different types of texts that express the same semantics.

[0253] F11: Obtain the third sample subtext based on the sample candidate subtext.

[0254] The candidate sample text is more likely to be the text corresponding to the first type of sub-speech in the third sample speech. Therefore, the candidate sample text is used as the third sample text and used as a prompt word to guide the large language model to perform the conversion, so as to improve the accuracy of the large language model conversion.

[0255] Continuing with the example of the third sample speech being Chinese speech mixed with English, we obtain K candidate sub-texts based on the mapping sub-texts of the top K samples in similarity ranking. These K candidate sub-texts are then concatenated to obtain the third sample sub-text, which can be represented as candidates=[w1,w2,w3,…,w…]. K ], where candidatess is the third sample subtext, and w1 is the first sample candidate subtext among the K sample candidate subtexts.

[0256] F12: Based on the acoustic features of the third sample speech and the third sample subtext, the large language model is used to analyze and obtain the converted text corresponding to the third sample speech.

[0257] The large language model is a pre-trained model, meaning that the already trained model can be used directly without further training, which reduces the training difficulty and does not affect the accuracy of subsequent conversions.

[0258] Continuing with the example of the third sample speech being Chinese speech mixed with English, the acoustic features of the third sample speech and the third sample sub-text are concatenated and input into the large language model. The large language model is then used for analysis to obtain the converted text corresponding to the third sample speech.

[0259] The acoustic features of the third sample speech and the third sample subtext, after being concatenated, can be represented as P = [w1, w2, w3, ..., w K ,h1,h2,…,h T The output of a large language model can be represented as Y = [y1, y2, y3, ..., y]. K ,y 1+K ,y 2+K ,…,y T+K ], where y1, y2, y3, ..., y K It corresponds to w1, w2, w3, ..., w K The output of Y has no practical meaning; the last T outputs are taken as the final recognition result, i.e., Y. ′ =[y 1+K ,y 2+K ,…,y T+K ], where Y ′ The converted text corresponding to the third sample speech, y 1+K For the first sub-sample speech in the third sample speech, the sub-text is y. T+K This is the subtext corresponding to the Tth subsample speech in the third sample speech.

[0260] It should be noted that if there are M sub-samples of speech belonging to the first type, then each sub-sample can yield K candidate sub-texts, which are then concatenated with the third sample speech using M*K candidate sub-texts.

[0261] F13: Based on the differences between the predicted types of the fifth label and the third sample speech, the differences between the sixth label and the predicted text, and the differences between the converted text corresponding to the sixth label and the third sample speech, adjust the model parameters of the initial second acoustic feature extraction model, the model parameters of the initial second classification model, and the initial feature conversion model to obtain the second acoustic feature extraction model, the second classification model, and the feature conversion model.

[0262] Based on the differences between the predicted types of the fifth label and the third sample speech, the differences between the sixth label and the predicted text, and the differences between the converted text corresponding to the sixth label and the third sample speech, the model parameters of the initial second acoustic feature extraction model are adjusted to obtain the second acoustic feature extraction model, the model parameters of the initial second classification model are adjusted to obtain the second classification model, and the initial feature conversion model is adjusted to obtain the feature conversion model.

[0263] Continuing with the example of the third sample speech being Chinese mixed with English, the difference between the predicted types of the fifth label and the various sub-samples in the third sample speech can be represented as follows: Where CE represents the loss function corresponding to the cross-entropy criterion, and D represents the prediction type of each subsample speech in the third sample speech. The fifth label indicates the true type of each subsample speech in the third sample speech.

[0264] The difference between the sixth label and the predicted text can be represented as Where CTC is the loss function corresponding to the Connectionist Temporal Classification criterion, and Y″ is the predicted text corresponding to the third sample speech. The sixth label is the actual text corresponding to the third sample speech.

[0265] The difference between the converted text corresponding to the sixth label and the third sample speech can be represented as: Where CE represents the loss function corresponding to the cross-entropy criterion, Y ′ This is the converted text corresponding to the third sample speech. The sixth label is the actual text corresponding to the third sample speech.

[0266] Therefore, the final loss function can be expressed as L = αL LID +βL ctc +L LLM , where α and β are hyperparameters, representing the weights of the corresponding loss function.

[0267] During training, the parameters of the BERT model and the large language model are frozen and not updated. Only a subset of parameters from the initial second acoustic feature extraction model, the initial second classification model, and the initial feature transformation model are updated during training. Model training can employ any stochastic gradient descent optimization algorithm, such as the Adamw algorithm.

[0268] Furthermore, the mixed error rate (MER) test results on code-switching tests are shown in Table 1 for methods that do not use pre-trained models and large language models (Aligning Speech to Languages ​​to Enhance Code-switching Speech Recognition).

[0269] Table 1

[0270] Align speech with language to enhance code-to-speech recognition 23.2％ The method of the present invention 18.7％

[0271] Therefore, since the large language model has the ability to learn multiple language types, the prediction model only predicts sub-features belonging to the second type. This means that the large language model and the prediction model are unaffected by the inclusion of multiple speech types, allowing the direct use of pre-trained speech-to-text and large language models. During training, the model parameters of the large language model and the prediction model can be frozen, preventing updates during training. Training is then performed on the second acoustic feature extraction model, the feature transformation model, and the second classification model, which are affected by multiple speech types. This involves adjusting the initial model parameters of the second acoustic feature extraction model, the initial model parameters of the feature transformation model, and the model parameters of the second classification model, resulting in the corresponding second acoustic feature extraction model, feature transformation model, and second classification model.

[0272] By combining a training prediction model, an initial feature transformation model, and an initial second classification model, third-sample speech containing both first and second types can be converted into predicted embedding features that only include samples of the second type. This reduces the training difficulty of the initial feature transformation model and the subsequent reliance on it. Furthermore, by converting acoustic features into predicted features that only include samples of the second type, the impact of sub-features belonging to the first type on semantics is reduced, thus decreasing the amount of training data. Moreover, by using a prediction approach, the impact of sparse, uncommon sub-speech belonging to that sparse type on robustness can be reduced, thereby improving the robustness of the speech-to-text process.

[0273] In addition to the speech-to-text method described above, this application also provides a corresponding speech-to-text apparatus so that the above speech-to-text method can be applied and implemented in practice.

[0274] See Figure 12 This figure is a schematic diagram of the structure of a speech-to-text device provided in an embodiment of this application. Figure 12 As shown, the speech-to-text device 1200 includes: an acquisition unit 1201, a feature extraction unit 1202, a recognition unit 1203, and an analysis unit 1204;

[0275] The acquisition unit 1201 is used to acquire multiple types of speech, which include a first sub-speech belonging to a first type and a second sub-speech belonging to a second type. In the multiple types of speech, the first number of the first sub-speech belonging to the first type is less than the second number of the second sub-speech belonging to the second type. The first type and the second type are different types.

[0276] The feature extraction unit 1202 is used to extract features from the multi-type speech to obtain the acoustic features of the multi-type speech.

[0277] The recognition unit 1203 is used to perform type recognition on the multi-type speech to obtain a first sub-text belonging to the first type, wherein the first sub-text is the text corresponding to the first sub-speech belonging to the first type in the multi-type speech;

[0278] The analysis unit 1204 is used to analyze the acoustic features of the multiple types of speech and the first sub-text through a large language model to obtain the converted text corresponding to the multiple types of speech.

[0279] As can be seen from the above technical solution, multiple types of speech are acquired, including first sub-speech belonging to the first type and second sub-speech belonging to the second type. The number of first sub-speech belonging to the first type is less than the number of second sub-speech belonging to the second type, meaning that the first sub-speech belonging to the first type is sparse data in the multiple speech types, making its conversion more difficult. Feature extraction is performed on the multiple speech types to obtain their acoustic features. These acoustic features reflect the characteristics of the speech, thus helping the large language model understand the multiple speech types. Recognition of the multiple speech types yields the first sub-text, which not only belongs to the first type but also corresponds to the first sub-speech belonging to the first type within the multiple speech types. Using this sub-text as a cue word helps the large language model understand the more difficult-to-convert first sub-speech belonging to the first type. Finally, the large language model analyzes the acoustic features and the first sub-text of the multiple speech types. The large language model itself possesses the ability to learn multiple language types, enabling it to understand the meaning of each type of sub-speech within the multiple speech types based on their acoustic features. Furthermore, the large language model also has context learning capabilities. Based on this capability, in the process of predicting the text corresponding to each sub-speech, the large language model matches the first sub-text of the first sub-speech belonging to the first type, which is more difficult to convert among multiple types of speech, as the prompt word, so as to reduce the recognition difficulty and thus obtain the converted text corresponding to multiple types of speech.

[0280] Therefore, by using a large language model with context learning and multi-type language learning capabilities, the acoustic features of multi-type speech that have a representational function, as well as the first sub-text that has a cueing function, can be analyzed to obtain the converted text corresponding to multi-type speech. This improves the large language model's ability to recognize the relatively sparse first-type sub-speech, and thus improves the accuracy of the large language model in converting multi-type speech.

[0281] As one possible implementation, the feature extraction unit 1202 is specifically used for:

[0282] The acoustic features of the multiple types of speech are obtained by extracting features from the first acoustic feature extraction model.

[0283] The identification unit 1203 is specifically used for:

[0284] The first sub-speech belonging to the first type is obtained by performing type recognition on the multi-type speech using the first classification model;

[0285] The first sub-speech belonging to the first type is converted using a speech-to-text model to obtain the first sub-text.

[0286] As one possible implementation, if the first classification model also needs to identify a second sub-speech belonging to the second type, then the identification unit 1203 is specifically used for:

[0287] The first classification model is used to identify the types of the multiple types of speech, thereby obtaining the types corresponding to each of the sub-speech in the multiple types of speech;

[0288] Based on the type corresponding to each of the sub-speech in the multi-type speech, the first sub-speech belonging to the first type in the multi-type speech is obtained.

[0289] As one possible implementation, the device further includes a first training unit for:

[0290] A first sample speech is obtained. The first sample speech includes sub-sample speech belonging to the first type and sub-sample speech belonging to the second type. In the first sample speech, the number of sub-sample speech belonging to the first type is less than the number of sub-sample speech belonging to the second type. The first sample speech includes a first tag and a second tag. The first tag is used to indicate the sub-sample speech belonging to the first type in the first sample speech, and the second tag is the real text corresponding to the first sample speech.

[0291] The acoustic features of the first sample speech are obtained by extracting features from the first sample speech using an initial first acoustic feature extraction model.

[0292] The first sample speech is identified by the initial first classification model to obtain the sub-sample speech belonging to the first type in the first sample speech;

[0293] The speech-to-text model is used to convert the sub-sample speech belonging to the first type to obtain the first sample sub-text.

[0294] The acoustic features of the first sample speech and the first sample subtext are analyzed using the large language model to obtain the converted text corresponding to the first sample speech.

[0295] Based on the differences between the first label and the sub-sample speech belonging to the first type in the first sample speech, and the differences between the second label and the converted text corresponding to the first sample speech, the model parameters of the initial first acoustic feature extraction model and the model parameters of the initial first classification model are adjusted to obtain the first acoustic feature extraction model and the first classification model.

[0296] As one possible implementation, if the first label is also used to indicate sub-sample speech belonging to the second type in the first sample speech, then the first training unit is specifically used for:

[0297] The first sample speech is identified by the initial first classification model to obtain the type corresponding to each of the sub-sample speech in the first sample speech. The type corresponding to each of the sub-sample speech in the first sample speech includes the sub-sample speech in the first sample speech that belongs to the first type.

[0298] Based on the differences between the types corresponding to each of the sub-sample speech in the first label and the first sample speech, and the differences between the converted text corresponding to the second label and the first sample speech, the model parameters of the initial first acoustic feature extraction model and the model parameters of the initial first classification model are adjusted to obtain the first acoustic feature extraction model and the first classification model.

[0299] As one possible implementation, the feature extraction unit 1202 is specifically used for:

[0300] The acoustic features of the multiple types of speech are obtained by extracting features from the multiple types of speech using a second acoustic feature extraction model.

[0301] The identification unit 1203 is specifically used for:

[0302] The acoustic features of the various speech types are transformed using a feature transformation model to obtain initial embedding features;

[0303] Obtain candidate embedding features of candidate subtexts, wherein the candidate subtexts are subtexts belonging to the first type;

[0304] Based on the similarity between the initial embedding features and each of the candidate embedding features, a target candidate subtext is obtained from the multiple candidate subtexts, and the similarity between the candidate embedding features of the target candidate subtext and the initial embedding features satisfies the first similarity condition.

[0305] The first subtext is obtained based on the target candidate subtext.

[0306] As one possible implementation, the device further includes a second training unit for:

[0307] Obtain a second sample speech, which includes sub-sample speech belonging to the first type and sub-sample speech belonging to the second type. In the second sample speech, the number of sub-sample speech belonging to the first type is less than the number of sub-sample speech belonging to the second type. The second sample speech includes a third tag, which is the real text corresponding to the second sample speech.

[0308] The acoustic features of the second sample speech are obtained by extracting features from the second sample speech using the initial second acoustic feature extraction model.

[0309] The acoustic features belonging to the second sample speech are transformed by the initial feature transformation model to obtain the initial sample embedding features of the second sample speech.

[0310] Based on the similarity between the initial sample embedding features of the second sample speech and each of the candidate embedding features, sample candidate subtexts are obtained from the multiple candidate subtexts, and the similarity between the candidate embedding features of the sample candidate subtexts and the initial sample embedding features of the second sample speech satisfies the first similarity condition.

[0311] Based on the candidate sample subtexts, a second sample subtext is obtained;

[0312] The acoustic features of the second sample speech and the second sample subtext are analyzed using the large language model to obtain the converted text corresponding to the second sample speech;

[0313] Based on the difference between the converted text corresponding to the third label and the second sample speech, the model parameters of the initial second acoustic feature extraction model and the model parameters of the initial feature conversion model are adjusted to obtain the second acoustic feature extraction model and the feature conversion model.

[0314] As one possible implementation, the identification unit 1203 is further configured to:

[0315] The second classification model is used to identify the type of the multi-type speech, and the position of the first sub-speech belonging to the first type in the multi-type speech is obtained;

[0316] Based on the position of the first sub-speech belonging to the first type in the multi-type speech, the initial embedding feature is transformed to obtain a mask embedding feature including semantically null sub-features, wherein the semantically null sub-features are obtained based on the position of the first sub-speech belonging to the first type in the multi-type speech;

[0317] Based on the mask embedding features, the semantically unrelated sub-features in the mask embedding features are predicted using a prediction model to obtain the predicted embedding features;

[0318] Obtain the correspondence between candidate subtexts and mapped subtexts, as well as the mapping embedding features corresponding to multiple mapped subtexts respectively. The mapped subtexts are subtexts belonging to the second type, and the candidate subtexts and mapped subtexts with the correspondence express the same semantics.

[0319] The identification unit 1203 is specifically used for:

[0320] Based on the similarity between the multiple mapping embedding features and the predicted embedding features, a target mapping sub-text is obtained. The target mapping sub-text is the mapping sub-text corresponding to the mapping text feature whose similarity with the predicted embedding feature satisfies the second similarity condition.

[0321] Based on the correspondence and the target mapping subtext, the target candidate subtext is obtained.

[0322] As one possible implementation, the device further includes a third training unit for:

[0323] Obtain a third sample speech, which includes sub-sample speech belonging to the first type and sub-sample speech belonging to the second type. In the third sample speech, the number of sub-sample speech belonging to the first type is less than the number of sub-sample speech belonging to the second type. The third sample speech includes a fifth tag and a sixth tag. The fifth tag is used to indicate the true type of each sub-sample speech in the third sample speech, and the sixth tag is the real text corresponding to the third sample speech.

[0324] The acoustic features of the third sample speech are obtained by extracting features from the third sample speech using the initial second acoustic feature extraction model.

[0325] The acoustic features of the third sample speech are identified by an initial second classification model to obtain the predicted type of each sub-sample speech in the third sample speech;

[0326] Based on the prediction type corresponding to each of the sub-samples in the third sample speech, the position of the sub-sample speech belonging to the first type in the third sample speech is obtained;

[0327] The acoustic features belonging to the third sample speech are transformed by the initial feature transformation model to obtain the initial sample embedding features of the third sample speech.

[0328] Based on the initial sample embedding features of the third sample speech, the predicted text corresponding to the third sample speech is obtained;

[0329] Based on the position of the sub-sample speech belonging to the first type in the third sample speech, the initial sample embedding features of the third sample speech are transformed to obtain the sample mask embedding features of the third sample speech. The sample mask embedding features include semantically null sample sub-features, which are obtained based on the position of the sub-sample speech of the first type in the third sample speech.

[0330] The prediction model is used to predict the semantically null sample sub-features in the sample mask embedding features to obtain the sample prediction embedding features;

[0331] Based on the similarity between the multiple mapping embedding features and the sample prediction embedding features, the sample mapping subtext is obtained;

[0332] Based on the correspondence and the sample mapping subtext, the sample candidate subtext is obtained;

[0333] Based on the candidate sample texts, a third sample text is obtained.

[0334] The acoustic features of the third sample speech and the third sample subtext are analyzed using the large language model to obtain the converted text corresponding to the third sample speech;

[0335] Based on the differences between the predicted types of the fifth label and each of the sub-sample speech in the third sample speech, the differences between the sixth label and the predicted text, and the differences between the sixth label and the converted text corresponding to the third sample speech, the model parameters of the initial second acoustic feature extraction model, the model parameters of the initial second classification model, and the initial feature conversion model are adjusted to obtain the second acoustic feature extraction model, the second classification model, and the feature conversion model.

[0336] As one possible implementation, if multiple sub-speech elements in the multi-type speech correspond to the same semantic meaning, then the recognition unit 1203 is specifically used for:

[0337] In the mask embedding features, the sub-features corresponding to multiple sub-speechs with the same semantic meaning are merged to obtain merged embedding features;

[0338] Based on the merged embedding features, the prediction model is used to predict the sub-features in the merged embedding features that have no actual semantic meaning, thereby obtaining the predicted embedding features.

[0339] As one possible implementation, the apparatus further includes a data construction unit for:

[0340] Obtain the initial text belonging to the second type, wherein the initial text includes multiple sub-initial texts;

[0341] Translate the target sub-initial text among the plurality of sub-initial texts to obtain the target translated text belonging to the first type;

[0342] Replace the target sub-initial text in the initial text with the target translated text to obtain the sample text;

[0343] Based on the sample text, the third sample speech is obtained.

[0344] As one possible implementation, the type is used to describe the language type, the degree of information compression, or the importance of information.

[0345] This application also provides a computer device, which can be a server or a terminal device. The computer device provided in this application will be described below from a hardware implementation perspective. Figure 13 The diagram shown is a structural schematic of the server. Figure 14 The diagram shown is a structural schematic of the terminal device.

[0346] See Figure 13 This figure is a schematic diagram of a server structure provided in an embodiment of this application. The server 1400 can vary considerably due to different configurations or performance. It may include one or more processors 1422, such as a central processing unit (CPU), memory 1432, and one or more application programs 1442 or data storage media 1430 (e.g., one or more mass storage devices). The memory 1432 and storage media 1430 can be temporary or persistent storage. The program stored in the storage media 1430 may include one or more modules (not shown in the figure), each module may include a series of instruction operations on the server. Furthermore, the processor 1422 may be configured to communicate with the storage media 1430 and execute the series of instruction operations in the storage media 1430 on the server 1400.

[0347] Server 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input / output interfaces 1458, and / or one or more operating systems 1441, such as Windows Server. TM Mac OS X TM Unix TM Linux TM FreeBSD TM etc.

[0348] The steps performed by the server in the above embodiments can be based on this Figure 13 The server structure shown.

[0349] The processor 1422 is used to perform the following steps:

[0350] Acquire multiple types of speech, the multiple types of speech including a first sub-speech belonging to a first type and a second sub-speech belonging to a second type, wherein in the multiple types of speech, the first number of the first sub-speech belonging to the first type is less than the second number of the second sub-speech belonging to the second type, and the first type and the second type are different types;

[0351] Feature extraction is performed on the multiple types of speech to obtain the acoustic features of the multiple types of speech;

[0352] Type recognition is performed on the multiple types of speech to obtain a first sub-text belonging to the first type. The first sub-text is the text corresponding to the first sub-speech of the first type in the multiple types of speech.

[0353] Based on the acoustic features of the multiple speech types and the first sub-text, a large language model is used for analysis to obtain the converted text corresponding to the multiple speech types. The converted text includes the text corresponding to the first sub-speech and the text corresponding to the second sub-speech.

[0354] Optionally, the processor 1422 may also execute method steps of any specific implementation of the speech-to-text method in the embodiments of this application.

[0355] See Figure 14 This figure is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. The description will be based on a smartphone as an example. Figure 14 The diagram shown is a partial structural block diagram of the smartphone, which includes: a radio frequency (RF) circuit 1510, a memory 1520, an input unit 1530, a display unit 1540, a sensor 1550, an audio circuit 1560, a Wi-Fi module 1570, a processor 1580, and a power supply 1590, among other components. Those skilled in the art will understand that... Figure 14 The smartphone structure shown does not constitute a limitation on smartphones and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0356] The following is combined Figure 14 A detailed introduction to the various components of a smartphone:

[0357] The RF circuit 1510 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 1580; in addition, it transmits uplink data to the base station.

[0358] The memory 1520 can be used to store software programs and modules, and the processor 1580 runs the software programs and modules stored in the memory 1520 to realize various functions and data processing of the smartphone.

[0359] Input unit 1530 can be used to receive input numeric or character information and generate key signal inputs related to user settings and function control of the smartphone. Specifically, input unit 1530 may include touch panel 1531 and other input devices 1532. Touch panel 1531, also known as a touch screen, can collect touch operations on or near the user and drive corresponding connected devices according to a pre-set program. In addition to touch panel 1531, input unit 1530 may also include other input devices 1532. Specifically, other input devices 1532 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0360] The display unit 1540 can be used to display information input by the user or information provided to the user, as well as various menus of the smartphone. The display unit 1540 may include a display panel 1541, which may optionally be configured as a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0361] Smartphones may also include at least one sensor 1550, such as a light sensor, a motion sensor, and other sensors. Other sensors that smartphones may also be equipped with, such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be detailed here.

[0362] Audio circuit 1560, speaker 1561, and microphone 1562 provide an audio interface between the user and the smartphone. Audio circuit 1560 converts received audio data into electrical signals and transmits them to speaker 1561, where speaker 1561 converts them into sound signals for output. On the other hand, microphone 1562 converts collected sound signals into electrical signals, which are received by audio circuit 1560, converted into audio data, and then processed by processor 1580 before being transmitted via RF circuit 1510 to, for example, another smartphone, or the audio data can be output to memory 1520 for further processing.

[0363] The processor 1580 is the control center of the smartphone, connecting various parts of the smartphone through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 1520, and by calling data stored in the memory 1520. Optionally, the processor 1580 may include one or more processing units.

[0364] The smartphone also includes a power supply 1590 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 1580 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0365] Although not shown, smartphones may also include a camera, Bluetooth module, etc., which will not be described in detail here.

[0366] In this embodiment of the application, the memory 1520 included in the smartphone can store computer programs and transmit the computer programs to the processor.

[0367] The processor 1580 included in the smartphone can execute the speech-to-text method provided in the above embodiments according to the instructions in the computer program.

[0368] This application also provides a computer-readable storage medium for storing a computer program that executes the speech-to-text method provided in the above embodiments.

[0369] On the other hand, embodiments of this application provide a computer program product including a computer program, which, when run on a computer device, causes the computer device to perform the speech-to-text method provided in various optional implementations of the above aspects.

[0370] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium can be at least one of the following media: read-only memory (ROM), RAM, magnetic disk or optical disk, and other media that can store computer programs.

[0371] The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such use of data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "corresponding to," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0372] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0373] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The device and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0374] The above description is merely one specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. Therefore, the scope of protection of this application should be determined by the scope of the claims.< / mask> < / mask> < / blank> < / mask> < / mask>

Claims

1. A method for converting speech to text, characterized in that, The method includes: Acquire multiple types of speech, the multiple types of speech including a first sub-speech belonging to a first type and a second sub-speech belonging to a second type, wherein in the multiple types of speech, the first number of the first sub-speech belonging to the first type is less than the second number of the second sub-speech belonging to the second type, and the first type and the second type are different types; Feature extraction is performed on the multiple types of speech to obtain the acoustic features of the multiple types of speech; Type recognition is performed on the multiple types of speech to obtain a first sub-text belonging to the first type. The first sub-text is the text corresponding to the first sub-speech of the first type in the multiple types of speech. The acoustic features of the multiple speech types and the first sub-text are analyzed using a large language model to obtain the converted text corresponding to the multiple speech types. The converted text includes the text corresponding to the first sub-speech and the text corresponding to the second sub-speech.

2. The method according to claim 1, characterized in that, The step of extracting features from the multiple types of speech to obtain the acoustic features of the multiple types of speech includes: The acoustic features of the multiple types of speech are obtained by extracting features from the first acoustic feature extraction model. The step of performing type recognition on the multiple types of speech to obtain a first sub-text belonging to the first type includes: The first sub-speech belonging to the first type is obtained by performing type recognition on the multi-type speech using the first classification model; The first sub-speech belonging to the first type is converted using a speech-to-text model to obtain the first sub-text.

3. The method according to claim 2, characterized in that, The step of performing type recognition on the multiple types of speech using a first classification model to obtain a first sub-speech belonging to the first type among the multiple types of speech includes: The first classification model is used to identify the types of the multiple types of speech, thereby obtaining the types corresponding to each of the sub-speech in the multiple types of speech; Based on the type corresponding to each of the sub-speech in the multi-type speech, the first sub-speech belonging to the first type in the multi-type speech is obtained.

4. The method according to claim 2 or 3, characterized in that, The method further includes: A first sample speech is obtained. The first sample speech includes sub-sample speech belonging to the first type and sub-sample speech belonging to the second type. In the first sample speech, the number of sub-sample speech belonging to the first type is less than the number of sub-sample speech belonging to the second type. The first sample speech includes a first tag and a second tag. The first tag is used to indicate the sub-sample speech belonging to the first type in the first sample speech, and the second tag is the real text corresponding to the first sample speech. The acoustic features of the first sample speech are obtained by extracting features from the first sample speech using an initial first acoustic feature extraction model. The first sample speech is identified by the initial first classification model to obtain the sub-sample speech belonging to the first type in the first sample speech; The speech-to-text model is used to convert the sub-sample speech belonging to the first type to obtain the first sample sub-text. The acoustic features of the first sample speech and the first sample subtext are analyzed using the large language model to obtain the converted text corresponding to the first sample speech. Based on the differences between the first label and the sub-sample speech belonging to the first type in the first sample speech, and the differences between the second label and the converted text corresponding to the first sample speech, the model parameters of the initial first acoustic feature extraction model and the model parameters of the initial first classification model are adjusted to obtain the first acoustic feature extraction model and the first classification model.

5. The method according to claim 4, characterized in that, If the first label is also used to indicate sub-sample speech belonging to the second type in the first sample speech, then the step of performing type recognition on the first sample speech through the initial first classification model to obtain sub-sample speech belonging to the first type in the first sample speech includes: The first sample speech is identified by the initial first classification model to obtain the type corresponding to each of the sub-sample speech in the first sample speech. The type corresponding to each of the sub-sample speech in the first sample speech includes the sub-sample speech in the first sample speech that belongs to the first type. The step of adjusting the model parameters of the initial first acoustic feature extraction model and the model parameters of the initial first classification model based on the differences between the first label and the sub-sample speech belonging to the first type in the first sample speech, and the differences between the second label and the converted text corresponding to the first sample speech, to obtain the first acoustic feature extraction model and the first classification model, includes: Based on the differences between the types corresponding to each of the sub-sample speech in the first label and the first sample speech, and the differences between the converted text corresponding to the second label and the first sample speech, the model parameters of the initial first acoustic feature extraction model and the model parameters of the initial first classification model are adjusted to obtain the first acoustic feature extraction model and the first classification model.

6. The method according to claim 1, characterized in that, The step of extracting features from the multiple types of speech to obtain the acoustic features of the multiple types of speech includes: The acoustic features of the multiple types of speech are obtained by extracting features from the multiple types of speech using a second acoustic feature extraction model. The step of performing type recognition on the multiple types of speech to obtain a first sub-text belonging to the first type includes: The acoustic features of the various speech types are transformed using a feature transformation model to obtain initial embedding features; Obtain candidate embedding features of candidate subtexts, wherein the candidate subtexts are subtexts belonging to the first type; Based on the similarity between the initial embedding features and each of the candidate embedding features, a target candidate subtext is obtained from the multiple candidate subtexts, and the similarity between the candidate embedding features of the target candidate subtext and the initial embedding features satisfies the first similarity condition. The first subtext is obtained based on the target candidate subtext.

7. The method according to claim 6, characterized in that, The method further includes: Obtain a second sample speech, which includes sub-sample speech belonging to the first type and sub-sample speech belonging to the second type. In the second sample speech, the number of sub-sample speech belonging to the first type is less than the number of sub-sample speech belonging to the second type. The second sample speech includes a third tag, which is the real text corresponding to the second sample speech. The acoustic features of the second sample speech are obtained by extracting features from the second sample speech using the initial second acoustic feature extraction model. The acoustic features belonging to the second sample speech are transformed by the initial feature transformation model to obtain the initial sample embedding features of the second sample speech. Based on the similarity between the initial sample embedding features of the second sample speech and each of the candidate embedding features, sample candidate subtexts are obtained from the multiple candidate subtexts, and the similarity between the candidate embedding features of the sample candidate subtexts and the initial sample embedding features of the second sample speech satisfies the first similarity condition. Based on the candidate sample subtexts, a second sample subtext is obtained; The acoustic features of the second sample speech and the second sample subtext are analyzed using the large language model to obtain the converted text corresponding to the second sample speech; Based on the difference between the converted text corresponding to the third label and the second sample speech, the model parameters of the initial second acoustic feature extraction model and the model parameters of the initial feature conversion model are adjusted to obtain the second acoustic feature extraction model and the feature conversion model.

8. The method according to claim 6, characterized in that, The method further includes: The second classification model is used to identify the type of the multi-type speech, and the position of the first sub-speech belonging to the first type in the multi-type speech is obtained; Based on the position of the first sub-speech belonging to the first type in the multi-type speech, the initial embedding feature is transformed to obtain a mask embedding feature including semantically null sub-features, wherein the semantically null sub-features are obtained based on the position of the first sub-speech belonging to the first type in the multi-type speech; Based on the mask embedding features, the semantically unrelated sub-features in the mask embedding features are predicted using a prediction model to obtain the predicted embedding features; Obtain the correspondence between candidate subtexts and mapped subtexts, as well as the mapping embedding features corresponding to multiple mapped subtexts respectively. The mapped subtexts are subtexts belonging to the second type, and the candidate subtexts and mapped subtexts with the correspondence express the same semantics. The step of obtaining the target candidate subtext from multiple candidate subtexts based on the similarity between the initial embedding features and each of the candidate embedding features includes: Based on the similarity between the multiple mapping embedding features and the predicted embedding features, a target mapping sub-text is obtained. The target mapping sub-text is the mapping sub-text corresponding to the mapping text feature whose similarity with the predicted embedding feature satisfies the second similarity condition. Based on the correspondence and the target mapping subtext, the target candidate subtext is obtained.

9. The solution according to claim 8, characterized in that, The method further includes: Obtain a third sample speech, which includes sub-sample speech belonging to the first type and sub-sample speech belonging to the second type. In the third sample speech, the number of sub-sample speech belonging to the first type is less than the number of sub-sample speech belonging to the second type. The third sample speech includes a fifth tag and a sixth tag. The fifth tag is used to indicate the true type of each sub-sample speech in the third sample speech, and the sixth tag is the real text corresponding to the third sample speech. The acoustic features of the third sample speech are obtained by extracting features from the third sample speech using the initial second acoustic feature extraction model. The acoustic features of the third sample speech are identified by an initial second classification model to obtain the predicted type of each sub-sample speech in the third sample speech; Based on the prediction type corresponding to each of the sub-samples in the third sample speech, the position of the sub-sample speech belonging to the first type in the third sample speech is obtained; The acoustic features belonging to the third sample speech are transformed by the initial feature transformation model to obtain the initial sample embedding features of the third sample speech. Based on the initial sample embedding features of the third sample speech, the predicted text corresponding to the third sample speech is obtained; Based on the position of the sub-sample speech belonging to the first type in the third sample speech, the initial sample embedding features of the third sample speech are transformed to obtain the sample mask embedding features of the third sample speech. The sample mask embedding features include semantically null sample sub-features, which are obtained based on the position of the sub-sample speech of the first type in the third sample speech. The prediction model is used to predict the semantically null sample sub-features in the sample mask embedding features to obtain the sample prediction embedding features; Based on the similarity between the multiple mapping embedding features and the sample prediction embedding features, the sample mapping subtext is obtained; Based on the correspondence and the sample mapping subtext, the sample candidate subtext is obtained; Based on the candidate sample texts, a third sample text is obtained; The acoustic features of the third sample speech and the third sample subtext are analyzed using the large language model to obtain the converted text corresponding to the third sample speech; Based on the differences between the predicted types of the fifth label and each of the sub-sample speech in the third sample speech, the differences between the sixth label and the predicted text, and the differences between the sixth label and the converted text corresponding to the third sample speech, the model parameters of the initial second acoustic feature extraction model, the model parameters of the initial second classification model, and the initial feature conversion model are adjusted to obtain the second acoustic feature extraction model, the second classification model, and the feature conversion model.

10. The method according to claim 8 or 9, characterized in that, If multiple sub-speech elements in the multi-type speech correspond to the same semantic meaning, then the step of predicting the features corresponding to the position in the mask embedding features using a prediction model to obtain predicted embedding features includes: In the mask embedding features, the sub-features corresponding to multiple sub-speechs with the same semantic meaning are merged to obtain merged embedding features; Based on the merged embedding features, the prediction model is used to predict the sub-features in the merged embedding features that have no actual semantic meaning, thereby obtaining the predicted embedding features.

11. The method according to any one of claims 8-10, characterized in that, The method further includes: Obtain the initial text belonging to the second type, wherein the initial text includes multiple sub-initial texts; Translate the target sub-initial text among the plurality of sub-initial texts to obtain the target translated text belonging to the first type; Replace the target sub-initial text in the initial text with the target translated text to obtain the sample text; Based on the sample text, the third sample speech is obtained.

12. The method according to any one of claims 1-11, characterized in that, The type is used to describe the language type, the degree of information compression, or the importance of information.

13. A speech-to-text device, characterized in that, The device includes: an acquisition unit, a feature extraction unit, an identification unit, and an analysis unit; The acquisition unit is used to acquire multiple types of speech, which include a first sub-speech belonging to a first type and a second sub-speech belonging to a second type. In the multiple types of speech, the first number of first sub-speech belonging to the first type is less than the second number of second sub-speech belonging to the second type. The first type and the second type are different types. The feature extraction unit is used to extract features from the multiple types of speech to obtain the acoustic features of the multiple types of speech. The recognition unit is used to perform type recognition on the multiple types of speech to obtain a first sub-text belonging to the first type, wherein the first sub-text is the text corresponding to the first sub-speech of the first type in the multiple types of speech; The analysis unit is used to analyze the acoustic features of the multiple speech types and the first sub-text using a large language model to obtain the converted text corresponding to the multiple speech types. The converted text includes the text corresponding to the first sub-speech and the text corresponding to the second sub-speech.

14. A computer device, characterized in that, The computer device includes a processor and memory: The memory is used to store computer programs and to transfer the computer programs to the processor; The processor is configured to perform the method according to any one of claims 1-12 according to the computer program.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method according to any one of claims 1-12.

16. A computer program product comprising a computer program, characterized in that, When it is run on a computer device, it causes the computer device to perform the method described in any one of claims 1-12.

Images