Speech processing method and apparatus, device, and storage medium

Abstract

Embodiments of the present disclosure relate to a speech processing method and apparatus, a device, and a storage medium. The method provided herein comprises: acquiring first speech content associated with a first language; determining, from a text set associated with the first language, first text content matching the first speech content, wherein the first text content corresponds to second text content associated with a second language; constructing an input sequence of a target model on the basis of the first text content, the second text content, and the first speech content; and on the basis of an output result of the target model, generating target text content and / or second speech content associated with the second language. In this way, the embodiments of the present disclosure can effectively improve the efficiency of speech processing.

Description

[0001]This disclosure relates generally to the field of computers, and particularly to a speech processing method, apparatus, device, and computer-readable storage medium. Background Art: Speech translation (ST) aims to translate source language speech into target language text, and is widely used in various scenarios such as conference speech translation, video subtitle translation, and AR (Augmented Reality) augmented translation. End-to-end simultaneous interpretation models incorporate a large amount of world knowledge into the model parameters, and in the aforementioned common scenarios, the translation performance is increasingly approaching the level of simultaneous interpreters. Summary of the Invention: In a first aspect of this disclosure, a speech processing method is provided, comprising: acquiring first speech content associated with a first language; determining first text content matching the first speech content from a text set associated with the first language, the first text content corresponding to second text content associated with a second language; constructing an input sequence for a target model based on the first text content, the second text content, and the first speech content; and generating target text content and / or second speech content associated with the second language based on the output of the target model. In a second aspect of this disclosure, an apparatus for speech processing is provided. The apparatus includes: a speech acquisition module configured to acquire first speech content associated with a first language; a speech matching module configured to determine first text content matching the first speech content from a set of texts associated with the first language, the first text content corresponding to second text content associated with a second language; an input construction module configured to construct an input sequence of a target model based on the first text content, the second text content, and the first speech content; and a content generation module configured to generate target text content and / or second speech content associated with a second language based on the output of the target model. In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. The instructions, when executed by the at least one processing unit, cause the device to perform the method of the first aspect. In a fourth aspect of this disclosure, a computer-readable storage medium is provided.The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect. It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Brief Description of the Drawings The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent when taken in conjunction with the accompanying drawings and with reference to the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: FIG1 shows a schematic diagram of an example environment in which some embodiments of this disclosure can be implemented; FIG2 shows a flowchart of a speech processing procedure according to some embodiments of this disclosure; FIG3 shows a structural diagram of a retrieval model according to some embodiments of this disclosure; FIG4 shows a structural diagram of a speech processing model according to some embodiments of this disclosure; FIG5 shows a structural diagram of a long short-term memory database according to some embodiments of this disclosure; FIG6 shows a schematic structural block diagram of an example device for speech processing according to some embodiments of this disclosure; and FIG7 shows a block diagram of a device in which multiple embodiments of this disclosure can be implemented. Detailed Description Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are illustrated in the accompanying drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure. It should be noted that the headings of any section / subsection provided herein are not restrictive. Various embodiments are described throughout this document, and embodiments of any type can be included under any section / subsection. Furthermore, embodiments described in any section / subsection can be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections. In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment".The term "some embodiments" should be understood as "at least some embodiments." Other explicit and implicit definitions may also be included below. The terms "first," "second," etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below. The embodiments of this disclosure may involve user data, data acquisition, and / or use. These aspects all comply with relevant laws, regulations, and related provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are carried out with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of the data or information that may be involved should be informed to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect. In this specification and the solutions in the embodiments, if personal information processing is involved, it will be processed under the premise of having a legal basis (e.g., obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be processed within the prescribed or agreed scope. Users refusing to process personal information beyond what is necessary for basic functions will not affect their use of those functions. As briefly mentioned above, speech translation (ST) aims to translate source language speech into target language text, and is widely used in various scenarios such as conference speech translation, video subtitle translation, and AR (Augmented Reality) augmented translation. End-to-end simultaneous interpretation models incorporate a large amount of world knowledge into their model parameters, and their translation performance in the aforementioned general scenarios is increasingly approaching the level of simultaneous interpreters. Traditional translation models may make translation errors when dealing with domains, specialties, and real-time data not covered during training. For these special terms or terms that appear in real time, translation models cannot quickly incorporate this knowledge into their model parameters. Embodiments of this disclosure propose a scheme for speech processing.According to various embodiments of this disclosure, a first speech content associated with a first language is obtained; a first text content matching the first speech content is determined from a text set associated with the first language, the first text content corresponding to second text content associated with a second language; an input sequence for a target model is constructed based on the first text content, the second text content, and the first speech content; and target text content and / or second text content associated with the second language are generated based on the output of the target model. In this way, embodiments of this disclosure can improve the quality of real-time speech translation by retrieving text content matching the speech content. Various example implementations of this scheme are further described in detail below with reference to the accompanying drawings. Example Environment Figure 1 shows a schematic diagram of an example environment 100 in which embodiments of this disclosure can be implemented. As shown in Figure 1, the example environment 100 may include an electronic device 110 and a speech processing model 136. In the environment 100 of Figure 1, the electronic device 110 may receive first speech content 130 from a user 140. Further, the electronic device 110 may invoke the speech processing model 136 to process the first speech content 130, obtaining target text content and / or second speech content 120 associated with a second language. In some embodiments, the speech processing model 136 includes at least a language model, a speech encoder, a converter, etc. The electronic device 110 may invoke the speech encoder in the speech processing model 136 to encode the first speech content into a speech feature representation. The electronic device 110 utilizes the language model in the speech processing model 136 to generate second speech content and / or target text content based on the speech feature representation, the first text content, and the second text content. In other embodiments, the speech processing model may run on a local device or a remote device. In some embodiments, the electronic device 110 may include various types of computing systems / servers capable of providing computing power, and the electronic device 110 may include a terminal device.Such terminal devices can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), voice / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. Electronic device 110 may, for example, include various types of computing systems / servers capable of providing computing power, such as mainframes, edge computing nodes, computing devices in cloud environments, virtual machines, etc. Although shown as a single device, electronic device 110 may include multiple physical devices. It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure. Some exemplary embodiments of this disclosure will continue to be described below with reference to the accompanying drawings. Figure 2 illustrates a flowchart of a speech processing procedure 200 according to some embodiments of the present disclosure. Procedure 200 can be implemented at an electronic device 110. Procedure 200 is described below with reference to Figure 1. As shown in Figure 2, at block 210, the electronic device 110 can acquire first speech content associated with a first language. In some embodiments, the first speech content can be input speech content from user 140 acquired by the electronic device 110 in real time, or it can be historical speech content. As an example, as shown in Figure 4, the speech processing model 136 can include a language encoding model 410, a converter 420 (optional), and a language model 430. In some embodiments, the electronic device 110 can acquire speech segments from a target speech stream as the first speech content. Taking a simultaneous interpretation scenario as an example, the target speech stream can be associated with the speaker's speech content. The electronic device 110 can, for example, determine speech segments of a certain duration from the target speech stream for translation. In some embodiments, such speech segments can have a preset duration, or such speech segments can be segmented based on semantic information of the speech content.In box 220, electronic device 110 determines first text content that matches the first speech content from a text set associated with a first language, the first text content corresponding to second text content associated with a second language. In some embodiments, electronic device 110 may, for example, utilize retrieval model 300 as shown in FIG3 to determine the first text content that matches the first speech content. As an example, as shown in FIG3, the first speech content 305 is, for example, "if it does not conform to the Ising model", and the text set may include, for example, "Ising Model", "Large Language Model - LLM", "Long Short-Term Memory Model - LSTM". The first text content that matches the first speech content is "Ising Model", and the second text content is "Ising Model". In some embodiments, electronic device 110 may utilize an audio encoder to process the first speech content to determine a speech feature representation. As shown in FIG3, retrieval model 300 includes an audio encoder 310-1, a text encoder 310-2, and a feature fusion module 320. The input to retrieval model 300 includes the first speech content 305 and the text content. After receiving the first speech content 305, the electronic device 110 calls the audio encoder 310-1 to encode the input speech content (i.e., "if it does not conform to the Ising model") to obtain a speech feature representation (also known as a speech vector). In some embodiments, the electronic device 110 uses a text encoder to generate multiple text feature representations corresponding to the text set. As an example, as shown in Figure 3, the text content includes multiple first text contents and multiple second text contents corresponding to the first text contents. The text contents may include, for example, "Ising Model", "Large Language Model - LLM", "Long Short-Term Memory Model - LSTM", etc. Chinese text can be set as the first text content, and English text can be set as the second text content. The electronic device 110 can input the first text content and the second text content to the text encoder 310-2 respectively to obtain the first text feature representation corresponding to the first text content and the second text feature representation corresponding to the second text content. Furthermore, the electronic device 110 can determine the first text content that matches the first speech content from the text set based on the comparison of speech feature representation and multiple text feature representations corresponding to the text set.In some embodiments, the electronic device 110 can provide a speech feature representation and multiple text feature representations to the feature fusion module to determine the correlation between the speech feature representation and each text feature representation. As shown in FIG3, the feature fusion module 320 includes: an attention layer 320-1, a pooling layer 320-2, and a linear projection layer 320-3. The electronic device 110 can input a speech feature representation, a first text feature representation, and a second text feature representation to the feature fusion module 320. After receiving the speech feature representation, the first text feature representation, and the second text feature representation, the attention layer 320-1 can update the speech feature representation, the first text feature representation, and the second text feature representation based on a multi-head attention mechanism to obtain a first intermediate feature table. Further, since the first intermediate feature representation is relatively complex, the pooling layer 320-2 can compress the first intermediate feature representation to obtain a second intermediate feature representation. Furthermore, the linear projection layer 320-3 in the feature fusion module 320 will obtain an output result 330 based on the second intermediate feature representation. Output 330 is used to determine the correlation between the speech feature representation and the corresponding text feature representation. Finally, the electronic device 110 can determine the first text content that matches the first speech content from the text set based on multiple correlations. The correlation can represent the probability of each text feature representation appearing in the speech feature representation. As an example, as shown in Figure 3. For example, the input speech content is "If it does not conform to the Ising model", and the input text content includes: "Ising Model", "Large Language Model - LLM", "Long Short-Term Memory Model - LSTM". After the retrieval model 300 processes the input speech content and text content, it obtains the probability of each text content appearing in the speech content. Examples of probabilities are: Ising Model: 0.85, Large Language Model: 0.1, Long Short-Term Memory Model: 0.03. The electronic device 110 can thus determine the first text content that matches the input speech content. A retrieval model identifies first text content that matches the first speech content. Based on this first text content, second text content in the second language is determined, thus aligning the first speech content with the second text content. In subsequent speech-to-text conversion, this allows for the accurate conversion of portions of the first speech content into the second text content, improving the overall accuracy of the speech-to-text conversion.In box 230, electronic device 110 constructs an input sequence for a target model based on first text content, second text content, and first speech content. As an example, the target model here can be a language model. For instance, electronic device 110 can construct an input sequence for a language model using "Ising Model" as the first text content, "Ismg Model" as the second text content, and speech data of "If it does not conform to the Ising Model, then" as the first speech content. In some embodiments, the input sequence includes a sequence portion corresponding to the first speech content, which is constructed based on a speech feature representation generated by an audio encoder. It should be understood that the input sequence also includes a first text feature representation and a second text feature representation. In some embodiments, electronic device 110 can, for example, utilize the speech processing model 400 shown in FIG. 4 to generate target text content and / or second speech content associated with the second speech. As an example, as shown in FIG. 4, the speech processing model 400 includes an audio encoder 410, an audio adapter 420, and a language model 430 (i.e., the target model). After receiving the first text content, the second text content, and the first speech content 405, the electronic device 110 encodes the first speech content 405 into a speech feature representation using the audio encoder 310. Further, the electronic device 110 can call the feature converter 420 to convert the speech feature representation into feature dimensions matching the speech model 430. The electronic device 110 also encodes the first and second text content to obtain first and second text feature representations. Finally, the electronic device 110 constructs an input sequence based on the converted speech feature representation, the first text feature representation, and the second text feature representation. In some examples, the encoder for encoding the text content can be a simpler text encoder, such as a word embedding encoder, to shorten the overall speech processing time. In some examples, the audio encoder 410 and the audio encoder 310-1 can share model parameters. For example, the audio encoder 410 and the audio encoder 310-1 are the same encoder. In box 240, the electronic device 110 can generate target text content and / or second speech content associated with the second language based on the output of the target model.As an example, as shown in Figure 4, after obtaining the input sequence, the electronic device 110 can input the input sequence into the language model 430 to obtain target text content and / or second speech content associated with the second language. Based on the above method, the embodiments of this disclosure do not require additional secondary retrieval of the input first speech content during speech processing, significantly shortening the speech encoding time and thus reducing speech processing latency. In some embodiments, the electronic device 110 can use the decoder to process the output of the language model 430 to generate target text content associated with the second language. The output of the language model 430 includes a text token sequence and / or a speech token sequence. As shown in Figure 4, after the speech model 430 processes the input sequence, it obtains a corresponding output result 440. The input result 440 can be a text token sequence and / or a speech token sequence corresponding to the input sequence. In some embodiments, if the obtained result is target text content, the electronic device 110 can generate second speech content associated with the second language based on the target text content. Specifically, the electronic device 110 can use a decoder to process the text token sequence to obtain the corresponding target text content, and then convert the target text content into second speech content corresponding to the target text content. In some embodiments, the electronic device 110 can provide the second speech content independently of the target speech stream. Specifically, if the obtained result is a speech token sequence, the electronic device 110 can call the decoder to decode the speech token sequence and directly obtain the second speech content. Based on this, the electronic device 110 can translate the first speech content associated with the first language into the second speech content associated with the second language. In some embodiments, the text set includes at least a first set of text content and a second set of text content. The first set of text content is independent of the real-time scenario of the first speech content, and the second set of text content is determined based on the scenario information associated with the first speech content. As an example, as shown in FIG5, the electronic device 110 can also pre-build a long short-term memory database 500 (also known as a knowledge base). The long short-term memory database 500 stores terminology data (i.e., the first set of text content) and unique vocabulary data (i.e., the second set of text content). The terminology database 510 stores terminology data, which may include Chinese terms in different professional fields, as well as the corresponding English translations of the Chinese terms.For example, "Long Short-Term Memory Model - LSTM" and "Ising Model". In some examples, terminology data is stored long-term in terminology database 510 to provide corresponding text references during speech processing. In some embodiments, the first speech content is associated with a target scene, and the scene information includes document information associated with the target scene and descriptive information associated with at least one participant in the target scene. As an example, as shown in FIG5, when electronic device 110 receives a real-time file 520 associated with the current scene, it can call a unique vocabulary extractor 530 to extract at least one unique vocabulary 540 from the real-time file and store at least one unique vocabulary 540 in the long short-term memory database 500. For example, the attendees of a meeting are Zhang San, Li Si, and Wang Wu, and the English names of Zhang San, Li Si, and Wang Wu are A, B, and C, respectively. The unique vocabulary extractor will extract the Chinese and English names of Zhang San, Li Si, and Wang Wu and store the extraction results as unique vocabulary in the long short-term memory database 500. The extraction results are, for example, “Zhang San-A”, “Li Si-B”, and “Wang Wu-C”. When changing the usage scenario of the speech processing model 400, the electronic device 110 will extract at least one unique word from the real-time file based on the changed usage scenario. The change of usage scenario can be, for example, from an academic conference scenario to a business conference scenario. Based on some embodiments of the present invention, by constructing a long short-term memory database, both commonly used terminology data and corresponding unique words can be extracted based on the current scenario, which can effectively improve the accuracy of speech processing. Based on the process described above, the embodiments of this disclosure can improve the quality of real-time speech translation by retrieving text content that matches the speech content. Example Apparatus and Device The embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 6 shows a schematic structural block diagram of an example apparatus 600 for speech processing according to some embodiments of this disclosure. The apparatus 600 can be implemented as or included in an electronic device. The various modules / components in the apparatus 600 can be implemented by hardware, software, firmware, or any combination thereof.As shown in Figure 6, the device 600 includes a speech acquisition module 610 configured to acquire first speech content associated with a first language; a speech matching module 620 configured to determine first text content matching the first speech content from a text set associated with the first language, wherein the first text content corresponds to second text content associated with a second language; an input construction module 630 configured to construct an input sequence for a target model based on the first text content, the second text content, and the first speech content; and a content generation module 640 configured to generate target text content and / or second speech content associated with the second language based on the output of the target model. In some embodiments, the speech matching module 620 is further configured to: process the first speech content using an audio encoder to determine speech feature representations; and determine first text content matching the first speech content from the text set based on a comparison of the speech feature representations and multiple text feature representations corresponding to the text set. In some embodiments, the speech matching module 620 is further configured to: generate multiple text feature representations corresponding to a text set using a text encoder; provide the speech feature representation and the multiple text feature representations to the feature fusion module to determine the correlation between the speech feature representation and each text feature representation; and determine, based on the multiple correlations, a first text content matching the first speech content from the text set. In some embodiments, the feature fusion module includes: an attention layer configured to update the speech feature representation and the corresponding text feature representation based on a multi-head attention mechanism to determine a first intermediate feature representation; a pooling layer configured to compress the first intermediate representation into a second intermediate feature representation; and a linear mapping layer configured to determine the correlation between the speech feature representation and the corresponding text feature representation based on the second intermediate feature representation. In some embodiments, the input sequence includes a sequence portion corresponding to the first speech content, the sequence portion being constructed based on the speech feature representation generated by the audio encoder. In some embodiments, the target model is a language model, the output of the language model includes a text token sequence, and the text generation module is further configured to: process the text token sequence using a decoder to generate target text content associated with a second language.In some embodiments, the device 600 further includes a speech generation module configured to: generate second speech content associated with a second language based on target text content. In some embodiments, the text set includes at least one of the following: a first set of text content, the first set of text content being independent of the real-time scene of the first speech content; and a second set of text content, the second set of text content being determined based on scene information associated with the first speech content. In some embodiments, the first speech content is associated with a target scene, and the scene information includes at least one of the following: document information associated with the target scene; and descriptive information associated with at least one participant in the target scene. In some embodiments, the speech acquisition module is further configured to: acquire speech segments in a target speech stream as the first speech content. In some embodiments, the device 600 further includes a second speech content providing module 650 configured to: provide the second speech content independently of the target speech stream. FIG7 shows a block diagram of a computing device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the computing device 700 shown in FIG7 is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. The computing device 700 shown in Figure 7 can be used to implement the electronic device in Figure 1. As shown in Figure 7, the computing device 700 is in the form of a general-purpose computing device. Components of the computing device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage devices 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the computing device 700. The computing device 700 typically includes multiple computer storage media. Such media can be any removable media accessible to the computing device 700, including but not limited to volatile and non-volatile media, removable and non-removable media.Memory 720 may be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 730 may be a removable or non-removable medium and may include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within computing device 700. Computing device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 7, disk drives for reading or writing from removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading or writing from removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules configured to perform various methods or actions according to various embodiments of the present disclosure. The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the computing device 700 may be implemented as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 700 can operate in a networked environment using a logical connection to one or more other servers, networked personal computers (PCs), or another network node. The input device 750 may be one or more input devices, such as a mouse, keyboard, trackball, remote control, etc. The output device 760 may be one or more output devices, such as a monitor, projector, television, etc. The computing device 700 can also communicate with one or more external devices (not shown) via the communication unit 740 as needed. These external devices, such as storage devices, can communicate with one or more devices that enable user interaction with the computing device 700, or with any device (e.g., a network interface card, modem, etc.) that enables the computing device 700 to communicate with one or more other computing devices. Such communication can be performed via an input / output (I / O) interface (not shown).According to exemplary implementations of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transient computer-readable medium and includes computer-executable instructions that are executed by a processor to implement the methods described above. Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, these instructions create means for implementing the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. These computer-readable program instructions can also be stored in a computer-readable storage medium. These instructions cause a computer, programmable data processing device, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram. Computer-readable program instructions can be loaded onto a computer, other programmable data processing device, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing device, or other device to produce a computer-implemented process, thereby causing the instructions executing on the computer, other programmable data processing device, or other device to implement the functions / actions specified in one or more blocks of a flowchart and / or block diagram. The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function.In some alternative implementations, the functions marked in the boxes may occur in a different order than those shown in the figures. For example, two consecutive boxes may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and combinations of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified functions or actions, or using a combination of dedicated hardware and computer instructions. Various implementations of this disclosure have been described above. The foregoing description is exemplary and not exhaustive, and is not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

CLAIM 1. A speech processing method, comprising: obtaining a first speech content associated with a first language; determining a first text content matching the first speech content from a text set associated with the first language, the first text content corresponding to a second text content associated with a second language; constructing an input sequence of a target model based on the first text content, the second text content, and the first speech content; and generating a target text content and / or a second speech content associated with the second language based on an output result of the target model.

2. The method of claim 1, wherein determining first text content that matches the first speech content from a set of text associated with the first language comprises: processing the first speech content with an acoustic encoder to determine a speech feature representation; and determining the first text content matching the first speech content from the text set based on a comparison of the speech feature representation and a plurality of text feature representations corresponding to the text set.

3. The method of claim 2, wherein determining the first text content from the text aggregation based on a comparison of the speech feature representation and a plurality of text feature representations corresponding to the text aggregation comprises: generating the plurality of text feature representations corresponding to the text set with a text encoder; providing the speech feature representation and the plurality of text feature representations to a feature fusion module to determine a correlation degree between the speech feature representation and each text feature representation; and determining the first text content matching the first speech content from the text set based on the plurality of correlation degrees.

4. The method of claim 3, wherein the feature fusion module comprises: an attention layer configured to update the speech feature representation and a corresponding text feature representation based on a multi-head attention mechanism to determine a first intermediate feature representation; a pooling layer configured to compress the first intermediate representation into a second intermediate feature representation; a linear mapping layer configured to determine the correlation degree between the speech feature representation and the corresponding text feature representation based on the second intermediate feature representation.

5. The method of claim 2, wherein the input sequence comprises a sequence portion corresponding to the first speech content, the sequence portion being constructed based on the speech feature representation generated by the acoustic encoder.

6. The method of claim 1, wherein the target model is a language model, wherein the output result of the language model comprises a text token sequence, and generating a target text content associated with the second language based on the output result of the target model comprises: processing the text token sequence with a decoder to generate the target text content associated with the second language.

7. The method of claim 1, wherein generating second speech content associated with the second language comprises: generating the second speech content associated with the second language based on the target text content. 8.The method of claim 1, wherein the set of texts comprises at least one of: a first set of text content independent of a real-time context of the first audio content; a second set of text content determined based on context information associated with the first audio content. 9.The method of claim 6, wherein the first audio content is associated with a target context, and the context information comprises at least one of: document information associated with the target context; description information associated with at least one participant of the target context. 10.The method of claim 1, wherein obtaining the first audio content associated with a first language comprises: obtaining a speech segment in a target audio stream as the first audio content. The first voice content includes: providing the second audio content independent of the target audio stream.

11. The method of claim 10, further comprising: a speech obtaining module configured to obtain a first audio content associated with a first language; 12. An apparatus for speech processing, comprising: a speech matching module configured to determine a first text content matching the first audio content from a set of texts associated with the first language, the first text content corresponding to a second text content associated with a second language; an input constructing module configured to construct an input sequence of a target model based on the first text content, the second text content, and the first audio content; and a content generating module configured to generate a target text content and / or a second audio content associated with the second language based on an output result of the target model.

13. An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit to cause the electronic device to perform the method according to any one of claims 1-11. 14.A computer-readable storage medium having stored thereon a computer program, the computer program being executable by a processor to implement the method according to any one of claims 1-11.

Images