Voice processing method and device, equipment and storage medium
By matching voice content from the text collection and building the input sequence of the target model, the problem of the decline in translation quality when traditional translation models deal with no domain and real-time data is solved, and efficient and accurate real-time voice translation is achieved.
Patent Information
- Application Number
- CN202480003551.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2025-03-25
AI Technical Summary
Traditional translation models are difficult to quickly and accurately translate terms when processing uninvolved fields, specialties and real-time data, resulting in a decline in translation quality.
By obtaining the voice content associated with the first language and determining the matching text content from the text set associated with the first language, an input sequence of the target model is constructed, and the target text content and/or the second voice content associated with the second language are finally generated.
Improve the quality of real-time voice translation, reduce voice encoding time, and reduce voice processing delay.
Smart Images
Figure CN119698655A_ABST
Abstract
Description
Technical Field
[0001] The exemplary embodiments of the present disclosure generally relate to the field of computers, and in particular, to a speech processing method, apparatus, device, and computer-readable storage medium. Background Art
[0002] Speech Translation (ST) aims to translate source language speech into target language text, and is widely used in various scenarios such as conference speech translation, video subtitle translation, and AR (Augmented Reality, AR) enhanced translation. The end-to-end simultaneous interpretation model incorporates a large amount of world knowledge into the model parameters, and the performance of translation in the above common scenarios is getting closer and closer to the level of simultaneous interpreters. Summary of the invention
[0003] In a first aspect of the present disclosure, a speech processing method is provided, comprising: acquiring first speech content associated with a first language; determining first text content matching the first speech content from a text set associated with the first language, the first text content corresponding to second text content associated with a second language; constructing an input sequence of a target model based on the first text content, the second text content and the first speech content; and generating target text content and / or second speech content associated with the second language based on an output result of the target model.
[0004] In a second aspect of the present disclosure, a device for speech processing is provided. The device includes: a speech acquisition module configured to acquire a first speech content associated with a first language; a speech matching module configured to determine a first text content matching the first speech content from a text set associated with the first language, the first text content corresponding to a second text content associated with a second language; an input construction module configured to construct an input sequence of a target model based on the first text content, the second text content and the first speech content; and a content generation module configured to generate a target text content and / or a second speech content associated with the second language based on an output result of the target model.
[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.
[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.
[0007] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:
[0009] Figure 1 A schematic diagram showing an example environment in which some embodiments of the present disclosure can be implemented;
[0010] Figure 2 A flowchart showing a speech processing process according to some embodiments of the present disclosure is shown;
[0011] Figure 3 shows a structural diagram of a retrieval model according to some embodiments of the present disclosure;
[0012] Figure 4 shows a structural diagram of a speech processing model according to some embodiments of the present disclosure;
[0013] Figure 5 shows a structural diagram of a long short-term memory database according to some embodiments of the present disclosure;
[0014] Figure 6 A schematic structural block diagram showing an example apparatus for speech processing according to some embodiments of the present disclosure; and
[0015] Figure 7 A block diagram of a device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION
[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.
[0017] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section / subsection. In addition, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.
[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.
[0019] The embodiments of the present disclosure may involve user data, data acquisition and / or use, etc. These aspects are subject to the corresponding laws, regulations and relevant provisions. In the embodiments of the present disclosure, all data collection, acquisition, processing, processing, forwarding, use, etc. are carried out on the premise that the user knows and confirms. Accordingly, when implementing each embodiment of the present disclosure, the type, scope of use, usage scenario, etc. of the data or information that may be involved should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with the relevant laws and regulations. The specific notification and / or authorization method can vary according to the actual situation and application scenario, and the scope of the present disclosure is not limited in this respect.
[0020] In this specification and the embodiments, if personal information processing is involved, it will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or it is necessary to perform a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.
[0021] As briefly mentioned above, speech translation (ST) aims to translate source language speech into target language text, and is widely used in various scenarios such as conference speech translation, video subtitle translation, and AR (Augmented Reality) enhanced translation. The end-to-end simultaneous interpretation model incorporates a large amount of world knowledge into the model parameters, and the performance of translation in the above common scenarios is getting closer and closer to the level of simultaneous interpreters.
[0022] Traditional translation models will make mistakes in translating terms when dealing with fields, professions, and real-time data that were not covered during training. For these special terms or terms that appear in real time, the translation model cannot quickly incorporate this knowledge into the model parameters.
[0023] The embodiments of the present disclosure propose a solution for speech processing. According to various embodiments of the present disclosure, a first speech content associated with a first language is obtained; a first text content matching the first speech content is determined from a text set associated with the first language, and the first text content corresponds to a second text content associated with a second language; an input sequence of a target model is constructed based on the first text content, the second text content, and the first speech content; and a target text content and / or second text content associated with the second language is generated based on an output result of the target model.
[0024] In this way, embodiments of the present disclosure can improve the quality of real-time speech translation by retrieving text content that matches the speech content.
[0025] Various example implementations of the solution are described in detail below in conjunction with the accompanying drawings.
[0026] Example Environment
[0027] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 As shown, example environment 100 may include electronic device 110 and speech processing model 136 .
[0028] exist Figure 1 In the environment 100, the electronic device 110 may receive the first voice content 130 from the user 140. Further, the electronic device 110 may call the voice processing model 136 to process the first voice content 130 to obtain the target text content and / or the second voice content 120 associated with the second language.
[0029] In some embodiments, the speech processing model 136 includes at least a language model, a speech encoder, a converter, etc. The electronic device 110 can call the speech encoder in the speech processing model 136 to encode the first speech content into a speech feature representation. The electronic device 110 uses the language model in the speech processing model 136 to generate the second speech content and / or the target text content based on the speech feature representation, the first text content and the second text content. In other embodiments, the speech processing model can run on a local device or a remote device.
[0030] In some embodiments, the electronic device 110 may include various types of computing systems / servers that can provide computing capabilities, and the electronic device 110 may include a terminal device. Such a terminal device may be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), a voice / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. The electronic device 110 may, for example, include various types of computing systems / servers that can provide computing capabilities, such as a mainframe, an edge computing node, a computing device in a cloud environment, a virtual machine, and the like. Although shown as a single device, the electronic device 110 may include multiple physical devices.
[0031] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.
[0032] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.
[0033] Speech Processing Example Procedure
[0034] Figure 2 FIG. 2 is a flow chart of a speech processing process 200 according to some embodiments of the present disclosure. The process 200 may be implemented at the electronic device 110. Figure 1 The process 200 is described below.
[0035] like Figure 2 As shown, in block 210, the electronic device 110 may acquire first voice content associated with the first language. In some embodiments, the first voice content may be input voice content of the user 140 acquired by the electronic device 110 in real time, or may be historical voice content.
[0036] As an example, Figure 4 As shown, the speech processing model 136 may include a language encoding model 410 , a converter 420 (optional), and a language model 430 .
[0037] In some embodiments, the electronic device 110 may obtain a voice segment in the target voice stream as the first voice content.
[0038] Taking the simultaneous interpretation scenario as an example, the target voice stream can be associated with the voice content of the speaker. The electronic device 110 can, for example, determine a voice segment with a certain duration from the target voice stream for translation. In some embodiments, such a voice segment can have a preset duration, or such a voice segment can also be segmented based on the semantic information of the voice content.
[0039] In block 220 , the electronic device 110 determines first text content that matches the first voice content from a text collection associated with the first language, the first text content corresponding to second text content associated with the second language.
[0040] In some embodiments, the electronic device 110 may use, for example, Figure 3 The illustrated retrieval model 300 determines a first text content that matches a first speech content.
[0041] As an example, Figure 3 As shown, the first voice content 305 is, for example, "If it does not conform to the Ising Model", and the text set may include, for example: "Ising Model", "Large Language Model-LLM", "Long Short-Term Memory Model-LSTM". The first text content matching the first voice content is "Ising Model", and the second text content is "Ising Model".
[0042] In some embodiments, the electronic device 110 may process the first speech content using an audio encoder to determine a speech feature representation.
[0043] like Figure 3 As shown, the retrieval model 300 includes an audio encoder 310-1, a text encoder 310-2 and a feature fusion module 320. The input of the retrieval model 300 includes a first voice content 305 and a text content. After the electronic device 110 receives the first voice content 305, it calls the audio encoder 310-1 to encode the input voice content (i.e., "if it does not conform to the Ising model") to obtain a voice feature representation (also called a voice vector).
[0044] In some embodiments, the electronic device 110 generates a plurality of text feature representations corresponding to a text collection using a text encoder.
[0045] As an example, Figure 3As shown, the text content includes a plurality of first text contents and a plurality of second text contents corresponding to the first text contents. The text content may include, for example: "Ising Model", "Large Language Model-LLM", "Long Short-Term Memory Model-LSTM". The Chinese text may be set as the first text content, and the English text may be set as the second text content. The electronic device 110 may input the first text content and the second text content into the text encoder 310-2 respectively to obtain a first text feature representation corresponding to the first text content and a second text feature representation corresponding to the second text content.
[0046] Furthermore, the electronic device 110 may determine the first text content matching the first voice content from the text collection based on a comparison between the voice feature representation and a plurality of text feature representations corresponding to the text collection.
[0047] In some embodiments, the electronic device 110 may provide the speech feature representation and the plurality of text feature representations to the feature fusion module to determine the degree of association between the speech feature representation and the text feature representations.
[0048] like Figure 3 As shown, the feature fusion module 320 includes: an attention layer 320-1, a pooling layer 320-2, and a linear mapping layer 320-3. The electronic device 110 can input a speech feature representation, a first text feature representation, and a second text feature representation to the feature fusion module 320. After the feature fusion module 320 receives the speech feature representation, the first text feature representation, and the second text feature representation, the attention layer 320-1 can update the speech feature representation, the first text feature representation, and the second text feature representation based on the multi-head attention mechanism to obtain a first intermediate feature representation.
[0049] Furthermore, since the first intermediate feature representation is relatively complex, the pooling layer 320 - 2 may compress the first intermediate feature representation to obtain a second intermediate feature representation.
[0050] Furthermore, the linear projection layer 320-3 in the feature fusion module 320 obtains an output result 330 based on the second intermediate feature representation. The output result 330 is used to determine the correlation between the speech feature representation and the corresponding text feature representation.
[0051] Finally, the electronic device 110 may determine the first text content matching the first voice content from the text collection based on the multiple association degrees. The association degree may indicate the probability of each text feature representation appearing in the voice feature representation.
[0052] As an example, Figure 3As shown. For example, the input voice content is "If it does not conform to the Ising model", and the input text content includes: "Ising Model-Ising Model", "Large Language Model-LLM", "Long Short-Term Memory Model-LSTM". After the retrieval model 300 processes the input voice content and text content, it obtains the probability of each text content appearing in the voice content. Examples of probabilities are: Ising model: 0.85, large language model: 0.1, long short-term memory model: 0.03. The electronic device 110 can thereby determine the first text content that matches the input voice content.
[0053] The first text content matching the first voice content is determined by the retrieval model, and the second text content in the second language is determined based on the first text content, so that the first voice content corresponds to the second text content. In the subsequent voice conversion process, part of the first voice content can be accurately converted into the second text content, thereby improving the accuracy of the voice conversion.
[0054] In block 230, the electronic device 110 constructs an input sequence of a target model based on the first text content, the second text content and the first voice content. As an example, the target model here can be set to a language model.
[0055] For example, the electronic device 110 may use "Ising Model" as the first text content, "Ising Model" as the second text content, and the voice data of "If it does not conform to the Ising Model, then" as the first voice content to construct the input sequence of the language model.
[0056] In some embodiments, the input sequence includes a sequence portion corresponding to the first speech content, and the sequence portion is constructed based on the speech feature representation generated by the audio encoder. It should be understood that the input sequence also includes a first text feature representation and a second text feature representation.
[0057] In some embodiments, the electronic device 110 may use, for example, Figure 4 The illustrated speech processing model 400 is used to generate target text content associated with the second speech and / or the second speech content.
[0058] As an example, Figure 4As shown, the speech processing model 400 includes an audio encoder 410, an audio adapter 420 and a language model 430 (i.e., a target model). After receiving the first text content, the second text content and the first voice content 405, the electronic device 110 encodes the first voice content 405 into a speech feature representation through the audio encoder 310. Further, the electronic device 110 can call the feature converter 420 to convert the speech feature representation into a feature dimension that matches the speech model 430. The electronic device 110 also encodes the first text content and the second text content to obtain a first text feature representation and a second text feature representation. Finally, the electronic device 110 constructs an input sequence based on the converted speech feature representation, the first text feature representation and the second text feature representation.
[0059] In some examples, the encoder for encoding the text content can select a text encoder with a simpler structure such as word embedding to shorten the overall time of speech processing. In some examples, the audio encoder 410 and the audio encoder 310-1 can share model parameters. For example, the audio encoder 410 and the audio encoder 310-1 are the same encoder.
[0060] In block 240 , the electronic device 110 may generate target text content and / or second voice content associated with the second language based on the output result of the target model.
[0061] As an example, Figure 4 As shown, after obtaining the input sequence, the electronic device 110 may input the input sequence into the language model 430 to obtain the target text content and / or the second voice content associated with the second language.
[0062] Based on the above approach, the embodiment of the present disclosure does not need to perform a secondary search on the input first voice content during voice processing, which greatly shortens the voice encoding time and thus reduces the voice processing delay.
[0063] In some embodiments, the electronic device 110 may process the output result of the language model 430 using a decoder to generate target text content associated with the second language, the output result of the language model 430 including a text token sequence and / or a speech token sequence.
[0064] like Figure 4 As shown, after the speech model 430 processes the input sequence, a corresponding output result 440 is obtained, and the input result 440 may be a text token sequence and / or a speech token sequence corresponding to the input sequence.
[0065] In some embodiments, if the target text content is obtained, the electronic device 110 may generate a second voice content associated with the second language based on the target text content.
[0066] Specifically, the electronic device 110 may utilize a decoder to process the text token sequence to obtain corresponding target text content, and then convert the target text content into second voice content corresponding to the target text content.
[0067] In some embodiments, the electronic device 110 may provide the second voice content independently of the target voice stream.
[0068] Specifically, if a voice token sequence is obtained, the electronic device 110 may call a decoder to decode the voice token sequence to directly obtain the second voice content.
[0069] Based on this, the electronic device 110 may translate the first voice content associated with the first language into the second voice content associated with the second language.
[0070] In some embodiments, the text collection includes at least a first group of text content and a second group of text content, the first group of text content is independent of the real-time scene of the first voice content, and the second group of text content is determined based on scene information associated with the first voice content.
[0071] As an example, Figure 5 As shown, the electronic device 110 can also pre-build a long short-term memory database 500 (also referred to as a knowledge base). The long short-term memory database 500 stores terminology data (i.e., a first set of text content) and specific vocabulary data (i.e., a second set of text content). The terminology database 510 stores terminology data, which may include Chinese terms in different professional fields, and English translations corresponding to the Chinese terms. For example: "Long Short-Term Memory Model - LSTM" and "Ising Model - Ising Model". In some examples, terminology data will be stored in the terminology database 510 for a long time to provide corresponding text references during speech processing.
[0072] In some embodiments, the first voice content is associated with a target scene, and the scene information includes document information associated with the target scene; and description information associated with at least one participant of the target scene.
[0073] As an example, Figure 5As shown, when the electronic device 110 receives a real-time file 520 associated with the current scene, it can call a unique vocabulary extractor 530 to extract at least one unique vocabulary 540 in the real-time file, and store at least one unique vocabulary 540 in the long-term and short-term memory database 500. For example, the people attending a meeting are Zhang San, Li Si, and Wang Wu, and the English names of Zhang San, Li Si, and Wang Wu are A, B, and C respectively. The unique vocabulary extractor will extract the Chinese names and English names of Zhang San, Li Si, and Wang Wu, and store the extraction results as unique vocabulary in the long-term and short-term memory database 500. The extraction results are, for example: "Zhang San-A", "Li Si-B", and "Wang Wu-C". When the use scenario of the speech processing model 400 is changed, the electronic device 110 will extract at least one unique vocabulary in the real-time file based on the changed use scenario. The change of the use scenario can be, for example, from an academic conference scenario to a business conference scenario.
[0074] Based on some embodiments of the present invention, by constructing a long short-term memory database, commonly used terminology data is retained and corresponding unique vocabulary can be extracted based on the current scenario, which can effectively improve the accuracy of speech processing.
[0075] Based on the process described above, the embodiments of the present disclosure can improve the quality of real-time speech translation by retrieving text content that matches the speech content.
[0076] Example devices and equipment
[0077] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 6 A schematic structural block diagram of an example apparatus 600 for speech processing according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in an electronic device. Each module / component in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.
[0078] like Figure 6 As shown, the device 600 includes a speech acquisition module 610, which is configured to acquire a first speech content associated with a first language; a speech matching module 620, which is configured to determine a first text content that matches the first speech content from a text set associated with the first language, wherein the first text content corresponds to a second text content associated with a second language; an input construction module 630, which is configured to construct an input sequence of a target model based on the first text content, the second text content and the first speech content; and a content generation module 640, which is configured to generate a target text content and / or a second speech content associated with the second language based on an output result of the target model.
[0079] In some embodiments, the speech matching module 620 is further configured to: process the first speech content using an audio encoder to determine a speech feature representation; and determine a first text content that matches the first speech content from the text collection based on a comparison of the speech feature representation and multiple text feature representations corresponding to the text collection.
[0080] In some embodiments, the speech matching module 620 is further configured to: generate multiple text feature representations corresponding to a text collection using a text encoder; provide the speech feature representation and multiple text feature representations to a feature fusion module to determine the degree of association between the speech feature representation and each text feature representation; and determine, from the text collection, a first text content that matches the first speech content based on the multiple degrees of association.
[0081] In some embodiments, the feature fusion module includes: an attention layer, configured to update the speech feature representation and the corresponding text feature representation based on a multi-head attention mechanism to determine a first intermediate feature representation; a pooling layer, configured to compress the first intermediate representation into a second intermediate feature representation; and a linear mapping layer, configured to determine the correlation between the speech feature representation and the corresponding text feature representation based on the second intermediate feature representation.
[0082] In some embodiments, the input sequence includes a sequence portion corresponding to the first speech content, the sequence portion being constructed based on a speech feature representation generated by an audio encoder.
[0083] In some embodiments, the target model is a language model, the output result of the language model includes a text token sequence, and the text generation module is further configured to: process the text token sequence using a decoder to generate target text content associated with the second language.
[0084] In some embodiments, the apparatus 600 further includes a speech generation module configured to generate a second speech content associated with a second language based on the target text content.
[0085] In some embodiments, the text collection includes at least one of the following: a first group of text content, the first group of text content is independent of the real-time scene of the first voice content; and a second group of text content, the second group of text content is determined based on scene information associated with the first voice content.
[0086] In some embodiments, the first voice content is associated with a target scene, and the scene information includes at least one of the following: document information associated with the target scene; description information associated with at least one participant of the target scene.
[0087] In some embodiments, the voice acquisition module is further configured to: acquire a voice segment in the target voice stream as the first voice content.
[0088] In some embodiments, the apparatus 600 further includes a second voice content providing module 650 configured to provide the second voice content independently of the target voice stream.
[0089] Figure 7 1 shows a block diagram of a computing device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 7 The computing device 700 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 7 The computing device 700 shown may be used to implement Figure 1 Electronic devices in.
[0090] like Figure 7 As shown, computing device 700 is in the form of a general-purpose computing device. Components of computing device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be an actual or virtual processor and is capable of performing various processes according to a program stored in memory 720. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capabilities of computing device 700.
[0091] The computing device 700 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the computing device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the computing device 700.
[0092] The computing device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.
[0093] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the computing device 700 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Therefore, the computing device 700 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.
[0094] Input device 750 may be one or more input devices, such as a mouse, keyboard, trackball, remote control, etc. Output device 760 may be one or more output devices, such as a display, projector, television, etc. Computing device 700 may also communicate with one or more external devices (not shown) through communication unit 740 as needed, such as storage devices, one or more devices that allow a user to interact with computing device 700, or any device that allows computing device 700 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).
[0095] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.
[0096] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0097] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0098] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0099] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.
[0100] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.
Claims
1. A speech processing method, comprising: Acquire first speech content associated with a first language; Determining, from a text collection associated with the first language, first text content that matches the first speech content, the first text content corresponding to second text content associated with a second language; Constructing an input sequence of a target model based on the first text content, the second text content and the first voice content; as well as Based on the output result of the target model, target text content and / or second voice content associated with the second language is generated.
2. The method according to claim 1, wherein determining first text content matching the first speech content from a text collection associated with the first language comprises: Processing the first speech content using an audio encoder to determine a speech feature representation; as well as Based on a comparison between the speech feature representation and a plurality of text feature representations corresponding to the text collection, the first text content matching the first speech content is determined from the text collection.
3. The method according to claim 2, wherein determining the first text content from the text collection based on comparison of the speech feature representation and a plurality of text feature representations corresponding to the text collection comprises: Generate the plurality of text feature representations corresponding to the text set using a text encoder; Providing the speech feature representation and the plurality of text feature representations to a feature fusion module to determine a degree of association between the speech feature representation and each text feature representation; as well as Based on the multiple association degrees, the first text content matching the first voice content is determined from the text collection.
4. The method according to claim 3, wherein the feature fusion module comprises: an attention layer configured to update the speech feature representation and the corresponding text feature representation based on a multi-head attention mechanism to determine a first intermediate feature representation; A pooling layer is configured to compress the first intermediate representation into a second intermediate feature representation: A linear mapping layer is configured to determine the correlation between the speech feature representation and the corresponding text feature representation based on the second intermediate feature representation.
5. The method according to claim 2, wherein the input sequence includes a sequence portion corresponding to the first speech content, the sequence portion being constructed based on the speech feature representation generated by the audio encoder.
6. The method according to claim 1, wherein the target model is a language model, wherein the output result of the language model includes a text token sequence, and generating target text content associated with the second language based on the output result of the target model comprises: The sequence of text tokens is processed using a decoder to generate target text content associated with the second language.
7. The method of claim 1, wherein generating second speech content associated with the second language comprises: Based on the target text content, the second speech content associated with the second language is generated.
8. The method according to claim 1, wherein the text collection includes at least one of the following: A first set of text content, the first set of text content is independent of the real-time scenario of the first voice content; A second set of text content is determined based on scene information associated with the first voice content.
9. The method according to claim 6, wherein the first voice content is associated with a target scene, and the scene information includes at least one of the following: Document information associated with the target scenario; Descriptive information associated with at least one participant of the target scenario.
10. The method according to claim 1, wherein obtaining the first speech content associated with the first language comprises: A voice segment in the target voice stream is obtained as the first voice content.
11. The method according to claim 10, further comprising: The second voice content is provided independently of the target voice stream.
12. A device for speech processing, comprising: A speech acquisition module, configured to acquire first speech content associated with a first language; a speech matching module configured to determine, from a text set associated with the first language, first text content that matches the first speech content, the first text content corresponding to second text content associated with a second language; An input construction module, configured to construct an input sequence of a target model based on the first text content, the second text content and the first voice content; as well as The content generation module is configured to generate target text content and / or second voice content associated with the second language based on the output result of the target model.
13. An electronic device comprising: at least one processing unit; as well as At least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 11 when executed by the at least one processing unit.
14. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 11.