Context-based voice processing method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to a context-based voice processing method and device, equipment and a storage medium. The method provided by the invention comprises the following steps: acquiring training data, wherein the training data comprises a voice sample, context information associated with the voice sample and a labeling text corresponding to the voice sample; a first feature sequence is processed through a voice recognition model, a first output probability corresponding to the annotation text is determined, and the first feature sequence is constructed based on the voice sample and the context information; processing a second feature sequence by using the speech recognition model to determine a second output probability corresponding to the tagged text, the second feature sequence being constructed based on the speech sample and being independent of the context information; and determining training loss at least based on a difference between the first output probability and the second output probability to adjust parameters of the speech recognition model. In this way, the voice recognition accuracy can be improved.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein relate generally to the field of computers, and more particularly to context-based speech processing methods, apparatuses, devices, and computer-readable storage media. Background Technology

[0002] In recent years, with the rapid development of computer technology, more and more applications and platforms are designed to provide users with various services. For example, applications / platforms are designed to provide users with speech recognition services. Applications / platforms can, for example, use speech recognition systems (such as speech recognition models) to convert speech to text, generating text corresponding to the speech. Summary of the Invention

[0003] In a first aspect of this disclosure, a context-based speech processing method is provided. The method includes: acquiring training data, the training data including speech samples, context information associated with the speech samples, and labeled text corresponding to the speech samples; determining a first output probability corresponding to the labeled text by processing a first feature sequence using a speech recognition model, the first feature sequence being constructed based on the speech samples and the context information; determining a second output probability corresponding to the labeled text by processing a second feature sequence using the speech recognition model, wherein the second feature sequence is constructed based on the speech samples and is independent of the context information; and determining a training loss based at least on the difference between the first output probability and the second output probability to adjust the parameters of the speech recognition model.

[0004] In a second aspect of this disclosure, an apparatus for context-based speech processing is provided. The apparatus includes: an acquisition module configured to acquire training data, the training data including speech samples, context information associated with the speech samples, and labeled text corresponding to the speech samples; a first determination module configured to determine a first output probability corresponding to the labeled text by processing a first feature sequence using a speech recognition model, the first feature sequence being constructed based on the speech samples and the context information; a second determination module configured to determine a second output probability corresponding to the labeled text by processing a second feature sequence using a speech recognition model, wherein the second feature sequence is constructed based on the speech samples and is independent of the context information; and an adjustment module configured to determine a training loss based at least on the difference between the first output probability and the second output probability to adjust the parameters of the speech recognition model.

[0005] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram is shown of an example environment in which embodiments of the present disclosure may be implemented;

[0010] Figure 2 A flowchart illustrating an example process of context-based speech processing according to some embodiments of the present disclosure is shown;

[0011] Figure 3 A schematic diagram of an example framework for a speech recognition model according to some embodiments of the present disclosure is shown;

[0012] Figure 4 A schematic structural block diagram of an example apparatus for context-based speech processing according to some embodiments of the present disclosure is shown; and

[0013] Figure 5 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0014] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0015] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0016] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0017] The embodiments of this disclosure may involve user data, data acquisition, and / or use. All of these aspects comply with applicable laws, regulations, and relevant provisions. In the embodiments of this disclosure, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, in implementing the embodiments of this disclosure, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained in accordance with relevant laws and regulations through appropriate means. The specific methods of notification and / or authorization may vary depending on the actual situation and application scenario, and the scope of this disclosure is not limited in this respect.

[0018] In this specification and the embodiments, any processing of personal information will be carried out only under the premise of legality (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information other than that necessary for basic functions will not affect the user's use of basic functions.

[0019] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0020] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as an input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. The testing phase can sometimes be integrated into the training phase. In the application or inference phase, the trained model can be used to process actual model inputs based on the trained parameter values ​​to determine the corresponding model output.

[0021] As mentioned above, with the rapid development of computer technology, more and more applications and platforms are designed to provide users with various services. For example, applications / platforms are designed to provide users with speech recognition services. Applications / platforms can, for example, use speech recognition systems (such as speech recognition models) to convert speech to text, generating text corresponding to the speech. However, the text content generated by traditional speech recognition systems is not accurate enough.

[0022] Embodiments of this disclosure propose a context-based speech processing scheme. According to the scheme, training data is acquired, including speech samples, context information associated with the speech samples, and labeled text corresponding to the speech samples; a first output probability corresponding to the labeled text is determined by processing a first feature sequence using a speech recognition model, the first feature sequence being constructed based on the speech samples and context information; a second output probability corresponding to the labeled text is determined by processing a second feature sequence using the speech recognition model, wherein the second feature sequence is constructed based on the speech samples and is independent of the context information; and a training loss is determined, at least based on the difference between the first and second output probabilities, to adjust the parameters of the speech recognition model.

[0023] In this way, the embodiments of this disclosure can improve the accuracy of speech recognition based on contextual information.

[0024] The following section provides a detailed description of various example implementations of this scheme, with reference to the accompanying drawings.

[0025] Example Environment

[0026] Figure 1A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In environment 100, an electronic device 110 and a speech recognition model 136 are deployed. In some embodiments, the electronic device 110 receives target speech 130 from a user 140, and then the electronic device 110 invokes the speech recognition model 136 to generate a speech recognition result 120 based on the target speech 130.

[0027] In some embodiments, the speech recognition model 136 includes at least a language model, a speech coding model, a converter, etc. The electronic device 110 can use the speech coding model in the speech recognition model 136 to generate speech feature representations. The electronic device 110 uses the language model in the speech recognition model 136 to generate a speech recognition result 120 based on the speech feature representations and contextual information. In some embodiments, the speech recognition model can run on a local device or a remote device.

[0028] In some embodiments, electronic device 110 may include various types of computing systems / servers capable of providing computing power, and electronic device 110 may include terminal devices. Such terminal devices may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), voice / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. Electronic device 110 may, for example, include various types of computing systems / servers capable of providing computing power, such as mainframes, edge computing nodes, computing devices in cloud environments, virtual machines, etc. Although shown as a single device, electronic device 110 may include multiple physical devices.

[0029] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0030] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0031] Context-based speech processing

[0032] Figure 2 A flowchart of an example process 200 for context-based speech processing according to some embodiments of the present disclosure is shown. Process 200 can be implemented at electronic device 110. Reference is made below. Figure 1 To describe process 200.

[0033] like Figure 2 As shown in box 210, electronic device 110 can acquire training data, which includes speech samples, context information associated with the speech samples, and labeled text corresponding to the speech samples.

[0034] The following will be referenced Figure 3 The illustrated speech recognition model 136 is used to describe an example process of training a speech recognition model according to an embodiment of the present disclosure.

[0035] Figure 3 A schematic diagram of an example framework 300 for a speech recognition model 136 according to some embodiments of the present disclosure is shown. Figure 3 As shown in the example framework 300, the speech recognition model 136 may include a speech coding model 310, a converter 315 (optional), and a language model 320.

[0036] In some embodiments, reference Figure 3 During the training phase, the input information for the speech recognition model 136 may include training data. The training data may include target speech 130 (e.g., referred to as a speech sample during the training phase), contextual information 335 associated with the speech sample, and labeled text corresponding to the speech sample. As an example, the labeled text may be the text content corresponding to the speech sample.

[0037] In some embodiments, the text generation model may be provided with annotated text corresponding to the speech sample. Optionally, the text generation model may also be provided with historical annotated text of historical speech content associated with the speech sample. Further, the text generation model may generate descriptive text about the annotated text. Further, the text generation model may construct contextual information corresponding to the speech sample based on the descriptive text. As an example, the descriptive text about the annotated text may describe one or more of the following: the dialogue scenario, dialogue partners, text content, title of the speech sample, etc. As an example, historical annotated text may be used to indicate the historical background of the annotated sample. As an example, the text generation model may be implemented as any suitable model such as a language model, and this disclosure is not intended to limit the specific implementation of the text generation model.

[0038] In some embodiments, reference Figure 3 The speech coding model 310 (e.g., also called a coding unit) can generate speech features 325 (e.g., also called speech feature sequences or speech coding representations) corresponding to the target speech 130 based on the target speech 130. As an example, the speech coding model 310 can be implemented as a suitable coding model such as a neural network.

[0039] In some embodiments, continue to refer to Figure 3The speech coding model 310 can generate a first speech feature corresponding to the target speech 130 based on the target speech 130. Further, the converter 315 (e.g., also called a conversion unit) can convert the first speech feature generated by the speech coding model 310 into speech features 325 suitable for processing by the language model 320. As an example, the converter 315 can be implemented based on a modality converter.

[0040] In box 220, electronic device 110 can determine a first output probability corresponding to the labeled text by processing a first feature sequence using speech recognition model 136. The first feature sequence can be constructed based on speech features 325 and contextual information 335 of the speech sample.

[0041] In some embodiments, continue to refer to Figure 3 The speech recognition model 136 can process a first feature sequence to determine a first output probability 340 corresponding to the labeled text. In some embodiments, the speech recognition model 136 may include a language model 320, which can be configured to process the first feature sequence to generate the first output probability 340.

[0042] In some embodiments, prompts 330 (e.g., also known as guiding feature sequences) are used to prompt language model 320 to perform speech recognition tasks.

[0043] In some embodiments, the first output probability 340 may indicate a first probability of the target token corresponding to the labeled text. It is understood that the target token may include at least one word or character. The first output probability may indicate a first probability corresponding to at least one word or character in the target token. The first output probability can be expressed as p(y n |x,c,y <n Let y be the sequence corresponding to speech feature 325, c be the sequence corresponding to context information 335, and n be the nth step in each decoding step. In some examples, during the training of speech recognition model 136, electronic device 110 inputs the sequence corresponding to cue item 330, the sequence corresponding to context information 335, and the sequence corresponding to speech feature 325 into language model 320 respectively, and generates the final speech recognition output sequence y based on these conditions. 1,2，…N .

[0044] In box 230, electronic device 110 can determine a second output probability corresponding to the labeled text by processing the second feature sequence using speech recognition model 136. The second feature sequence can be constructed based on speech features 325 of speech samples, and the second feature sequence is independent of context information.

[0045] In some embodiments, continue to refer to Figure 3The speech recognition model 136 can also process a second feature sequence to determine a second output probability 345 corresponding to the labeled text. In some embodiments, the language model 320 can process the second feature sequence to generate the second output probability 345. Figure 3 As shown, unlike the first feature sequence, the second feature sequence does not include the sequence portion corresponding to the context information 335.

[0046] In some embodiments, the second output probability 345 may indicate a second probability of the target token corresponding to the labeled text. The second output probability may indicate a second probability corresponding to at least one word or character in the target token. The second output probability can be expressed as p(y n |x,y <n Let y be the sequence corresponding to speech feature 325, where x indicates the sequence corresponding to speech feature 325, and n indicates the nth step in each decoding step. In some examples, during the training of speech recognition model 136, electronic device 110 inputs the sequence corresponding to cue item 330 and the sequence corresponding to speech feature 325 into language model 320 respectively, and generates the final speech recognition output sequence y based on these conditions. 1,2，…N .

[0047] In box 240, electronic device 110 can determine training loss based at least on the difference between the first output probability and the second output probability in order to adjust the parameters of the speech recognition model 136.

[0048] In some embodiments, continue to refer to Figure 3 The training loss can be constructed based on the difference between the first output probability and the second output probability. The training loss can include a first part corresponding to the first output probability, a second part corresponding to the second output probability, and a third part corresponding to the difference.

[0049] As an example, the first part corresponding to the first output probability can be represented by λ*logp2(y). n |x,c,y <n The second part, corresponding to the second output probability, can be represented as (λ-1)*logp1(y). n |x,y <n ) represents the formula, where λ can be the weighting coefficient associated with the first and second output probabilities as needed, log can be the natural logarithm with base e (Euler's number, approximately 2.71828), and p1 can be p1(y n |x,y <n p2 is an abbreviation for p2(y). n |x,c,y <n (abbreviation of ).

[0050] In some embodiments, the difference may include a JS (Jensen-Shannon) divergence determined based on a first output probability and a second output probability. As an example, the third part corresponding to the difference can be represented by α*JSD(p1||p2). JSD(p1||p2) can be expressed as... M can be represented as D KL (p1||M) can be represented as D KL (p2||M) can be represented as α can be represented as a weighting coefficient associated with the difference, set as needed.

[0051] In some embodiments, the training loss based on the first output probability, the second output probability, and the JS divergence can be expressed as:

[0052] (λ-1)*logp1(y n |x,y <n )-λ*logp2(y n |x,c,y <n )+α*JSD(p1||p2) (1)

[0053] In some embodiments, the difference may include a KL (Kullback-Leibler) divergence determined based on a first output probability and a second output probability. As an example, the third part corresponding to the difference can be... D indicates KL (p1||p2) can be represented as D KL (p2||p1) can be represented as α can be represented as a weighting coefficient associated with the difference, set as needed.

[0054] In some embodiments, the training loss determined based on the first output probability, the second output probability, and the KL divergence can be expressed as:

[0055]

[0056] In some embodiments, continue to refer to Figure 3 During the inference phase, the trained speech recognition model 136 can acquire the target speech 130 to be processed (e.g., the target speech content) and the target context information associated with the target speech content.

[0057] In some embodiments, the target context information can be used to indicate at least one of the following: text content, scene information, and object information. In some embodiments, the text content is generated based on historical speech content associated with the target speech content. That is, text content can be generated as context information based on historical speech content associated with the target speech content 130.

[0058] In some embodiments, scenario information describes the dialogue scenario associated with the target speech content. For example, the conversation scenario associated with the current target speech can be used as context information. In some embodiments, object information describes at least one object associated with the target speech content. For example, user names, digital assistant names, etc., involved in interactions related to the target speech content can serve as context information. As another example, the topics involved, documents, etc., in a meeting scenario associated with the target speech content can also serve as context information.

[0059] It should be understood that the text content, scene information, object information and other data (including but not limited to the data itself, the acquisition or use of the data) mentioned in this disclosure shall comply with the requirements of relevant laws, regulations and related provisions.

[0060] Furthermore, the trained speech recognition model 136 can utilize the speech coding model 310 (e.g., also called a coding unit) and the converter 320 (e.g., also called a conversion unit) to generate speech features (e.g., also called a speech feature sequence) corresponding to the target speech content.

[0061] Furthermore, the trained speech recognition model 136 can construct an input feature sequence based on the speech feature sequence, the context feature sequence, and the guidance feature sequence. Specifically, the context feature sequence can correspond to target context information, and the guidance feature sequence can correspond to a preset guidance item (e.g., an instruction to perform speech recognition). Furthermore, the trained speech recognition model 136 can utilize the language model 320 to process the input feature sequence to generate a speech recognition result for the target speech content.

[0062] Based on the context-based speech processing scheme described above, embodiments of this disclosure can suppress the model illusion problem and improve the accuracy of speech recognition models by fusing output results that are related to context information and output results that are not related to context information.

[0063] Example devices and equipment

[0064] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4A schematic structural block diagram of an example device 400 for context-based speech processing according to certain embodiments of the present disclosure is shown. Device 400 may be implemented as or included in electronic device 110. Various modules / components in device 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0065] like Figure 4 As shown, the device 400 includes an acquisition module 410 configured to acquire training data, the training data including speech samples, context information associated with the speech samples, and labeled text corresponding to the speech samples; a first determination module 420 configured to determine a first output probability corresponding to the labeled text by processing a first feature sequence using a speech recognition model, the first feature sequence being constructed based on the speech samples and context information; a second determination module 430 configured to determine a second output probability corresponding to the labeled text by processing a second feature sequence using a speech recognition model, wherein the second feature sequence is constructed based on the speech samples and is independent of the context information; and an adjustment module 440 configured to determine a training loss based at least on the difference between the first output probability and the second output probability, in order to adjust the parameters of the speech recognition model.

[0066] In some embodiments, the apparatus 400 further includes a construction module, which is further configured to: provide labeled text to a text generation model to generate descriptive text about the labeled text; and construct contextual information corresponding to the speech samples based on the descriptive text.

[0067] In some embodiments, the first determining module 420 is further configured such that the speech recognition model includes a language model, and a first output probability or a second output probability indicates the probability of the target token corresponding to the labeled text, as determined by the language model.

[0068] In some embodiments, the adjustment module 440 is further configured to: determine the training loss based on the difference between the first output probability and the second output probability, including: constructing a training loss based on the difference between the first output probability and the second output probability, wherein the training loss includes: a first part corresponding to the first output probability, a second part corresponding to the second output probability, and a third part corresponding to the difference.

[0069] In some embodiments, the adjustment module 440 is further configured to: determine the JS divergence based on the first output probability and the second output probability; or determine the KL divergence based on the first output probability and the second output probability.

[0070] In some embodiments, the acquisition module 410 is further configured to: context information indicating at least one of the following: text content generated based on historical speech content associated with the speech sample; scene information describing the dialogue scene associated with the speech sample; and object information describing at least one object associated with the speech sample.

[0071] In some embodiments, the first determining module 420 is further configured to: the speech recognition model includes an encoding unit, a conversion unit, and a language model, and the method further includes: acquiring target speech content to be processed and target context information associated with the target speech content; using the encoding unit and the conversion unit to generate a speech feature sequence corresponding to the target speech content; constructing an input feature sequence based on the speech feature sequence, the context feature sequence, and the guiding feature sequence, wherein the context feature sequence corresponds to the target context information and the guiding feature sequence corresponds to a preset guiding item; and processing the input feature sequence using the language model to generate a speech recognition result for the target speech content.

[0072] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to achieve Figure 1 Electronic devices 110.

[0073] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0074] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0075] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0076] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0077] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0078] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0079] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0080] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0081] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0082] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0083] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A context-based speech processing method, comprising: Acquire training data, which includes speech samples, context information associated with the speech samples, and labeled text corresponding to the speech samples; By processing the first feature sequence using a speech recognition model, a first output probability corresponding to the labeled text is determined. The first feature sequence is constructed based on the speech sample and the context information. By processing the second feature sequence using the speech recognition model, a second output probability corresponding to the labeled text is determined, wherein the second feature sequence is constructed based on the speech sample and is independent of the context information; as well as The training loss is determined based at least on the difference between the first output probability and the second output probability in order to adjust the parameters of the speech recognition model.

2. The method according to claim 1, further comprising: The labeled text is provided to the text generation model to generate descriptive text about the labeled text; as well as Based on the descriptive text, the context information corresponding to the speech sample is constructed.

3. The method of claim 1, wherein the speech recognition model includes a language model, and the first output probability or the second output probability indicates the probability of a target token corresponding to the labeled text, determined by the language model.

4. The method of claim 1, wherein determining the training loss based at least on the difference between the first output probability and the second output probability comprises: Based on the difference between the first output probability and the second output probability, the training loss is constructed, and the training loss includes: a first part corresponding to the first output probability, a second part corresponding to the second output probability, and a third part corresponding to the difference.

5. The method of claim 1, wherein the difference comprises: The JS divergence is determined based on the first output probability and the second output probability; or The KL divergence is determined based on the first output probability and the second output probability.

6. The method of claim 1, wherein the context information indicates at least one of the following: Text content, which is generated based on historical speech content associated with the speech sample; Scene information, which describes the dialogue scene associated with the voice sample; Object information, which describes at least one object associated with the speech sample.

7. The method according to claim 1, wherein the speech recognition model includes an encoding unit, a conversion unit, and a language model, and the method further includes: Obtain the target speech content to be processed and the target context information associated with the target speech content; Using the encoding unit and the conversion unit, a speech feature sequence corresponding to the target speech content is generated; Based on the speech feature sequence, context feature sequence, and guidance feature sequence, an input feature sequence is constructed, wherein the context feature sequence corresponds to the target context information, and the guidance feature sequence corresponds to a preset guidance item; The input feature sequence is processed using the language model to generate a speech recognition result for the target speech content.

8. An apparatus for context-based speech processing, comprising: The acquisition module is configured to acquire training data, which includes speech samples, context information associated with the speech samples, and labeled text corresponding to the speech samples. The first determining module is configured to determine a first output probability corresponding to the labeled text by processing a first feature sequence using a speech recognition model, wherein the first feature sequence is constructed based on the speech sample and the context information; The second determining module is configured to determine a second output probability corresponding to the labeled text by processing a second feature sequence using the speech recognition model, wherein the second feature sequence is constructed based on the speech sample and is independent of the context information; as well as The adjustment module is configured to determine the training loss based at least on the difference between the first output probability and the second output probability in order to adjust the parameters of the speech recognition model.

9. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 7.

Images