Voice recognition method and device based on scene prompt and related medium

Abstract

The invention discloses a voice recognition method and device based on scene prompt and a related medium. The method comprises the steps of converting a target audio into a spectrogram to output audio time sequence features; obtaining a corresponding scene prompt text, performing word segmentation processing, and performing embedding processing through a prompt encoder to obtain prompt features; carrying out attention calculation on the audio time sequence features and the prompt features, and outputting enhanced audio features; carrying out feature splicing and fusion processing on the audio time sequence features and the enhanced audio features, inputting the audio time sequence features and the enhanced audio features into a decoder, and outputting an initial logic value corresponding to an original vocabulary in each step in an autoregression mode; and performing normalization processing on the initial logic value to obtain probability distribution, generating a transcriptional text sequence by adopting beam search or sampling, and taking the transcriptional text sequence as a speech recognition result. According to the method, the audio time sequence features and the prompt features are processed, and the transcriptional text sequence is generated by adopting beam search or sampling based on the probability distribution, so that the transcriptional accuracy is improved.

Description

A speech recognition method, device, and related medium based on scene prompts Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a speech recognition method, device and related medium based on scene prompts. Background Technology

[0002] Automatic speech recognition technology is used to transcribe speech content in target audio into text, and it has been widely deployed in applications such as voice search, message transcription, and real-time translation. Existing speech recognition solutions are mostly designed for general scenarios, primarily relying on the temporal characteristics of the audio signal itself for modeling and decoding. In specialized scenarios such as medicine and law, when the audio contains specialized terms or proper nouns that are insufficiently covered in the training, there is often a lack of processing mechanisms to incorporate the corresponding contextual prompts into the recognition calculation and to bias the decoding process. This leads to misidentification, substitution, or omission of specialized terms, making it difficult to meet the transcription accuracy requirements of specialized scenarios. Summary of the Invention

[0003] This invention provides a speech recognition method, device, and related medium based on scene prompts, aiming to solve the technical problem in the prior art where the lack of corresponding scene prompt text to guide the recognition and decoding in specific scenarios leads to low transcription accuracy.

[0004] In a first aspect, embodiments of the present invention provide a speech recognition method based on scene prompts, comprising: acquiring target audio and converting the target audio into a spectrogram to output corresponding audio temporal features; acquiring scene prompt text corresponding to the target audio, performing word segmentation on the scene prompt text, and then performing embedding processing through a prompt encoder to obtain prompt features; inputting the audio temporal features and the prompt features into a preset audio text cross-attention module for attention calculation, and outputting enhanced audio features; performing feature concatenation and fusion processing on the audio temporal features and the enhanced audio features to obtain fused features; inputting the fused features into a decoder to output initial logical values ​​corresponding to the original vocabulary at each step in an autoregressive manner; normalizing the initial logical values ​​to obtain a probability distribution, and generating a transcribed text sequence based on the probability distribution using beam search or sampling, and using the transcribed text sequence as the speech recognition result of the target audio.

[0005] Secondly, embodiments of the present invention provide a speech recognition device based on scene prompts, comprising: an audio acquisition unit for acquiring target audio and converting the target audio into a spectrogram to output corresponding audio temporal features; an audio processing unit for acquiring scene prompt text corresponding to the target audio, performing word segmentation on the scene prompt text, and then embedding it through a prompt encoder to obtain prompt features; a feature enhancement unit for inputting the audio temporal features and the prompt features into a preset audio text cross-attention module for attention calculation, and outputting enhanced audio features; a feature fusion unit for performing feature concatenation and fusion processing on the audio temporal features and the enhanced audio features to obtain fused features; a logic processing unit for inputting the fused features into a decoder and outputting initial logic values ​​corresponding to the original vocabulary at each step in an autoregressive manner; and a recognition output unit for normalizing the initial logic values ​​to obtain a probability distribution, and generating a transcribed text sequence based on the probability distribution using beam search or sampling, and using the transcribed text sequence as the speech recognition result of the target audio.

[0006] Thirdly, embodiments of the present invention provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the scene-based speech recognition method of the first aspect.

[0007] Fourthly, embodiments of the present invention provide a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the scene-based speech recognition method of the first aspect.

[0008] This invention provides a speech recognition method based on scene prompts, including: acquiring target audio and converting it into a spectrogram to output corresponding audio temporal features; acquiring scene prompt text corresponding to the target audio, performing word segmentation on the scene prompt text, and then embedding it through a prompt encoder to obtain prompt features; inputting the audio temporal features and the prompt features into a preset audio text cross-attention module for attention calculation to output enhanced audio features; performing feature concatenation and fusion processing on the audio temporal features and the enhanced audio features to obtain fused features; inputting the fused features into a decoder to output initial logical values ​​corresponding to the original vocabulary at each step in an autoregressive manner; normalizing the initial logical values ​​to obtain a probability distribution, and generating a transcribed text sequence based on the probability distribution using beam search or sampling; and using the transcribed text sequence as the speech recognition result of the target audio. This invention improves transcription accuracy by processing audio temporal features and prompt features, and then generating a transcribed text sequence based on the probability distribution using beam search or sampling.

[0009] This invention also provides a scene-based voice recognition device, computer equipment, and storage medium, which have the same beneficial effects as described above. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 is a flowchart illustrating a scene-based speech recognition method according to an embodiment of the present invention; Figure 2 is a schematic block diagram illustrating a scene-based speech recognition device according to an embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0014] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0015] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] Please refer to Figure 1 below. Figure 1 is a flowchart of a speech recognition method based on scene prompts provided in an embodiment of the present invention, specifically including steps S101 to S106.

[0017] S101. Acquire the target audio and convert it into a spectrogram to output the corresponding audio temporal features; S102. Acquire the scene prompt text corresponding to the target audio, perform word segmentation on the scene prompt text, and then perform embedding processing through a prompt encoder to obtain prompt features; S103. Input the audio temporal features and the prompt features into a preset audio text cross-attention module for attention calculation and output enhanced audio features; S104. Perform feature concatenation and fusion processing on the audio temporal features and the enhanced audio features to obtain fused features; S105. Input the fused features into a decoder and output the initial logical value corresponding to the original vocabulary at each step in an autoregressive manner; S106. Normalize the initial logical value to obtain a probability distribution, and generate a transcribed text sequence based on the probability distribution using beam search or sampling, and use the transcribed text sequence as the speech recognition result of the target audio.

[0018] In step S101, after acquiring the target audio, the target audio is converted into a spectrogram, and the corresponding audio temporal features are output. That is, the target audio can first be converted into a spectrogram, and then subjected to one-dimensional convolution and activation processing to reduce the temporal resolution. After being superimposed with relative position encoding, it is input into a Transformer encoder to obtain audio temporal features containing temporal context information.

[0019] In one embodiment, step S101 includes: acquiring target audio and resampling the target audio to obtain resampled audio data; performing Fourier transform on the resampled audio data and projecting and logarithmic transform on a Mel filter bank to obtain a spectrogram; performing two-layer one-dimensional convolution and Gaussian error linear unit activation on the spectrogram to obtain an intermediate feature sequence; adding a first relative position code to the intermediate feature sequence; and then inputting the added relative position code into a first encoder for encoding to output audio temporal features.

[0020] In this embodiment, a target audio segment to be translated is first acquired and used as the input signal for subsequent speech recognition calculations. The target audio is then resampled to ensure its sampling rate matches a preset feature extraction configuration, resulting in resampled audio data. Resampling may involve interpolation or extraction of the original audio's sampling point sequence, as well as uniform processing of basic parameters such as the number of channels and amplitude scale, ensuring the resampled audio data meets the input requirements for subsequent spectral analysis. After obtaining the resampled audio data, a Fourier transform is performed to obtain a time-frequency domain representation. In a specific implementation, the resampled audio data can be divided into frames according to a preset window function and frame shift, and a Fourier transform is performed on each frame to obtain an amplitude spectrum or power spectrum. The spectral data is then projected onto a Mel filter bank to map the linear frequency scale to the Mel scale, and a logarithmic transform is performed on the projection result to obtain a spectrogram. The spectrogram can be an 80-dimensional Log-Mel spectrogram, used as input to characterize the energy distribution of the target audio in the time and frequency dimensions, providing a basic input for the subsequent encoder to extract advanced acoustic representations.

[0021] After obtaining the spectrogram, it is input into the front-end feature compression module for temporal downsampling. This module performs two layers of one-dimensional convolution on the spectrogram sequentially, followed by GELU activation processing to obtain an intermediate feature sequence. The two-layer one-dimensional convolution can use a sliding window approach to compress and fuse consecutive audio frames, reducing the time step length while preserving key acoustic patterns, thus reducing the sequence size for subsequent Transformer encoder self-attention calculations. After obtaining the intermediate feature sequence, a first relative position encoding is added to it to introduce temporal order and inter-frame relative distance information, enabling subsequent encoding calculations to utilize the temporal prior of the target audio. The intermediate feature sequence with the added first relative position encoding is then input into the first encoder, which can be a Transformer encoder. This encoder uses a self-attention mechanism to mine global dependencies and outputs audio temporal features A. e Audio timing features A eThis can be represented as a tensor of shape [B, T, D], where B is the batch size, T is the time step, and D is the feature dimension. Furthermore, in one implementation, the two one-dimensional convolutional layers, the GELU activation layer, the first relative position encoding, and the Transformer encoder can be initialized using Whisper-Tiny's initial weights to reuse Whisper-Tiny's pre-training capabilities in acoustic modeling while maintaining end-to-end trainability.

[0022] In step S102, the scene prompt text corresponding to the target audio is obtained, and after word segmentation, it is embedded by a prompt encoder to obtain prompt features. That is, the prompt encoder can embed the segmented prompt words and output prompt features carrying scene semantic information after encoding.

[0023] In one embodiment, step S102 includes: taking the scene prompt text as prompt input, performing word segmentation on the scene prompt text, and converting the word segmentation result into a token identifier sequence; performing token embedding processing on the token identifier sequence through the prompt encoder to obtain an embedded sequence; adding a second relative position code and an embedding domain label to the embedded sequence to obtain a prompt encoding input sequence; and inputting the prompt encoding input sequence into a preset second encoder to output prompt features.

[0024] In this embodiment, the scene prompt text is a textual description of the scene in which the target audio is located. Several prompt words or phrases can be used to represent the scene semantics, such as "treatment" or "CT scan," to provide the model with prior prompt information related to the target audio context. In the specific implementation, the scene prompt text is used as the prompt input. Word segmentation is performed on the scene prompt text, mapping the segmented words or characters to Token IDs. The mapping results are then organized into a Token ID sequence, which serves as the input to the subsequent prompt encoder. This Token ID sequence preserves the computable representation of the prompt text content while retaining the order and length information of the prompt words, where the length of each prompt word can be denoted as P. The prompt encoder then performs Token embedding processing on the Token ID sequence to obtain an embedding sequence. The prompt encoder can include a word embedding encoding module (Embedding) to map discrete Token IDs to a continuous vector space. Based on this, positional encoding is added to the embedding sequence to introduce positional identifiers for each Token and preserve the order relationship between words. A sinusoidal positional encoding module (PE(pos)) can be used, where pos represents the absolute position index of each token in the prompt word sequence, thus obtaining an embedding representation containing positional information.

[0025] Furthermore, while adding a second relative position encoding to the embedded sequence, a domain label can be embedded to introduce the domain information of the scene. The domain label can be used to distinguish different application domains such as medical and legal, so that the prompt encoding input sequence carries the semantic information, positional information, and domain information of the prompt text simultaneously. Through the above processing, the prompt encoding input sequence is obtained and input into a preset second encoder. The second encoder can be a small Transformer encoder, which extracts the contextual association of the prompt text through self-attention calculation and outputs prompt features containing scene semantics. Its shape can be [B, P, D], where B is the batch size, P is the prompt word length, and D is the feature dimension. The second encoder can be initialized with Whisper-Tiny parameters so as to reuse the representation ability of the pre-trained model in the prompt-side encoding stage. The prompt features output by the second encoder are used as one of the inputs of the subsequent audio-text cross-attention module to complete the attention calculation in conjunction with the audio temporal features. Correspondingly, the prompt features can be calculated according to the following formula: in, For the prompt word token sequence, This refers to the cue side representation after adding location encoding, i.e., the cue feature.

[0026] In step S103, the audio temporal features and cue features are input into a preset audio-text cross-attention module for attention calculation to output enhanced audio features. Through this attention calculation, the audio features focus on cue information related to the current audio content during the calculation process, thereby obtaining an enhanced audio representation of the scene cue.

[0027] In one embodiment, step S103 includes: constructing an attention matrix by using the audio temporal features as a query vector set and the prompt features as a key vector set and a value vector set; performing attention calculation on the constructed attention matrix through the audio-text cross-attention module to output enhanced audio features.

[0028] In this embodiment, the output audio timing features are first denoted as audio acoustic features A. e The output prompt features are denoted as P. e and A e As the query vector set Q, P e Simultaneously serving as the set of key vectors K and the set of value vectors V, the two types of features are aligned on a unified feature dimension D, thus preparing the input tensor for subsequent attention calculations.

[0029] After constructing Q, K, and V, when building the attention matrix, the relevance score between the query vector set Q and the key vector set K can be calculated first, and this relevance score can be used as the basic term of the attention matrix. Correspondingly, the enhanced audio features can be calculated using the following expression. : in, Let K be the dimension of the key vector, used to analyze ( The scaling process is applied to ensure that the relevance scores under different dimensions are within a comparable range. Softmax is a normalization function used to normalize the scaled relevance scores, resulting in attention weights. These attention weights can be organized into a weight matrix, where each element represents the proportion of each V-value vector under the current query conditions when aggregating the prompt features. The higher the weight, the greater the proportion of the corresponding prompt information during aggregation.

[0030] After obtaining the weight matrix, it is multiplied by the value vector set V to achieve weighted aggregation of the cue features and output enhanced audio features. Through the calculations of the aforementioned audio-text cross-attention module, enhanced audio features are achieved. While maintaining the audio context, it introduces cue information related to the current audio content, giving higher weight to relevant cue words in the attention allocation of audio-side features. During the weight calculation process, the weight of cue words is allowed to be adaptively adjusted with the audio content, thereby completing the association modeling between cue words and audio and outputting an enhanced representation that can be called for subsequent fusion and decoding.

[0031] In step S104, the audio temporal features and the enhanced audio features are concatenated and fused to obtain the fused features. The original audio features and the enhanced features can be concatenated, and a gating network can be used to calculate a gating value to adaptively adjust the fusion ratio of the two types of features, thereby outputting the fused features.

[0032] In one embodiment, step S104 includes: concatenating the audio temporal features and the enhanced audio features in the last dimension to obtain concatenated features; multiplying the concatenated features by a first gating parameter and adding the multiplication result to a second gating parameter to obtain a gating value; performing element-wise weighting on the enhanced audio features based on the gating value to obtain a first weighted feature; performing complementary calculation on the gating value to obtain a complementary gating value, and performing element-wise weighting on the audio temporal features based on the complementary gating value to obtain a second weighted feature; and adding the first weighted feature and the second weighted feature to obtain a fused feature.

[0033] In this embodiment, audio timing features are... With enhanced audio features Feature concatenation is performed on the last dimension to obtain the concatenated features. This concatenation can be denoted as [ ; The semicolon indicates concat in the last dimension, ensuring that the concatenated features simultaneously contain the desired features at each time step. and The characteristic components provide joint inputs for the gating network.

[0034] Furthermore, the gating value is calculated based on the concatenated features. The gating network is a learnable structure, and a common implementation is a linearlayer + Sigmoid: first, the concatenated features are multiplied by the first gating parameter, and the result is added to the second gating parameter to obtain the gated pre-activation result; then, the Sigmoid activation function is applied to the gating pre-activation result. This yields a gate value G between 0 and 1. When expressed in parametric notation, the first gate parameter can be represented as W. g The second gating parameter is mapped to b. g Then the gate value satisfies: in, It is Sigmoid. and These are learnable parameters.

[0035] After obtaining the gate value G, element-wise weighting is performed on the enhanced audio features based on the gate value. Specifically, element-wise multiplication is represented by the symbol ⊙, and the enhanced audio features are weighted accordingly. By weighting each element according to the gate value, the first weighted feature is obtained, which is G⊙. Here, G is numerically related to... Element-wise correspondence is used to ensure that each time step and each feature dimension receives a corresponding weighting coefficient. Then, complementary gating values ​​are calculated on the gating value G to obtain complementary gating values. The complementary gating value can be written as (1-G), where 1 is a tensor of all 1s with the same shape as G. Subsequently, audio temporal features are analyzed based on the complementary gating values. Performing element-wise weighting yields the second weighted feature, which is (1-G)⊙. The above complementary calculations and element-wise weighting make... and Complementary coefficients are used to calculate the proportions at the same location. Finally, the first weighted feature and the second weighted feature are added together to obtain the fused feature. The fused feature is represented symbolically. satisfy .

[0036] In step S105, the fused features are input into the decoder, which then outputs the initial logical values ​​corresponding to the original vocabulary at each step in an autoregressive manner. As one implementation, after the fused features are input into the decoder, the decoder outputs the initial logical values ​​for each step in an autoregressive decoding process.

[0037] In one embodiment, step S105 includes: using the fused feature as a decoding input feature, inputting the decoding input feature into the decoder to initialize the decoding state of the decoding input feature; performing autoregressive decoding calculation of the current step based on the decoding state to obtain the decoding output of the current step; and performing vocabulary mapping processing on the decoding output of the current step to output the initial logical value of the current step on the original vocabulary.

[0038] In this embodiment, the fused feature is used as the decoding input feature, and is denoted as the fused audio feature. and will Input a standard Whisper decoder, causing the decoder to read The acoustic information carried and the injected scene prompts are used to initialize the decoding state of the decoding input feature. This decoding state describes the historical context of the current decoding process, the state of the generated tokens, and other related information. Intermediate results of related attention calculations.

[0039] Internally, the Whisper decoder may contain a transformer decoder module. The transformer decoder mainly consists of a multi-head self-attention mechanism and stacked Transformer layers. After initializing the decoding state, the decoder performs multi-head self-attention computation for each position in the input sequence, dynamically calculating the attention weight distribution for each position in the sequence (e.g., the position corresponding to an audio frame). This weight distribution is used to characterize the correlation strength between that position and other positions in the sequence. Other positions may include other audio frame positions and the positions corresponding to cue words, enabling the decoding state to read and integrate contextual information related to the current decoding step during subsequent autoregressive inference.

[0040] During the autoregressive decoding computation of the current step, the decoder, based on the initialized decoding state, performs a cross-validation of the current step input token and... Attention calculation and inter-layer transformation are performed to obtain the decoding output for the current step. The autoregressive approach means the decoder generates output step-by-step over time, utilizing the previously generated historical token states at each decoding step. The provided temporal acoustic representations are used to compute the output of the current step, ensuring that the current step's decoding output remains consistent with the context of the previous decoding steps.

[0041] After obtaining the decoding output of the current step, a vocabulary mapping process is performed on the current step decoding output to output the initial logical value Logits of the current step on the original vocabulary. original The vocabulary mapping process projects the current step's decoding output onto a space with the original vocabulary as the output dimension, enabling Logits to... original On this dimension, corresponding logical values ​​are given for each candidate lexical unit, thus obtaining the initial logical value result corresponding to the current decoding step, which is used for subsequent Logits. original Provides input for probability calculations and sequence generation.

[0042] In step S106, the initial logic value is normalized to obtain a probability distribution, and a transcribed text sequence is generated based on the probability distribution using beam search or sampling. Finally, the transcribed text sequence is output as the speech recognition result of the target audio.

[0043] In one embodiment, step S106 includes: constructing a candidate lexicon set using the original vocabulary as the output dimension; normalizing the initial logical value sequence to obtain a probability distribution corresponding to the candidate lexicon set; performing beam search processing or sampling processing based on the probability distribution to generate a transcribed text sequence; performing sequence selection processing on the transcribed text sequence, and outputting the sequence-selected transcribed text sequence as the speech recognition result of the target audio.

[0044] In this embodiment, the initial logical value Logits original The sequence of logistic values ​​that the decoder outputs stepwise on the original vocabulary in an autoregressive manner, with each step being Logits. original Each step corresponds to a candidate evaluation along the original vocabulary dimension. At the start of the generation phase, a candidate lexicon set is constructed using the original vocabulary as the output dimension. The candidate lexicon set can consist of each lexicon in the original vocabulary and its Token ID, ensuring that each Logits step... original It can correspond one-to-one with the candidate lexical set; under this correspondence, Logits original Each decoding step provides relative preference information for each candidate lexical in the candidate lexical set, which serves as input for subsequent normalization. The initial logits sequence is then normalized to obtain the probability distribution corresponding to the candidate lexical set. Specifically, this can be achieved by normalizing the Logits for each decoding step. original Apply the Softmax operation to convert the logical value into a probability distribution, such that the probability distribution takes non-negative values ​​and sums to 1 along the dimension of the candidate word set; the resulting probability distribution is used to represent the probability of each candidate word being the next output token in the current decoding step.

[0045] After obtaining the stepwise probability distribution, beam search or sampling is performed based on this probability distribution to generate transcribed text sequences. Taking beam search as an example, an empty set of partial sequences can be established at the initial moment. At each decoding step, the partial sequences are expanded using the current probability distribution to obtain multiple candidate partial sequences. Subsequently, the candidate partial sequences are preserved and pruned, allowing subsequent decoding steps to iteratively expand based on the preserved candidate sequences until a set of candidate transcribed text sequences that meets the termination condition is generated. Taking sampling as an example, at each decoding step, the next token can be extracted according to the current probability distribution, and the extracted token is appended to the current sequence, gradually obtaining one or more transcribed text sequences.

[0046] After obtaining candidate transcribed text sequences, sequence selection processing is performed on the transcribed text sequences, and the sequence-selected transcribed text sequences are output as the speech recognition results of the target audio. For multiple candidate sequences output by the beam search, the candidate sequences can be sorted and filtered, and one of them can be selected as the final output; for the sampled sequences, the sampled sequences can be directly used as the output, or multiple sequences obtained from multiple samplings can be filtered and output.

[0047] It should be noted that the Logits corresponding to the probability distribution are... original Autoregressive inference from the decoding stage; in this inference process, the model uses attention weights to weightedly fuse information from different positions, enabling it to reference and integrate the contextual information of the entire sequence when processing the current segment. Through multi-layered computation, it gradually obtains an expression of the semantic relationships between input information, thus supporting the sequence generation process in the Softmax, Beam Search, or Sampling stages. In one implementation, the initial weights of the Transformer encoder can adopt the weights of the Whisper-Tiny model, making the logits from feature encoding to decoded output... original Furthermore, the entire process of Softmax and sequence generation can reuse the representational capabilities provided by the pre-trained parameters.

[0048] In summary, this application converts the target audio into a spectrogram and extracts its temporal features. Simultaneously, it encodes the corresponding scene cue text to obtain cue features. Then, it utilizes an audio-text cross-attention module to enhance the acoustic representation through the cue information. Furthermore, it employs gated fusion to adaptively balance the original audio features with the enhanced features, enabling the decoder to output Logits in an autoregressive manner. originalAfter obtaining the probability distribution through Softmax, it can more stably select candidate lexical units and transcription paths that match the scene context during Beam Search or Sampling generation, thereby improving the recognition accuracy and contextual consistency of professional terms and proper nouns in specialized scenarios. At the same time, under the control of the gating mechanism, it suppresses the risk of misleading information introduced by prompts, takes into account both general speech content and scene-related expressions, and improves the transcription accuracy.

[0049] Referring to Figure 2, which is a schematic block diagram of a scene-based speech recognition device 200 provided in an embodiment of the present invention, the scene-based speech recognition device 200 includes: an audio acquisition unit 201, used to acquire target audio and convert the target audio into a spectrogram to output corresponding audio temporal features; an audio processing unit 202, used to acquire scene-based prompt text corresponding to the target audio, perform word segmentation on the scene-based prompt text, and then perform embedding processing through a prompt encoder to obtain prompt features; and a feature enhancement unit 203, used to input the audio temporal features and the prompt features into a preset audio input. The text cross-attention module performs attention calculations and outputs enhanced audio features; the feature fusion unit 204 is used to perform feature concatenation and fusion processing on the audio temporal features and the enhanced audio features to obtain fused features; the logic processing unit 205 is used to input the fused features into the decoder and output the initial logic value corresponding to the original vocabulary at each step in an autoregressive manner; the recognition output unit 206 is used to normalize the initial logic value to obtain a probability distribution, and generate a transcribed text sequence based on the probability distribution using beam search or sampling, and use the transcribed text sequence as the speech recognition result of the target audio.

[0050] In this embodiment, the audio acquisition unit 201 acquires the target audio and converts it into a spectrogram to output the corresponding audio temporal features; the audio processing unit 202 acquires the scene prompt text corresponding to the target audio, performs word segmentation on the scene prompt text, and then performs embedding processing through a prompt encoder to obtain prompt features; the feature enhancement unit 203 inputs the audio temporal features and the prompt features into a preset audio text cross-attention module for attention calculation and outputs enhanced audio features; the feature fusion unit 204 performs feature splicing and fusion processing on the audio temporal features and the enhanced audio features to obtain fused features; the logic processing unit 205 inputs the fused features into a decoder and outputs the initial logic value corresponding to the original vocabulary at each step in an autoregressive manner; the recognition output unit 206 normalizes the initial logic value to obtain a probability distribution, and generates a transcribed text sequence based on the probability distribution using beam search or sampling, and uses the transcribed text sequence as the speech recognition result of the target audio.

[0051] In one embodiment, the audio acquisition unit 201 is specifically used for: acquiring target audio and resampling the target audio to obtain resampled audio data; performing Fourier transform processing on the resampled audio data and projecting and logarithmic transforming it onto a Mel filter bank to obtain a spectrogram; performing two-layer one-dimensional convolution processing and Gaussian error linear unit activation processing on the spectrogram to obtain an intermediate feature sequence; adding a first relative position code to the intermediate feature sequence; and then inputting the added relative position code into a first encoder for encoding to output audio temporal features.

[0052] In one embodiment, the audio processing unit 202 is specifically configured to: take the scene prompt text as prompt input, perform word segmentation on the scene prompt text, and convert the word segmentation result into a token identifier sequence; perform token embedding processing on the token identifier sequence through the prompt encoder to obtain an embedded sequence; add a second relative position code and an embedding domain label to the embedded sequence to obtain a prompt encoding input sequence; and input the prompt encoding input sequence into a preset second encoder to output prompt features.

[0053] In one embodiment, the feature enhancement unit 203 is specifically used to: construct an attention matrix by using the audio temporal features as a query vector set and the prompt features as a key vector set and a value vector set; and perform attention calculation on the constructed attention matrix through the audio-text cross-attention module to output enhanced audio features.

[0054] In one embodiment, the feature fusion unit 204 is specifically configured to: concatenate the audio temporal features and the enhanced audio features in the last dimension to obtain concatenated features; multiply the concatenated features by a first gating parameter and add the multiplication result to a second gating parameter to obtain a gating value; perform element-wise weighting on the enhanced audio features based on the gating value to obtain a first weighted feature; perform complementary calculation on the gating value to obtain a complementary gating value, and perform element-wise weighting on the audio temporal features based on the complementary gating value to obtain a second weighted feature; and add the first weighted feature and the second weighted feature to obtain a fused feature.

[0055] In one embodiment, the logic processing unit 205 is specifically configured to: use the fused feature as a decoding input feature, input the decoding input feature into the decoder to initialize the decoding state of the decoding input feature; perform autoregressive decoding calculation of the current step based on the decoding state to obtain the decoding output of the current step; and perform vocabulary mapping processing on the decoding output of the current step to output the initial logic value of the current step on the original vocabulary.

[0056] In one embodiment, the recognition output unit 206 is specifically configured to: construct a candidate word set by using the original vocabulary as the output dimension; normalize the initial logical value sequence to obtain a probability distribution corresponding to the candidate word set; perform beam search processing or sampling processing based on the probability distribution to generate a transcribed text sequence; perform sequence selection processing on the transcribed text sequence, and output the sequence-selected transcribed text sequence as the speech recognition result of the target audio.

[0057] Since the embodiments of the apparatus and the embodiments of the method correspond to each other, please refer to the description of the embodiments of the method for the embodiments of the apparatus, which will not be repeated here.

[0058] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed, can perform the steps provided in the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0059] This invention also provides a computer device, which may include a memory and a processor. The memory stores a computer program, and when the processor calls the computer program in the memory, it can implement the steps provided in the above embodiments. Of course, the computer device may also include various network interfaces, a power supply, a graphics card, etc., to utilize the graphics card's performance to operate the model, such as for inference and training.

[0060] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0061] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A speech recognition method based on scene prompts, characterized in that, include: Acquire the target audio and convert it into a spectrogram to output the corresponding audio temporal features; The process involves obtaining scene prompt text corresponding to the target audio, performing word segmentation on the scene prompt text, embedding it through a prompt encoder to obtain prompt features, inputting the audio temporal features and the prompt features into a preset audio text cross-attention module for attention calculation, and outputting enhanced audio features, performing feature concatenation and fusion processing on the audio temporal features and the enhanced audio features to obtain fused features, and inputting the fused features into a decoder to output the initial logical values ​​corresponding to the original vocabulary at each step in an autoregressive manner. The initial logic value is normalized to obtain a probability distribution, and a transcribed text sequence is generated based on the probability distribution using beam search or sampling. The transcribed text sequence is then used as the speech recognition result of the target audio.

2. The speech recognition method based on scene prompts according to claim 1, characterized in that, The step of acquiring the target audio and converting it into a spectrogram to output the corresponding audio temporal features includes: acquiring the target audio and resampling it to obtain resampled audio data; performing Fourier transform on the resampled audio data and projecting and logarithmic transforming it onto a Mel filter bank to obtain a spectrogram; performing two-layer one-dimensional convolution and Gaussian error linear unit activation on the spectrogram to obtain an intermediate feature sequence; adding a first relative position code to the intermediate feature sequence; and then inputting the added relative position code into a first encoder for encoding to output the audio temporal features.

3. The speech recognition method based on scene prompts according to claim 1, characterized in that, The process of obtaining the scene prompt text corresponding to the target audio, performing word segmentation on the scene prompt text, and then embedding it through a prompt encoder to obtain prompt features includes: taking the scene prompt text as prompt input, performing word segmentation on the scene prompt text, and converting the word segmentation result into a token identifier sequence; performing token embedding on the token identifier sequence through the prompt encoder to obtain an embedded sequence; adding a second relative position code and an embedding domain label to the embedded sequence to obtain a prompt encoding input sequence; and inputting the prompt encoding input sequence into a preset second encoder to output prompt features.

4. The speech recognition method based on scene prompts according to claim 1, characterized in that, The step of inputting the audio temporal features and the prompt features into a preset audio-text cross-attention module for attention calculation and outputting enhanced audio features includes: constructing an attention matrix by using the audio temporal features as a query vector set and the prompt features as a key vector set and a value vector set; and performing attention calculation on the constructed attention matrix through the audio-text cross-attention module to output enhanced audio features.

5. The speech recognition method based on scene prompts according to claim 1, characterized in that, The step of performing feature concatenation and fusion processing on the audio temporal features and the enhanced audio features to obtain fused features includes: concatenating the audio temporal features and the enhanced audio features in the last dimension to obtain concatenated features; multiplying the concatenated features by a first gating parameter and adding the multiplication result to a second gating parameter to obtain a gating value; performing element-wise weighting on the enhanced audio features based on the gating value to obtain a first weighted feature; performing complementary calculation on the gating value to obtain a complementary gating value, and performing element-wise weighting on the audio temporal features based on the complementary gating value to obtain a second weighted feature; and adding the first weighted feature and the second weighted feature to obtain the fused feature.

6. The speech recognition method based on scene prompts according to claim 1, characterized in that, The step of inputting the fused features into the decoder and outputting the initial logical value corresponding to the original vocabulary at each step in an autoregressive manner includes: using the fused features as decoding input features; inputting the decoding input features into the decoder to initialize the decoding state of the decoding input features; performing autoregressive decoding calculation for the current step based on the decoding state to obtain the decoding output for the current step; and performing vocabulary mapping processing on the decoding output for the current step to output the initial logical value for the current step on the original vocabulary.

7. The speech recognition method based on scene prompts according to claim 1, characterized in that, The process of normalizing the initial logical values ​​to obtain a probability distribution, and generating a transcribed text sequence based on the probability distribution using beam search or sampling, and using the transcribed text sequence as the speech recognition result of the target audio, includes: constructing a candidate word set using the original vocabulary as the output dimension; normalizing the initial logical value sequence to obtain a probability distribution corresponding to the candidate word set; performing beam search or sampling processing based on the probability distribution to generate a transcribed text sequence; performing sequence selection processing on the transcribed text sequence, and outputting the sequence-selected transcribed text sequence as the speech recognition result of the target audio.

8. A speech recognition device based on scene prompts, characterized in that, include: An audio acquisition unit is used to acquire target audio and convert the target audio into a spectrogram to output the corresponding audio temporal features; An audio processing unit is used to acquire scene prompt text corresponding to the target audio, perform word segmentation on the scene prompt text, and then perform embedding processing through a prompt encoder to obtain prompt features; The feature enhancement unit is used to input the audio temporal features and the prompt features into a preset audio-text cross-attention module for attention calculation, and output the enhanced audio features. feature The fusion unit is used to perform feature concatenation and fusion processing on the audio temporal features and the enhanced audio features to obtain fused features; The logic processing unit is used to input the fused features into the decoder and output the initial logic value corresponding to the original vocabulary at each step in an autoregressive manner. The recognition output unit is used to normalize the initial logic value to obtain a probability distribution, and generate a transcribed text sequence based on the probability distribution using beam search or sampling, and use the transcribed text sequence as the speech recognition result of the target audio.

9. A computer device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the scene-based speech recognition method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the scene-based speech recognition method as described in any one of claims 1 to 7.

Images