Speech recognition method and apparatus, electronic device, and computer-readable storage medium

Abstract

The application provides a speech recognition method and device, electronic equipment and computer readable storage medium, which can be applied to the technical field of speech recognition. The method comprises the following steps: in response to receiving a voice signal from a power construction site, extracting a target voice signal from the de-noised voice signal by jointly analyzing the energy distribution characteristics and spectral entropy characteristics of the de-noised voice signal. The frequency domain characteristics of the target voice signal are divided in the frequency domain to obtain first frequency characteristics and second frequency characteristics. By dynamically adjusting the weights of the first frequency characteristics and the second frequency characteristics, the first frequency characteristics and the second frequency characteristics are fused based on the multi-head self-attention mechanism to generate initial fusion characteristics. Based on the interaction attention mechanism, the initial fusion characteristics are modified by the power industry terms matched with the initial fusion characteristics to generate target semantic characteristics. The target semantic characteristics are recognized to generate a target task type and target task content.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and more specifically, to a speech recognition method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In power industry applications such as telephone customer service, dispatch instructions, or voice self-service, the accuracy of speech recognition is low due to differences in accents, complex background noise, and frequent changes in speech rate in the voice input signal. Summary of the Invention

[0003] In view of this, this application provides a speech recognition method, apparatus, electronic device, and computer-readable storage medium.

[0004] One aspect of this application provides a speech recognition method, comprising: in response to receiving a speech signal from a terminal device deployed at a power construction site, extracting a target speech signal from the denoised speech signal by jointly analyzing the energy distribution features and spectral entropy features of the denoised speech signal; wherein the target speech signal is a semantically continuous audio sequence; dividing the frequency domain features of the target speech signal into frequency domain features to obtain a first frequency feature and a second frequency feature; wherein the first frequency feature represents the global semantics of the target speech signal; the second frequency feature represents the keyword semantics of the target speech signal; and dynamically adjusting the weights of the first frequency feature and the second frequency feature respectively. The system firstly integrates the first and second frequency features based on a multi-head self-attention mechanism to generate initial fused features. Specifically, when the target speech signal contains noise, the weight of the first frequency feature is greater than the weight of the second frequency feature; when the target speech signal contains keywords, the weight of the first frequency feature is less than the weight of the second frequency feature. Based on an interactive attention mechanism, power industry terminology matching the initial fused features is used to modify the expression of the initial fused features, generating target semantic features. These target semantic features correct the colloquial expressions in the initial semantic features. Finally, the target semantic features are identified to generate the target task type and target task content.

[0005] According to an embodiment of this application, the above-mentioned method of dynamically adjusting the weights of the first frequency feature and the second frequency feature, and fusing the first frequency feature and the second frequency feature based on a multi-head self-attention mechanism to generate an initial fused feature, includes: weighting and aggregating the first frequency feature according to its frequency dimension by calling a learnable first weight matrix to obtain a first aggregated feature; wherein the first aggregated feature focuses on the energy envelope, prosodic contour, and time-varying global semantic structure in the target speech signal; weighting and aggregating the second frequency feature according to its frequency dimension by calling a learnable second weight matrix to obtain a second aggregated feature; wherein the second aggregated feature focuses on the articulatory units and articulation abrupt boundary in the target speech signal; processing the first aggregated feature and the second aggregated feature by calling a probability distribution function to generate the weights of the first frequency feature and the second frequency feature; and fusing the first frequency feature and the second frequency feature according to their respective weights based on the multi-head attention mechanism to obtain the initial fused feature.

[0006] According to an embodiment of this application, the above-mentioned frequency domain division of the frequency domain features of the target speech signal to obtain a first frequency feature and a second frequency feature includes: performing a Fourier transform on the time domain features of the target speech signal to generate the frequency domain features of the target speech signal; processing the frequency domain features of the target speech signal by calling an activation function to generate a frequency domain mask; and using the frequency domain mask to perform frequency domain division of the frequency domain features of the target speech signal to obtain the first frequency feature and the second frequency feature.

[0007] According to an embodiment of this application, the above-described method of modifying the initial fusion features by using power industry terms that match the initial fusion features based on an interactive attention mechanism to generate target semantic features includes: fusing the initial fusion features and the power industry terms based on an interactive attention mechanism to generate semantically enhanced fusion features. The semantically enhanced fusion features strengthen the industry terminology representation in the initial fusion features. The semantically enhanced fusion features and the initial fusion features are then average-pooled to generate intermediate features. The intermediate features are processed using a multilayer perceptron to generate target weights. Based on the target weights, the semantically enhanced fusion features and the initial fusion features are weighted and fused to generate the target semantic features.

[0008] According to an embodiment of this application, the above-mentioned response to receiving a voice signal from a terminal device deployed at a power construction site, extracting a target voice signal from the denoised voice signal by jointly analyzing the energy distribution characteristics and spectral entropy characteristics of the denoised voice signal, includes: extracting a continuous voice signal from the denoised voice signal in response to receiving a voice signal from a terminal device deployed at a power construction site; performing joint analysis of the energy distribution characteristics and spectral entropy characteristics of the continuous voice signal to obtain target segmentation points; segmenting the continuous voice signal based on the target segmentation points to obtain segmented voice signals; and performing pre-emphasis filtering and symmetrical Hanning window weighting processing on the segmented voice signals to obtain the target voice signal.

[0009] According to embodiments of this application, the above-mentioned identification of the target semantic features to generate a target task type and target task content includes: performing task type identification on the target semantic features to obtain a target task type; performing task content identification on the target semantic features to obtain the target task content; in response to determining that the target task type and the target task content do not match, performing speech recognition on the denoised speech signal to generate target text; performing intent recognition on the target text to generate a target intent; the target intent includes an expected task type and an expected task content; correcting the target task type using the expected task type to generate a first corrected task type; and correcting the target task content using the expected task content to generate a first corrected task content.

[0010] According to embodiments of this application, the method further includes: obtaining historical task types and historical task content that match the target text; correcting the target task type using the historical task type and the expected task type to generate a second corrected task type; and correcting the target task type using the historical task content and the expected task content to generate a second corrected task content.

[0011] Another aspect of this application provides a speech recognition device, comprising: an extraction module, configured to, in response to receiving a speech signal from a terminal device deployed at a power construction site, extract a target speech signal from the denoised speech signal by jointly analyzing the energy distribution features and spectral entropy features of the denoised speech signal; wherein the target speech signal is a semantically continuous audio sequence; a segmentation module, configured to perform frequency domain segmentation on the frequency domain features of the target speech signal to obtain a first frequency feature and a second frequency feature; wherein the first frequency feature characterizes the global semantics of the target speech signal; and the second frequency feature characterizes the keyword semantics of the target speech signal; and a first fusion module, configured to dynamically adjust the first frequency feature and the second frequency feature... The first and second frequency features are fused based on a multi-head self-attention mechanism to generate an initial fused feature, with their respective weights. Specifically, when the target speech signal contains noise, the weight of the first frequency feature is greater than the weight of the second frequency feature; when the target speech signal contains keywords, the weight of the first frequency feature is less than the weight of the second frequency feature. A second fusion module is used to modify the initial fused feature using power industry terminology that matches it, based on an interactive attention mechanism, to generate a target semantic feature. The target semantic feature corrects the colloquial expressions in the initial semantic feature. A recognition module is used to recognize the target semantic feature and generate a target task type and target task content.

[0012] Another aspect of this application provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the methods described above.

[0013] Another aspect of this application provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform any of the methods described above.

[0014] According to embodiments of this application, in response to receiving a voice signal from a terminal device deployed at a power construction site, a target voice signal is extracted from the denoised voice signal by jointly analyzing its energy distribution characteristics and spectral entropy characteristics, thereby obtaining a stable target voice signal and improving the speed and accuracy of subsequent target voice signal processing. The frequency domain features of the target voice signal are divided into first and second frequency features, thereby extracting global semantic features and fine-grained voice features contained in the target voice signal. By dynamically adjusting the weights of the first and second frequency features, the first and second frequency features are fused based on a multi-head self-attention mechanism to generate initial fused features, thus preserving the phoneme, prosody, and contextual semantic features of the target voice signal and reducing the loss of acoustic information. Based on an interactive attention mechanism, power industry terminology matching the initial fused features is used to modify the initial fused features, generating target semantic features; the target semantic features correct the colloquial expressions in the initial semantic features, facilitating the identification of the task type and task content of the target semantic features. The target semantic features are identified to generate the target task type and target task content, thereby improving the accuracy of speech recognition methods in scenarios with strong noise or drastic accent changes. Attached Figure Description

[0015] The above and other objects, features and advantages of this application will become clearer from the following description of embodiments of this application with reference to the accompanying drawings, in which the accompanying drawings are shown.

[0016] Figure 1 The present invention illustrates application scenarios where the speech recognition method can be applied according to embodiments of the present application.

[0017] Figure 2 A flowchart of a speech recognition method according to an embodiment of this application is shown.

[0018] Figure 3 A schematic diagram is shown of an acoustic representation encoder extracting speech features according to an embodiment of this application.

[0019] Figure 4 A schematic diagram of extracting target speech signals according to an embodiment of this application is shown.

[0020] Figure 5 A schematic diagram of a speech recognition method according to an embodiment of this application is shown.

[0021] Figure 6 A block diagram of a speech recognition device according to an embodiment of this application is shown.

[0022] Figure 7 A block diagram of an electronic device 700 suitable for implementing a speech recognition method according to an embodiment of this application is shown. Detailed Implementation

[0023] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0027] In voice interaction scenarios within the power industry, the input speech signals for speech recognition exhibit diverse accents, strong background noise, uneven speech rates, and colloquial semantics. Furthermore, speech recognition relies on industry-specific terminology. Current technologies employ a cascaded pipeline architecture for speech recognition, first converting the speech signal into text via automatic speech recognition, then inputting the text into a natural language understanding module for task type and content identification. This approach suffers from the problem that errors generated during the automatic speech recognition stage are propagated to the natural language understanding module, leading to accumulated errors. The cascaded pipeline architecture also suffers from accumulated processing latency, reducing speech recognition efficiency.

[0028] In some approaches, word segmentation can be performed on the input speech signal, and feature vectors can be extracted from the segmented words. By comparing the similarity between the feature vectors and semantic vectors, intent information based on the power industry can be obtained. However, this approach can only achieve intent recognition and cannot simultaneously identify task type and task content. Alternatively, a corpus dataset can be constructed using local dialects, local Mandarin, and standard Mandarin. This corpus dataset can then be used to train a speech recognition model to recognize power grid dispatch speech signals in the power industry. However, this approach is limited to recognizing dispatch speech, thus restricting the application scenarios of speech recognition.

[0029] These methods have limited application scenarios for speech recognition in the power industry, and their speech recognition efficiency is low in high-noise environments such as power construction sites.

[0030] In view of the above, embodiments of this application provide a speech recognition method, apparatus, electronic device, and computer-readable storage medium.

[0031] Figure 1 An application scenario 100 in which the speech recognition method can be applied according to an embodiment of this application is shown. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this application, in order to help those skilled in the art understand the technical content of this application, but do not mean that the embodiments of this application cannot be used in other devices, systems, environments or scenarios.

[0032] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0033] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social media platform software, etc. (for example only).

[0034] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0035] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0036] It should be noted that the speech recognition method provided in this application embodiment can generally be executed by server 105. Correspondingly, the speech recognition device provided in this application embodiment can be located in server 105. The speech recognition method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the speech recognition device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Alternatively, the speech recognition method provided in this application embodiment can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the voice recognition device provided in this application embodiment can also be disposed in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or disposed in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0037] For example, the voice signal may be originally stored in any one of the first terminal device 101, the second terminal device 102, or the third terminal device 103 (e.g., the first terminal device 101, but not limited thereto), or it may be stored on an external storage device and imported into the first terminal device 101. Then, the first terminal device 101 may execute the voice recognition method provided in the embodiments of this application locally, or send the voice signal to other terminal devices, servers, or server clusters, and have the other terminal devices, servers, or server clusters that receive the voice signal execute the voice recognition method provided in the embodiments of this application.

[0038] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0039] Figure 2 A flowchart of a speech recognition method according to an embodiment of this application is shown.

[0040] like Figure 2 As shown, the speech recognition method may include operations S210 to S250.

[0041] In operation S210, in response to receiving a voice signal from a terminal device deployed at a power construction site, the target voice signal is extracted from the denoised voice signal by performing a joint analysis of the energy distribution characteristics and spectral entropy characteristics of the denoised voice signal.

[0042] The target speech signal is a semantically continuous audio sequence.

[0043] Voice signals can be captured by terminal devices with audio acquisition capabilities, such as terminal devices with wired microphones, wireless microphones, or headphones. Voice signals can be voice input carried by the user through the terminal device during voice interaction, reflecting the user's needs.

[0044] At power construction sites, voice signals from terminal equipment may contain background noise or electrical noise. Therefore, upon receiving a voice signal from a terminal equipment, an adaptive noise reduction unit is first used to reduce the noise, resulting in a denoised voice signal that at least partially suppresses background noise and electrical interference. User needs may include telephone customer service inquiries, dispatch instructions, or voice self-service.

[0045] The energy distribution characteristics of a speech signal reflect its intensity variations. For example, semantically continuous speech segments have concentrated and continuous energy, while silent or noisy segments exhibit abrupt energy changes. The spectral entropy characteristics of a speech signal reflect its complexity. For instance, semantically continuous speech segments have stable spectral entropy, while noise segments exhibit significant fluctuations in spectral entropy. Therefore, by jointly analyzing the energy distribution and spectral entropy characteristics of a speech signal, a semantically continuous audio sequence, i.e., the target speech signal, can be extracted from the denoised speech signal.

[0046] In operation S220, the frequency domain features of the target speech signal are divided into frequency domains to obtain the first frequency features and the second frequency features.

[0047] Among them, the first frequency feature represents the global semantics of the target speech signal; the second frequency feature represents the keyword semantics of the target speech signal.

[0048] As one implementation method, an acoustic representation encoder can be used to segment the frequency domain features of a target speech signal to obtain speech features. Speech features may include first frequency features and second frequency features. Speech features can be represented using dense vector form to retain more speech features of the target speech signal. For example, speech features may be prosodic features such as speech rate, stress, or pauses, or phoneme features such as the speaker's accent. By processing speech signals from different noise environments or different speakers, an acoustic representation encoder can extract phoneme and prosodic features from the speech signal, thereby at least solving the problem of low recognition accuracy caused by interfering phonemes such as accents and noise in speech recognition scenarios.

[0049] Figure 3 A schematic diagram is shown of an acoustic representation encoder extracting speech features according to an embodiment of this application.

[0050] like Figure 3 As shown, the acoustic representation encoder can be a pre-trained speech model (e.g., a hybrid structure of WavLM-Base and lightweight Conformer layers). WavLM-Base (Waveform Language Model - Base, a unified pre-trained model for speech and language understanding) serves as the base. WavLM-Base is configured with approximately 94M parameters, a 12-layer Transformer Encoder, with a hidden dimension d=768 per layer and 12 attention heads h=12. Inference speed can be improved and overfitting risk reduced by freezing the parameters of the first 6 layers and fine-tuning the parameters of the last 6 layers. Two additional lightweight Conformer layers are cascaded on top of the output of WavLM-Base to enhance the local time-frequency modeling capability of the acoustic representation encoder (each lightweight Conformer layer has a convolution kernel size k=31, a dot product attention head number h=8, and a feedforward network (FNN) dimensionality expansion coefficient set to 4), thereby better capturing the non-stationary noise features of the speech signal.

[0051] As one implementation method, upon receiving a speech signal, the target speech signal extracted from the denoised speech signal is: The speech features of the target speech signal are extracted by the acoustic representation encoder. ,in, is a vector of discrete sampling points of the target speech signal, where T is the number of sampling points of the speech signal, d is the dimension of the encoder hidden layer, and L is the number of time frames obtained after the speech is processed by the acoustic representation encoder. Then... Input the Transformer encoder for context modeling.

[0052] In operation S230, by dynamically adjusting the weights of the first frequency feature and the second frequency feature, the first frequency feature and the second frequency feature are fused based on a multi-head self-attention mechanism to generate an initial fused feature.

[0053] Specifically, when the target speech signal contains noise, the weight of the first frequency feature is greater than the weight of the second frequency feature; when the target speech signal contains keywords, the weight of the first frequency feature is less than the weight of the second frequency feature.

[0054] In one implementation, the first frequency feature can be a low-frequency feature of the target speech signal, and the second frequency feature can be a high-frequency feature of the target speech signal. The multi-head self-attention mechanism can dynamically adjust the weights of the first and second frequency features. For example, in situations with strong noise or drastic accent changes, the multi-head self-attention mechanism will increase...

[0055] The initial fusion features characterize the phonemes, prosody, and semantic features of the fused target speech signal.

[0056] Multi-head self-attention mechanisms can map semantic features of phoneme, prosody, and context dimensions to multiple independent subspaces. Each subspace corresponds to a set of attention heads, meaning different attention heads focus on different dimensions of speech features. For example, some attention heads focus on phoneme associations, some focus on rhythmic changes in prosody, and some capture the semantic logic of the context.

[0057] One implementation method is to use a multi-layer deep self-attention network (Transformer Encoder) as the backbone structure. Feature fusion is performed using a multi-head self-attention mechanism layer within the multi-layer deep self-attention network.

[0058] Multilayer deep self-attention networks lack sequence position awareness, therefore, it is necessary to improve the sensitivity of speech features H. (1) Add positional coding. Adding positional coding can be expressed as formula (1).

[0059] (1)

[0060] in These are speech features with added positional coding; PE stands for positional coding.

[0061] The position code can be a sinusoidal position code, represented by formula (2).

[0062] (2)

[0063] Where pos is the index of the speech feature in the sequence, d is the hidden layer dimension of the speech feature, and i is the index of the current dimension. It is the sinusoidal encoded value of the 2i-th dimension of the feature vector at position pos. It is the cosine encoding value of the (2i+1)th dimension of the feature vector at position pos.

[0064] The processing of the l-th layer by the multi-head self-attention mechanism can be represented by formula (3).

[0065] (3)

[0066] in, . This represents the output of the previous Transformer layer (or the speech features from layer 1), with dimensions of... ,in d is the sequence length and d is the feature dimension. These are linear transformation matrices for the query, key, and value, respectively. Query matrix (Query), with dimensions of , is used to indicate the type of information that the current feature vector wants to focus on; The key matrix (Key) has dimensions of , representing the content tags at each position in the sequence, used for matching with the query; Value matrix, with dimensions of Z represents the feature information that will actually be weighted and aggregated. l This represents the output feature of the l-th layer.

[0067] In a feedforward network (FFN), each Transformer layer contains two feedforward layers: , It is a two-level linear transformation matrix. and This is a bias term. It is ReLU or GELU. Each sub-layer is connected to a normalized layer (LayerNorm) using residual connections, and the final output can be represented by formula (4).

[0068] (4)

[0069] Where S is the initial fusion feature. Z L This represents the output feature of the Lth layer.

[0070] By performing parallel computation with multiple attention heads, the semantic features of multi-dimensional target speech signals, such as phonemes, prosody, and context, are fused to achieve the fusion of multi-dimensional semantic features and generate initial fused features.

[0071] In operation S240, based on the interactive attention mechanism, the initial fusion features are modified by using power industry terms that match the initial fusion features, thereby generating target semantic features.

[0072] The target semantic features corrected the colloquial expressions in the initial semantic features.

[0073] Initial fusion features characterize the phonemes, prosody, and semantic features of the context of the speech signal.

[0074] The initial fusion features are obtained from processed speech signals, and their semantic features may include colloquialisms or speaker accents. These features do not perfectly match the specialized terminology of the power industry. Power industry terminology is prior knowledge obtained from a predefined knowledge base, which can be used to refine the initial fusion features, thereby enhancing the applicability of the speech recognition method to the power industry.

[0075] The target semantic features generated by modifying the initial fusion features using power industry terminology can be applied to subsequent task type identification and task content identification.

[0076] In operation S250, the target semantic features are identified, and the target task type and target task content are generated.

[0077] The target task type can be scene-level recognition obtained by coarse-grained recognition of target semantic features. The target task content can be fine-grained recognition, such as specific requirements, obtained by coarse-grained recognition of target semantic features.

[0078] According to embodiments of this application, in response to receiving a speech signal from a power construction site, a target speech signal is extracted from the denoised speech signal by jointly analyzing its energy distribution characteristics and spectral entropy characteristics, thereby obtaining a stable target speech signal and improving the speed and accuracy of subsequent target speech signal processing. The frequency domain features of the target speech signal are divided into first and second frequency features, thereby extracting global semantic features and fine-grained speech features contained in the target speech signal. By dynamically adjusting the weights of the first and second frequency features, the first and second frequency features are fused based on a multi-head self-attention mechanism to generate initial fused features, thus preserving the phoneme, prosody, and contextual semantic features of the target speech signal and reducing the loss of acoustic information. Based on an interactive attention mechanism, power industry terminology matching the initial fused features is used to modify the initial fused features, generating target semantic features; the target semantic features correct the colloquial expressions in the initial semantic features, facilitating the identification of the task type and task content of the target semantic features. The target semantic features are identified to generate the target task type and target task content, thereby improving the accuracy of speech recognition methods in scenarios with strong noise or drastic accent changes.

[0079] Figure 4 A schematic diagram of extracting target speech signals according to an embodiment of this application is shown.

[0080] The following will combine Figure 4 The process of extracting the target speech signal in the speech recognition method of the present application embodiment will be described.

[0081] like Figure 4 As shown, the process of extracting the target speech signal may include: receiving the speech signal, performing noise reduction and gain normalization on the speech signal, speech activity detection, and segmentation. This process may include operations C1 to C4.

[0082] In operation C1, in response to receiving a voice signal from a terminal device deployed at the power construction site, a continuous voice signal is extracted from the noise-reduced voice signal.

[0083] It can perform noise reduction and gain normalization on the speech signal to obtain the noise-reduced speech signal.

[0084] As one implementation method, a short-time Fourier transform is first performed on the speech signal to obtain the complex spectrum of the speech signal. Denoising is performed using a deep neural network-based denoising system. The deep neural network-based denoising system utilizes an improved deep complex spectrum mapping model to perform denoising. The denoising process can be represented by formula (5).

[0085] (5)

[0086] in, It is the enhanced complex spectrum output by the noise reduction system. It is a complex mask predicted by a deep neural network. This represents element-wise complex multiplication.

[0087] To reduce the impact of excessively large or small input amplitude of the speech signal on subsequent feature extraction, it is necessary to perform gain normalization on the enhanced complex spectrum output by the denoising system, so that the root mean square amplitude of the denoised speech signal is maintained within a preset dynamic range. ,in, It is the preset minimum dynamic range value. It is the preset maximum dynamic range.

[0088] Continuous speech signals are extracted from the denoised speech signals to obtain semantically continuous speech segments.

[0089] As one implementation method, an end-to-end lightweight convolutional recurrent neural network (CRNN) can be used to determine frame-level speech segments and silence segments. The input to the CRNN is a speech signal with a frame length of 25 ms and a frame shift of 10 ms. In the CRNN, multi-feature fusion (energy threshold, Mel-frequency cepstral coefficient change rate, and endpoint detection depth model) is used to extract the F-bank features corresponding to each frame of the speech signal. The CRNN then classifies the F-bank features of each frame and outputs a binary label for each frame. Based on the binary tag corresponding to each frame, silent frames are discarded, and continuous speech frames are retained to obtain a continuous speech signal.

[0090] In operation C2, the energy distribution characteristics and spectral entropy characteristics of the continuous speech signal are jointly analyzed to obtain the target segmentation point.

[0091] The energy distribution characteristics of a continuous speech signal can reflect the intensity of the speech signal within a frame. For example, the energy of the speech component in a speech signal is higher than that of the background noise. Therefore, by calculating the energy distribution characteristics of the speech signal, the speech and non-speech components can be distinguished, and thus, the intensity of the speech signal within a frame can be obtained. Its short-term energy It can be calculated according to formula (6).

[0092] (6)

[0093] Where N is the frame length.

[0094] Spectral entropy features reflect the frequency distribution in a speech signal and can be used to help determine the endpoints of speech. The calculation process for spectral entropy can be as follows: First, perform a Fast Fourier Transform (FFT) on each frame of the speech signal to obtain the amplitude of each frequency component. Then, calculate the normalized spectral probability density function. (Normalized spectral probability density function) It can be calculated according to formula (7).

[0095] (7)

[0096] in It is the component of the qth frequency in the l-th frame, and N is the FFT length.

[0097] Finally, the spectral entropy is calculated based on the normalized spectral probability density function. It can be calculated according to formula (8).

[0098] (8)

[0099] Joint analysis of energy distribution characteristics and spectral entropy characteristics of continuous speech signals yields target segmentation points. Target split point It can be expressed by formula (9).

[0100] (9)

[0101] in The short-term energy at the target cutting point. The short-time spectral entropy of the target segmentation point. Represents the statistical mean of short-term energy; The standard deviation represents the short-time energy. The adjustment coefficient representing short-term energy; The statistical mean of the short-time spectral entropy; The standard deviation represents the short-time spectral entropy; This represents the adjustment coefficient for short-time spectral entropy.

[0102] The continuous speech signal is segmented based on the target segmentation point to obtain the target speech signal.

[0103] The target segmentation point is the semantic breakpoint of a continuous speech signal obtained through joint analysis of energy distribution characteristics and spectral entropy characteristics. A continuous speech signal may contain multiple semantics. Segmenting the continuous speech signal according to the target segmentation point can divide the continuous speech signal without destroying its semantics, thereby obtaining the target speech signal.

[0104] In operation C3, the continuous speech signal is segmented based on the target segmentation point to obtain the segmented speech signal.

[0105] The segmented speech signal has removed noise and semantic redundancy, but it may still have problems such as high-frequency attenuation or spectrum leakage.

[0106] In operation C4, the segmented speech signal is pre-emphasized and weighted by a symmetrical Hanning window to obtain the target speech signal.

[0107] The segmented speech signal is pre-emphasized and filtered to compensate for high-frequency components. Symmetric Hanning window weighting reduces spectral leakage. As one implementation, the segmented speech signal can be pre-emphasized and filtered with a coefficient of 0.97 and then a symmetric Hanning window function can be applied to obtain the target speech signal.

[0108] According to embodiments of this application, the speech signal is denoised and gain normalized to obtain a denoised speech signal, thereby suppressing background noise and electrical interference and reducing the impact of speech signal volume differences on subsequent recognition. Continuous speech signals are extracted from the denoised speech signal to remove invalid segments and reduce redundant computation. Joint analysis of energy distribution characteristics and spectral entropy characteristics is performed on the continuous speech signal to obtain target segmentation points. Based on these segmentation points, the continuous speech signal is segmented to obtain the target speech signal, resulting in a target speech signal with high signal-to-noise ratio, amplitude normalization, temporal integrity, and consistent sampling rate, facilitating direct feature extraction. The continuous speech signal is segmented based on the target segmentation points to obtain segmented speech signals. Pre-emphasis filtering and symmetrical Hanning window weighting are then applied to the segmented speech signals to alleviate high-frequency attenuation and reduce spectral leakage.

[0109] The following explains the process of dividing the frequency domain features of the target speech signal to obtain the first frequency feature and the second frequency feature.

[0110] Dividing the frequency domain features of the target speech signal into frequency domain features to obtain the first frequency features and the second frequency features may include operations A1 to A3.

[0111] In operation A1, a Fourier transform is performed on the time-domain features of the target speech signal to generate the frequency-domain features of the target speech signal.

[0112] To explicitly model the target speech signal in the frequency domain, a short-time Fourier transform can be performed on the time-domain waveform or intermediate features of the target speech signal to obtain its frequency domain features, i.e., its complex spectrum. , where F is the number of frequency bands.

[0113] In operation A2, the frequency domain features of the target speech signal are processed by calling the activation function to generate a frequency domain mask.

[0114] The process of generating a frequency domain mask by processing the frequency domain features of the target speech signal through activation functions is a learnable process. The frequency domain mask can be dynamically adjusted by modifying learnable parameters to adapt to different target speech signals.

[0115] Frequency domain masks can include low-frequency domain masks and high-frequency domain masks.

[0116] The process of generating the frequency domain mask can be represented by formulas (10) and (11).

[0117] (10)

[0118] (11)

[0119] in, It is a low-frequency domain mask. Here, f is a high-frequency domain mask, a and b are learnable parameters, and σ(·) is the activation function. As one implementation, the activation function can be the sigmoid function.

[0120] In operation A3, the frequency domain features of the target speech signal are divided into frequency domains using a frequency domain mask to obtain the first frequency feature and the second frequency feature.

[0121] The process of dividing the frequency domain features of the target speech signal into frequency domains using a low-frequency frequency domain mask to obtain the first frequency feature can be represented by formula (12).

[0122] (12)

[0123] in, It is the first frequency characteristic. This represents element-wise complex multiplication. This represents the target speech signal.

[0124] The process of dividing the frequency domain features of the target speech signal into frequency domains using a high-frequency domain mask to obtain the second frequency features can be represented by formula (13).

[0125] (13)

[0126] in, It is the second frequency characteristic.

[0127] The first frequency feature and the second frequency feature can be two complementary speech signal frequency features, which can respectively represent the low-frequency component and the high-frequency component in the speech signal.

[0128] According to embodiments of this application, the time-domain features of the target speech signal are transformed into frequency-domain features by performing a Fourier transform, thereby reflecting the energy distribution of the target speech signal at different frequencies. An activation function is then used to process the frequency-domain features of the target speech signal to generate a frequency-domain mask, which helps distinguish between low-frequency and high-frequency features in the frequency domain of the speech signal, thereby improving the accuracy of speech recognition.

[0129] By dynamically adjusting the weights of the first and second frequency features, the first and second frequency features are fused based on a multi-head self-attention mechanism to generate initial fused features, which may include operations B1 to B4.

[0130] In operation B1, according to the frequency dimension of the first frequency feature, the learnable first weight matrix is ​​called to perform weighted aggregation on the first frequency feature to obtain the first aggregated feature.

[0131] Among them, the first aggregation feature focuses on the energy envelope, prosodic contour, and global semantic structure that changes over time in the target speech signal.

[0132] Weighted aggregation of the first frequency features can map the first frequency features to a low-frequency frame-level temporal representation. The process of obtaining the first aggregated feature by weighted aggregation of the first frequency features can be represented by formula (14).

[0133] (14)

[0134] in, It is the first aggregation feature. It is the first weight matrix. It is the first linear transformation matrix. As one implementation, the first weight matrix can be derived from... The derived normalized weight matrix is ​​a learnable weight matrix.

[0135] In operation B2, according to the frequency dimension of the second frequency feature, the learnable second weight matrix is ​​called to perform weighted aggregation on the second frequency feature to obtain the second aggregated feature.

[0136] Among them, the second aggregation feature focuses on the articulation units and articulation abrupt change boundaries in the target speech signal.

[0137] Weighted aggregation of the second frequency features can map the second frequency features into a high-frequency frame-level temporal representation.

[0138] The second aggregation feature focuses on the articulatory units and articulatory transition boundaries in the target speech signal. The articulatory units and articulatory transition boundaries can represent fine-grained features such as consonants or plosives, thereby enabling the speech recognition method to have a higher ability to distinguish keywords such as tripping, disconnecting switch, switching or grounding in power scenarios.

[0139] The process of weighted aggregation of the second frequency features to obtain the second aggregated features can be represented by formula (15).

[0140] (15)

[0141] in, It is the second aggregation feature. It is the second weight matrix. This is the second linear transformation matrix. As one implementation, the second weight matrix can be derived from... The derived normalized weight matrix is ​​a learnable weight matrix.

[0142] In operation B3, the first aggregated feature and the second aggregated feature are processed by calling the probability distribution function to generate the weights of the first frequency feature and the second frequency feature respectively.

[0143] By calling the probability distribution function, the priority of the first frequency feature and the second frequency feature is determined, thereby judging the degree of contribution of the two frequency features to speech recognition, and then generating the weights of the first frequency feature and the second frequency feature respectively.

[0144] To achieve unified modeling of the first and second aggregated features, frame-level fusion processing can be performed on them. For each time step t, the weights of the first and second frequency features are calculated. It can be expressed by formula (16).

[0145] (16)

[0146] in, It is a learnable matrix. It's a splicing operation.

[0147] In operation B4, based on the multi-head attention mechanism, the first frequency feature and the second frequency feature are fused according to their respective weights to obtain the initial fused feature.

[0148] According to the respective weights of the first frequency feature and the second frequency feature, the process of fusing the first frequency feature and the second frequency feature can be represented by formula (17).

[0149] (17)

[0150] in, It is a global modeling of the first and second frequency features after frame-level fusion at time step t. It is the weight of the first frequency feature. It is the weight of the second frequency feature.

[0151] According to embodiments of this disclosure, by setting different weights for the first frequency feature and the second frequency feature respectively, the two frequency features are fused, so that the global semantic information and keyword information in the speech signal can be effectively utilized, thereby improving the accuracy of the speech recognition method.

[0152] To enhance the applicability of speech recognition methods in the power industry, a pre-defined knowledge base can be introduced to refine the representation of the initial fusion features. The following explains how to obtain power industry terms from the pre-defined knowledge base that match the initial fusion features.

[0153] Retrieve power industry terms that match the initial fusion characteristics from a predefined knowledge base, which may include operations D1 to D2.

[0154] In operation D1, the initial fused features are averaged and pooled to generate a semantic query vector.

[0155] To facilitate matching power industry terms from a predefined knowledge base, the multiple semantic features included in the initial fusion feature S need to be transformed into a fixed-format vector form, namely the semantic query vector q, through average pooling.

[0156] In operation D2, power industry terms are retrieved from a predefined knowledge base based on semantic query vectors.

[0157] A predefined knowledge base can store a set of professional knowledge related to the power industry. This professional knowledge can include specialized terms such as power business terminology (e.g., tripping, closing, or electricity charges) or business logic (e.g., power outage reporting or billing inquiries). These specialized terms can be stored in the predefined knowledge base as vectors, and external knowledge encoders can be used to encode these vectors to obtain a knowledge vector table. , where M is the total number of knowledge items in the knowledge vector table.

[0158] To query power industry terminology from a pre-defined knowledge base, the semantic query vector q can be sequentially compared with the specialized word vectors in the knowledge vector table stored in the pre-defined knowledge base to obtain multiple candidate knowledge items. , This represents the number of knowledge items that match the semantic query vector. Multiple candidate knowledge items are considered as power industry terms. Vector similarity retrieval can be performed using cosine similarity or inner product calculation.

[0159] According to embodiments of this application, by generating semantic query vectors and querying corresponding power industry terms from a predetermined knowledge base, biases in power industry speech recognition can be reduced and the generalization ability of speech recognition methods can be improved.

[0160] Based on the interactive attention mechanism, the initial fusion features are modified by using power industry terms that match the initial fusion features, and the generated target semantic features may include operations E1 to E4.

[0161] In operation E1, the initial fusion features and power industry terms are fused based on the interactive attention mechanism to generate semantically enhanced fusion features.

[0162] The semantically enhanced fusion features strengthen the industry terminology in the initial fusion features.

[0163] The interactive attention mechanism was used to analyze the initial fused features S and power industry terms. By integrating the semantic features of speech signals with information from the power industry, semantically enhanced fusion features can be generated.

[0164] The process of fusing initial fusion features and power industry terms based on the interactive attention mechanism can be represented by formula (18).

[0165] (18)

[0166] Where A represents the semantically enhanced fusion feature. , and These are linear transformation matrices representing the query, key, and value, respectively. It is a scaling factor for the vector dimension, used to normalize the attention score to avoid gradient instability caused by excessively large values.

[0167] The semantically enhanced fusion features and the initial fusion features are weighted and fused to generate the target semantic features.

[0168] After obtaining the semantically enhanced fusion features, it is necessary to further balance the contributions of the initial fusion features and the semantically enhanced fusion features to speech recognition. Therefore, the semantically enhanced fusion features and the initial fusion features can be weighted and fused to obtain target semantic features that can reflect both the original semantics of the speech signal and the professionalism of the power industry.

[0169] In weighted fusion, different weights need to be assigned to the semantically enhanced fusion features and the initial fusion features. The weights can be fixed, meaning that the semantically enhanced fusion features and the initial fusion features are weighted with the same weights when performing speech recognition on different speech signals. Alternatively, the weights can be dynamic, meaning that different weights are assigned to the initial fusion features and their corresponding semantically enhanced fusion features for different speech signals.

[0170] In operation E2, the semantically enhanced fused features and the initial fused features are averaged and pooled to generate intermediate features.

[0171] Average pooling is performed on the semantically enhanced fusion features and the initial fusion features to smooth the impact of local noise on the features. Then, intermediate features are generated through feature integration to improve the accuracy of target weight generation.

[0172] In operation E3, a multilayer perceptron is used to process the intermediate features and generate target weights.

[0173] By learning intermediate features, multilayer perceptrons can dynamically output target weights that adapt to the current speech recognition scenario, thereby dynamically adjusting the proportion of semantically enhanced fusion features and initial fusion features in weighted fusion.

[0174] In operation E4, the semantically enhanced fusion features and the initial fusion features are weighted and fused based on the target weights to generate the target semantic features.

[0175] The process of weighted fusion of semantically enhanced fusion features and initial fusion features based on target weights can be represented by formula (19).

[0176] (19)

[0177] in, For target semantic features, Let A be the target weight, A be the semantically enhanced fusion feature, and S be the initial fusion feature.

[0178] According to embodiments of this application, initial fusion features and power industry terminology are fused based on an interactive attention mechanism to generate semantically enhanced fusion features. This enables the association of the initial fusion features obtained from the speech signal with power industry terminology in the knowledge base. Weighted fusion of the semantically enhanced fusion features and the initial fusion features is then performed to generate target semantic features, making them more aligned with the business scenarios of the power industry and improving the accuracy of subsequent task type and content recognition. Average pooling is used to smooth the semantically enhanced fusion features and the initial fusion features, generating intermediate features that reduce the impact of noise in the intermediate features. A multilayer perceptron is used to process the intermediate features to generate target weights, which can be dynamically adjusted. Weighted fusion of the semantic features and the initial fusion features is then performed based on the target weights, ensuring that the contribution of the two features matches the scenario and improving the accuracy of feature fusion.

[0179] To address the issue of inaccurate speech recognition results, this application also includes a rollback mechanism to correct the output task type and content.

[0180] To measure the accuracy of speech recognition methods, confidence levels can be evaluated for both the target task type and the target task content. The accuracy of each method can be judged based on the maximum probability confidence level. The maximum probability confidence level can be expressed as:

[0181] (20) Among them, It represents the highest confidence level for the target task type. It represents the highest confidence level of the target task content.

[0182] The process of triggering the rollback mechanism can be defined as follows:

[0183] ,if ,otherwise .

[0184] in, It is the lowest confidence level for the target task type. It represents the minimum confidence level of the target task content. Indicates a rule conflict (e.g., the target task type and target task content are not in a predefined mapping table). This indicates that the rollback mechanism has been triggered. This indicates normal output.

[0185] If the highest confidence level of the target task type or target task content is lower than a preset security threshold, or if the mapping relationship between the target task type and target task content is incorrect, it will be determined that the target task type and target task content do not match, thereby triggering a rollback mechanism to correct the target task type and target task content. For example, if the target task type is identified as fault warranty and the target task content is as balance inquiry, it will be determined that the target task type and target task content are not correctly associated.

[0186] Identifying the semantic features of the target and generating the target task type and target task content can include operations F1 to F6.

[0187] In operation F1, the target semantic features are used to identify the target task type, thus obtaining the target task type.

[0188] Task type identification performs scenario-level recognition of target semantic features. For example, by identifying the target task type from the target semantic features, the target task type can be a repair request, business inquiry, or complaint feedback, thus completing a coarse-grained scenario-level identification.

[0189] Task type identification is performed on the target semantic features, resulting in a fixed-dimensional class probability distribution vector. Each class probability distribution vector represents the confidence level for each predefined question type. It can be expressed by formula (21).

[0190] (twenty one)

[0191] Where, N c It is the number of predefined question types. This represents the probability that the target semantic feature belongs to the j-th predefined problem type.

[0192] Based on the task type identification results, the category with the highest probability is selected from the category probability distribution vector as the target task type obtained by identifying the target semantic features.

[0193] In operation F2, the target semantic features are used to identify the task content, thus obtaining the target task content.

[0194] Task type identification identifies specific user needs based on target semantic features. For example, task content identification based on target semantic features could yield task content such as checking bills, modifying service plans, or applying for new services, thus achieving fine-grained identification of specific user needs.

[0195] Task content recognition based on target semantic features also yields a fixed-dimensional task content probability distribution vector. Each task content probability distribution vector represents the confidence level of each predefined task content. It can be expressed by formula (22).

[0196] (twenty two)

[0197] Where, N k It is the number of predefined task contents. This represents the probability that the target semantic feature belongs to the k-th predefined task content.

[0198] Based on the task content recognition results, the category with the highest probability is selected from the task content probability distribution vector as the target task content obtained by recognizing the target semantic features.

[0199] Before sending the target task type and content, first ensure that the target task type and content match. For example, if the target task type is a business inquiry, the target task content should be checking bills, modifying service plans, or applying for new services. Sending the target task type and content only after confirming a match ensures the accuracy of the speech recognition results.

[0200] When the target task type and target task content do not match, a fallback mechanism can be used to correct the speech recognition results.

[0201] The following describes the processing steps for correcting speech recognition results using a rollback mechanism.

[0202] When operating F3, in response to the determination that the target task type and target task content do not match, speech recognition is performed on the denoised speech signal to generate the target text.

[0203] When the target task type and target task content are determined to be mismatched, Automatic Speech Recognition (ASR) technology can be used to re-recognize the noise-reduced speech signal and convert the speech signal into target text.

[0204] When operating F4, intent recognition is performed on the target text to generate the target intent; the target intent includes the expected task type and the expected task content.

[0205] Given the target text, Natural Language Understanding (NLU) methods can be used to identify the expected task type and expected task content of the target text and generate the target intent.

[0206] When operating F5, the target task type is corrected using the expected task type, and the first corrected task type is generated.

[0207] When operating F6, the target task content is corrected using the expected task content, and the first corrected task content is generated.

[0208] The expected task type and expected task content are identified using the NLU method. The mismatched target task type and target task content are corrected to generate the first corrected task type and the first corrected task content.

[0209] After correcting the target task type and target task content using the rollback mechanism, the first corrected task type and first corrected task content are sent to improve the accuracy of speech recognition.

[0210] According to embodiments of this application, when the target task type and target task content are determined to match, the target task type and target task content are sent to improve the accuracy of the speech recognition method. When the target task type and target task content do not match, speech recognition is performed on the denoised speech signal to generate target text. Intent recognition is then performed based on the target text to generate target intent, thereby obtaining corrected content. A backoff mechanism reduces misjudgments in speech recognition.

[0211] The rollback mechanism can also correct the target task type and content based on historical task data. The following explains how to correct the target task type and content based on historical data. Correcting the target task type and content based on historical data can include operations G1 through G3.

[0212] In operation G1, retrieve the historical task type and historical task content that match the target text.

[0213] Historical task types and content can be historical data on identified and confirmed task types and content from previous voice interaction scenarios. This historical data can be stored in a historical knowledge base built upon historical tasks. Similarity matching or frequently asked questions (FAQ) systems can be used to retrieve historical task types and content that match the target text from the historical knowledge base, thereby correcting the speech recognition results.

[0214] In operation G2, the target task type is corrected using historical task types and expected task types, and a second corrected task type is generated.

[0215] In G3, the target task type is corrected by using historical task content and expected task content, and a second corrected task content is generated.

[0216] After obtaining the target intent using the NLU method, the target task type is corrected a second time using historical task types and expected task types, and the target task content is corrected a second time using historical task content, generating the second corrected task type and task content, thereby completing the correction of the speech recognition result.

[0217] According to an embodiment of this application, a second corrected task content and a second corrected task type are obtained based on historical task types and historical task content. Through an error correction mechanism based on historical data, the accuracy of speech recognition is improved.

[0218] Figure 5 A schematic diagram of a speech recognition method according to an embodiment of this application is shown.

[0219] like Figure 5 As shown, upon receiving a speech signal, preprocessing is first performed. Preprocessing may include noise reduction, gain normalization, speech activity detection, and sampling rate unification. Noise reduction and gain normalization are applied to unify the speech signal amplitude, resulting in a denoised speech signal. This suppresses background noise and electrical interference, reducing the impact of volume differences on subsequent recognition. Speech activity detection identifies speech segments and silence segments in the denoised speech signal, segments them, and removes invalid segments to obtain a continuous speech signal. Then, a sampling rate unification module converts the sampling rate of the continuous speech signal to a preset optimal sampling rate. The preset sampling rate can be set to 16 kHz to ensure consistency between the sampling rate of the continuous speech signal and the acoustic encoder parameters. Finally, joint analysis of the energy distribution characteristics and spectral entropy characteristics of the continuous speech signal is performed to obtain target segmentation points. Based on these target segmentation points, the continuous speech signal is segmented to obtain the target speech signal.

[0220] As one implementation method, an audio acquisition device with a sampling frequency range of 8 kHz–48 kHz and a bit depth supporting 16-bit / 24-bit can be used to acquire speech signals. The audio signal is then converted to a digital signal via an analog-to-digital conversion and channel synchronization through a digital interface module. The speech signal is then preprocessed to obtain the target speech signal. The target speech signal possesses a high signal-to-noise ratio, amplitude normalization, temporal integrity, and a consistent sampling rate, and can be directly input into an acoustic representation encoder for feature modeling.

[0221] The acoustic representation encoder converts the target speech signal into a dense vector representation to obtain speech features. The acoustic representation encoder also possesses robustness in handling speech signals under different noise environments and cross-speaker generalization capabilities.

[0222] Positional encoding is added to speech features through a speech semantic backbone, and the speech features are fused based on a multi-head self-attention mechanism to obtain initial fused features S. The initial fused features preserve the phoneme, prosody, and semantic features of the target speech signal's context.

[0223] The initial fused features are averaged and pooled to generate a semantic query vector q. In a predefined knowledge base, the semantic query vector q is sequentially compared with the dedicated word vectors in the knowledge vector table stored in the predefined knowledge base to perform vector similarity searches, resulting in multiple candidate knowledge items. Multiple candidate knowledge items are used as power industry terms. An interactive attention mechanism is employed to refine the initial fusion feature S based on these power industry terms, thereby fusing the semantic features of the speech signal with power industry information to generate a semantically enhanced fusion feature A. The semantically enhanced fusion feature A and the initial fusion feature S are then weighted and fused to generate the target semantic feature. .

[0224] The target semantic features are used for task type identification and task content identification, respectively. The task type identification model can be a lightweight structure composed of a multilayer perceptron and Softmax, which can perform coarse-grained discrimination at the scene level while retaining computational efficiency. The output probability distribution corresponds to different target task types, such as repair requests, business inquiries, or complaint feedback. The target task type is also used as a priori input to the task content identification module. The task content identification model is based on a hierarchical Transformer structure. While sharing the target semantic features, it combines the task type identification results to further infer the user's fine-grained needs under that category to obtain the task content. For example, checking bills, modifying service plans, or applying for new services.

[0225] Determine if the target task type and target task content match. If they match, send the target task type and target task content.

[0226] The task type recognition model and the task content recognition model form a hierarchical recognition relationship from coarse to fine, and share an acoustic representation encoder during training. By jointly optimizing the multi-task loss function, the accuracy of task content recognition is improved while maintaining the robustness of task type recognition, thereby realizing task-oriented speech recognition.

[0227] When training the task type recognition model and the task content recognition model, the multi-task cross-entropy loss can be used. The multi-task cross-entropy loss can be expressed by formula (23) and formula (24).

[0228] (twenty three)

[0229] (twenty four)

[0230] in, and The weights for task type recognition and task content recognition can be controlled separately; for example, one could take... . It is the total loss used to simultaneously optimize task type recognition and task content recognition during multi-task training. It is the cross-entropy loss corresponding to task type identification. It is the cross-entropy loss corresponding to task content recognition.

[0231] In the speech recognition process, a backoff mechanism is introduced to address situations where there is excessive uncertainty in task type and content recognition, the confidence level is below a set threshold, or unknown categories and new intents are encountered. The backoff mechanism is activated when the target task type and target task content do not match. The backoff process can use a cascaded structure: first, the speech signal is transcribed into text to obtain the target text; then, intent recognition is performed on the target text content; and the target task type and target task content are corrected based on the expected task type and expected task content obtained from the intent recognition. Furthermore, similarity matching can be performed between the target text and historical task types and content, and the target task type and target task content can be corrected based on the matching results.

[0232] Figure 6 A block diagram of a speech recognition device according to an embodiment of this application is shown.

[0233] like Figure 6 As shown, the speech recognition device 600 includes an extraction module 610, a segmentation module 620, a first fusion module 630, a second fusion module 640, and a recognition module 650.

[0234] The extraction module 610 is used to extract a target speech signal from the noise-reduced speech signal by jointly analyzing the energy distribution characteristics and spectral entropy characteristics of the noise-reduced speech signal; wherein the target speech signal is a semantically continuous audio sequence.

[0235] The segmentation module 620 is used to perform frequency domain segmentation on the frequency domain features of the target speech signal to obtain a first frequency feature and a second frequency feature; wherein, the first frequency feature represents the global semantics of the target speech signal; and the second frequency feature represents the keyword semantics of the target speech signal.

[0236] The first fusion module 630 is used to dynamically adjust the weights of the first frequency feature and the second frequency feature, and fuse the first frequency feature and the second frequency feature based on a multi-head self-attention mechanism to generate an initial fused feature; wherein, when the target speech signal contains noise, the weight of the first frequency feature is greater than the weight of the second frequency feature; when the target speech signal contains keywords, the weight of the first frequency feature is less than the weight of the second frequency feature.

[0237] The second fusion module 640 is used to modify the initial fusion features by using power industry terms that match the initial fusion features based on an interactive attention mechanism, thereby generating target semantic features; the target semantic features correct the colloquial expressions in the initial semantic features.

[0238] The recognition module 650 is used to recognize the semantic features of the target and generate the target task type and target task content.

[0239] Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or implemented by hardware or firmware in any other reasonable manner by integrating or packaging circuits, or implemented in any one of software, hardware, and firmware, or in a suitable combination of any of these. Alternatively, one or more of the modules, submodules, units, and subunits according to the embodiments of this application can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0240] For example, any plurality of the extraction module 610, partitioning module 620, first fusion module 630, second fusion module 640, and identification module 650 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this application, at least one of the extraction module 610, partitioning module 620, first fusion module 630, second fusion module 640, and identification module 650 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the extraction module 610, the segmentation module 620, the first fusion module 630, the second fusion module 640, and the identification module 650 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0241] It should be noted that the speech recognition method part in the embodiments of this application corresponds to the data processing method part in the embodiments of this application. The description of the speech recognition device part is specifically referred to in the speech recognition method part, and will not be repeated here.

[0242] Figure 7 A block diagram of an electronic device suitable for implementing the speech recognition method described above, according to an embodiment of this application, is shown. Figure 7 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0243] like Figure 7As shown, an electronic device 700 according to an embodiment of this application includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0244] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0245] According to embodiments of this application, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0246] According to embodiments of this application, the method flow according to embodiments of this application can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by processor 701, it performs the functions defined in the system of embodiments of this application. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0247] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0248] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0249] For example, according to embodiments of this application, a computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0250] Embodiments of this application also include a computer program product, which includes a computer program containing program code for performing the methods provided in the embodiments of this application. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the speech recognition method provided in the embodiments of this application.

[0251] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0252] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0253] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0254] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments of this application can be combined and / or combined in various ways. All such combinations and / or combinations fall within the scope of this application.

[0255] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A speech recognition method, characterized in that, include: In response to receiving a voice signal from a terminal device deployed at a power construction site, a target voice signal is extracted from the denoised voice signal by performing joint analysis of energy distribution characteristics and spectral entropy characteristics; wherein the target voice signal is a semantically continuous audio sequence. Perform Fourier transform on the time-domain features of the target speech signal to generate the frequency-domain features of the target speech signal; The frequency domain features of the target speech signal are processed by calling an activation function to generate a frequency domain mask; the frequency domain mask includes a low-frequency frequency domain mask and a high-frequency frequency domain mask. Frequency domain features of the target speech signal are divided into frequency domains using a frequency domain mask to obtain a first frequency feature and a second frequency feature. The first frequency feature and the second frequency feature are two complementary speech signal frequency features. The first frequency feature represents the low-frequency component in the target speech signal, and the second frequency feature represents the high-frequency component in the target speech signal. The first frequency feature characterizes the global semantics of the target speech signal; the second frequency feature characterizes the keyword semantics of the target speech signal. According to the frequency dimension of the first frequency feature, the first frequency feature is weighted and aggregated by a learnable first weight matrix to obtain the first aggregated feature; wherein, the first aggregated feature focuses on the energy envelope, prosodic contour and time-varying global semantic structure in the target speech signal; According to the frequency dimension of the second frequency feature, the learnable second weight matrix is ​​invoked to perform weighted aggregation on the second frequency feature to obtain the second aggregated feature; wherein, the second aggregated feature focuses on the articulatory units and articulatory transition boundaries in the target speech signal; By calling a probability distribution function to process the first aggregated feature and the second aggregated feature, weights are generated for the first frequency feature and the second frequency feature respectively; wherein, when the target speech signal contains noise, the weight of the first frequency feature is greater than the weight of the second frequency feature; when the target speech signal contains keywords, the weight of the first frequency feature is less than the weight of the second frequency feature. Based on the multi-head self-attention mechanism, the first frequency feature and the second frequency feature are fused according to their respective weights to obtain the initial fused feature; Based on the interactive attention mechanism, the initial fusion features are modified by using power industry terms that match them, thereby generating target semantic features; the target semantic features correct the colloquial expressions in the initial fusion features. The target semantic features are identified to generate the target task type and target task content.

2. The method according to claim 1, characterized in that, The interactive attention mechanism involves modifying the initial fusion features using power industry terms that match them, thereby generating target semantic features. This includes: The initial fusion features and the power industry terms are fused based on an interactive attention mechanism to generate semantically enhanced fusion features; the semantically enhanced fusion features strengthen the industry terminology in the initial fusion features. The semantically enhanced fusion features and the initial fusion features are average-pooled to generate intermediate features; The intermediate features are processed using a multilayer perceptron to generate target weights; and The semantically enhanced fusion feature and the initial fusion feature are weighted and fused based on the target weight to generate the target semantic feature.

3. The method according to claim 1, characterized in that, In response to receiving a voice signal from a terminal device deployed at a power construction site, the system extracts a target voice signal from the denoised voice signal by jointly analyzing its energy distribution characteristics and spectral entropy characteristics, including: In response to receiving a voice signal from a terminal device deployed at a power construction site, a continuous voice signal is extracted from the noise-reduced voice signal; The target segmentation point is obtained by jointly analyzing the energy distribution characteristics and spectral entropy characteristics of the continuous speech signal. The continuous speech signal is segmented based on the target segmentation point to obtain the segmented speech signal; and The segmented speech signal is pre-emphasized and weighted by a symmetrical Hanning window to obtain the target speech signal.

4. The method according to claim 1, characterized in that, The process of identifying the target semantic features and generating the target task type and target task content includes: The target semantic features are used to identify the target task type to obtain the target task type; The target semantic features are used to perform task content recognition to obtain the target task content; In response to determining that the target task type does not match the target task content, speech recognition is performed on the denoised speech signal to generate target text; The target text is subjected to intent recognition to generate a target intent; the target intent includes the expected task type and the expected task content. The target task type is corrected using the expected task type to generate a first corrected task type; The target task content is corrected using the expected task content to generate the first corrected task content.

5. The method according to claim 4, characterized in that, The method further includes: Obtain the historical task types and historical task content that match the target text; The target task type is corrected using the historical task type and the expected task type to generate a second corrected task type; The target task type is corrected using the historical task content and the expected task content to generate a second corrected task content.

6. A voice recognition device, characterized in that, include: An extraction module is used to extract a target speech signal from a terminal device deployed at a power construction site in response to receiving a speech signal. The target speech signal is a semantically continuous audio sequence. The segmentation module performs a Fourier transform on the time-domain features of the target speech signal to generate frequency-domain features. It then processes these frequency-domain features by calling an activation function to generate a frequency-domain mask. The frequency-domain mask includes a low-frequency mask and a high-frequency mask. The frequency-domain features of the target speech signal are then segmented using the frequency-domain mask to obtain a first frequency feature and a second frequency feature. These first and second frequency features are complementary speech signal frequency features. The first frequency feature represents the low-frequency components of the target speech signal, and the second frequency feature represents the high-frequency components. The first frequency feature characterizes the global semantics of the target speech signal; the second frequency feature characterizes the keyword semantics of the target speech signal. The fusion module is used to weight and aggregate the first frequency feature according to its frequency dimension by calling a learnable first weight matrix to obtain a first aggregated feature; wherein the first aggregated feature focuses on the energy envelope, prosodic contour, and time-varying global semantic structure in the target speech signal; and to weight and aggregate the second frequency feature according to its frequency dimension by calling a learnable second weight matrix to obtain a second aggregated feature; wherein the second aggregated feature focuses on the articulatory units and articulation abrupt boundary in the target speech signal; and to process the first aggregated feature and the second aggregated feature by calling a probability distribution function to generate weights for the first frequency feature and the second frequency feature respectively; wherein, when the target speech signal contains noise, the weight of the first frequency feature is greater than the weight of the second frequency feature; and when the target speech signal contains keywords, the weight of the first frequency feature is less than the weight of the second frequency feature. Based on the multi-head self-attention mechanism, the first frequency feature and the second frequency feature are fused according to their respective weights to obtain an initial fused feature; wherein, when the target speech signal contains noise, the weight of the first frequency feature is greater than the weight of the second frequency feature; when the target speech signal contains keywords, the weight of the first frequency feature is less than the weight of the second frequency feature. The first generation module is used to modify the initial fusion features by using power industry terms that match the initial fusion features based on an interactive attention mechanism, thereby generating target semantic features; the target semantic features correct the colloquial expressions in the initial fusion features. The second generation module is used to identify the target semantic features and generate the target task type and target task content.

7. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium having executable instructions stored thereon, characterized in that, When executed by the processor, this instruction causes the processor to implement the method of any one of claims 1 to 5.

Images