Timing sampling method and device, speech recognition method and device

Abstract

This application provides a temporal sampling method and apparatus, and a speech recognition method and apparatus, relating to the field of artificial intelligence technology. The temporal sampling method includes: acquiring target speech data; and determining the temporal sampling result of the target speech data using a sampling model based on the target speech data. The sampling model includes an l-layer downsampling network and an l-layer upsampling network. Each downsampling network layer includes parallel skip modules and downsampling modules, and each upsampling network layer includes parallel skip modules and upsampling modules. The sampling model is used to match the optimal sampling path in the sampling model for the target speech data based on the skip modules and downsampling modules of each downsampling network layer and the skip modules and upsampling modules of each upsampling network layer. l is a positive integer greater than 1. The temporal sampling method in this application can improve the learning ability of acoustic features of speech data and the adaptability of modeling granularity.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a time-series sampling method and apparatus, and a speech recognition method and apparatus. Background Technology

[0002] Relevant speech recognition models typically utilize multilingual acoustic modeling based on shared model parameters. However, the model structure of multilingual acoustic modeling is fixed when dealing with different speech data, which can easily lead to insufficient computation or redundancy. Furthermore, relevant adaptation fine-tuning models learn at the feature level of speech data and cannot capture the long-range correlations between frames at the temporal level of speech data. Summary of the Invention

[0003] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide a timing sampling method and apparatus, and a speech recognition method and apparatus.

[0004] In a first aspect, one embodiment of this application provides a temporal sampling method, comprising: acquiring target speech data; and determining the temporal sampling result of the target speech data using a sampling model based on the target speech data, wherein the sampling model includes a l-layer downsampling network and a l-layer upsampling network, each downsampling network including a parallel skip module and a downsampling module, each upsampling network including a parallel skip module and an upsampling module, and the sampling model is used to match the optimal sampling path in the sampling model for the target speech data based on the skip modules and downsampling modules of each downsampling network and the skip modules and upsampling modules of each upsampling network, where l is a positive integer greater than 1.

[0005] In conjunction with the first aspect, in some implementations of the first aspect, the optimal sampling path includes an optimal downsampling path and an optimal upsampling path. Based on the target speech data, using a sampling model, the temporal sampling result of the target speech data is determined, including: determining the output result of each downsampling network layer in the optimal downsampling path; based on the output result of each downsampling network layer in the optimal downsampling path, determining the output result of each upsampling network layer in the optimal upsampling path corresponding to the downsampling network; and determining the output results of each downsampling network layer in the optimal downsampling path and the output results of each upsampling network layer in the optimal upsampling path as the temporal sampling result of the target speech data.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, based on the output results of each downsampling network in the optimal downsampling path, the output results of each upsampling network in the optimal upsampling path corresponding to the downsampling network are determined, including: for each upsampling network in the optimal upsampling path, determining the initial processing result of the input data of the upsampling network; determining the output result of the downsampling network corresponding to the upsampling network; and using the initial processing result of the input data of the upsampling network and the output result of the downsampling network as the output result of the upsampling network.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the optimal sampling path includes an optimal downsampling path and an optimal upsampling path. Based on the skip and downsampling modules of each downsampling network layer and the skip and upsampling modules of each upsampling network layer, the optimal sampling path in the sampling model is matched for the target speech data, including: for the l-layer downsampling network, matching the optimal downsampling path for the target speech data based on the skip and downsampling modules of each downsampling network layer; and based on the optimal downsampling path, for the l-layer upsampling network layer, matching the optimal upsampling path for the target speech data from the skip and upsampling modules of each upsampling network layer.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the training method of the sampling model includes: obtaining a training corpus; constructing a sampling model to be trained, wherein the network structure of the sampling model to be trained is the same as that of the sampling model; determining a loss function based on the execution probability of the skip module and the execution probability of the downsampling module in each layer of the downsampling network in the sampling model to be trained; and training the sampling model to be trained based on the loss function and the training corpus to obtain the sampling model.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the loss function is determined based on the execution probability of the skip module and the execution probability of the downsampling module in each layer of the downsampling network in the model to be trained. This includes: determining the sampling probability of the downsampling path corresponding to layer l of the downsampling network in the model to be trained based on the execution probabilities of the skip module and the downsampling module; constructing a loss function about the weights and sampling parameters of the downsampling path based on the sampling probability of the downsampling path; and determining the loss function of the model to be trained based on the loss function of the weights and sampling parameters of the downsampling path.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the sampling model to be trained is trained based on the loss function to obtain the sampling model, including: determining the derivative of the loss function with respect to the weights and the derivative of the sampling parameters based on the loss function; and training the sampling model to be trained using the derivative of the loss function with respect to the weights and the derivative of the sampling parameters to obtain the sampling model.

[0011] Secondly, one embodiment of this application provides a speech recognition method, comprising: acquiring speech data to be recognized; performing time-series sampling on the speech data to be recognized using the time-series sampling method as described in the first aspect to obtain a sampling result of the speech data to be recognized; and determining a recognition result of the speech data to be recognized based on the sampling result of the speech data to be recognized.

[0012] Thirdly, one embodiment of this application provides a timing sampling apparatus, comprising: an acquisition module for acquiring target speech data; and a determination module for determining the timing sampling result of the target speech data based on the target speech data and using a sampling model, wherein the sampling model includes a l-layer downsampling network and a l-layer upsampling network, each downsampling network including a parallel skip module and a downsampling module, each upsampling network including a parallel skip module and an upsampling module, and the sampling model is used to match the optimal sampling path in the sampling model for the target speech data based on the skip modules and downsampling modules of each downsampling network and the skip modules and upsampling modules of each upsampling network, where l is a positive integer greater than 1.

[0013] Fourthly, one embodiment of this application provides a speech recognition device, comprising: an acquisition module for acquiring speech data to be recognized; a sampling module for performing time-series sampling on the speech data to be recognized using the time-series sampling method as described in the first aspect, to obtain a sampling result of the speech data to be recognized; and a determination module for determining a recognition result of the speech data to be recognized based on the sampling result of the speech data to be recognized.

[0014] Fifthly, one embodiment of this application provides a computer-readable storage medium storing a computer program for performing the methods described in the first and second aspects.

[0015] In a sixth aspect, one embodiment of this application provides an electronic device, the electronic device comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to perform the methods described in the first and second aspects.

[0016] The timing sampling method provided in this application has the following beneficial effects:

[0017] Each downsampling network layer includes parallel skip and downsampling modules, and each upsampling network layer includes parallel skip and upsampling modules. The sampling model can match the skip or downsampling module in each downsampling network layer to the target speech data based on its features, and also match the corresponding skip or upsampling module in each upsampling network layer. Unlike a single static sampling model structure, this scheme can adjust the temporal sampling amplitude of speech data based on different languages ​​by using any module in the parallel network. Furthermore, this application includes at least one downsampling network layer and at least one upsampling network layer. For each downsampling and upsampling network layer, the skip, downsampling, and upsampling modules can be adaptively selected based on the features of the target speech data. This improves the feature learning ability for multilingual target speech data and helps capture long-range frame correlations in the temporal dimension of the target speech data. Attached Figure Description

[0018] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 The diagram shows an application scenario of the timing sampling method provided in an exemplary embodiment of this application.

[0020] Figure 2 The diagram shown is a flowchart of a timing sampling method provided in an exemplary embodiment of this application.

[0021] Figure 3 The diagram shown is a schematic diagram of the sampling model provided in an exemplary embodiment of this application.

[0022] Figure 4 The diagram shown is a flowchart illustrating the determination of timing sampling results provided in an exemplary embodiment of this application.

[0023] Figure 5 The diagram shown is a flowchart illustrating the matching of the optimal sampling path provided in an exemplary embodiment of this application.

[0024] Figure 6 The diagram shown is a flowchart illustrating a training method for a sampling model provided in an exemplary embodiment of this application.

[0025] Figure 7 The diagram shown is a flowchart of a speech recognition method provided in an exemplary embodiment of this application.

[0026] Figure 8 The diagram shown is a schematic diagram of the timing sampling device provided in an exemplary embodiment of this application.

[0027] Figure 9 The diagram shown is a flowchart of a speech recognition device provided in an exemplary embodiment of this application.

[0028] Figure 10 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] Application Overview

[0031] With the rapid development of the multimedia industry and communication technology, multilingual voice data in human communication is increasing daily, and speech recognition is the core issue of speech analysis and processing. In order to quickly extend the processing capabilities of a few multi-resource languages ​​such as Chinese and English to more low-resource languages, artificial intelligence speech recognition technology has been widely applied.

[0032] Related AI speech recognition technologies can be broadly categorized as follows: multilingual acoustic modeling based on shared model parameters, multilingual acoustic modeling based on language classification information, and end-to-end keyword retrieval techniques based on frame-level alignment. In research on learning low-resource multilingual audio features, further fine-tuning is typically achieved by adding structural elements to a model pre-trained with mixed multilingual corpora. Specific fine-tuning schemes include: adding language adapters between shared hidden layers and specific output layers, adaptively selecting activation functions for training, and introducing self-attention operations in the feature dimension. These schemes all learn the acoustic features of specific languages ​​during the fine-tuning stage.

[0033] The aforementioned AI speech recognition technologies have the following drawbacks. First, end-to-end speech recognition technology uses a single static model structure to train on a corpus of speech data from multiple languages, generating a single model to handle speech recognition in different languages. However, each language has a unique information distribution in the temporal dimension. Without targeted feature sampling modules and relying solely on shared network parameters, it is difficult to optimize for multiple languages ​​simultaneously. Furthermore, the static model structure may include redundant information and lose relevant information, leading to excessively long training times and wasted computational resources. Second, existing adaptive fine-tuning models focus on feature-level learning and fail to capture the long-range correlations between frames in the temporal dimension. Third, besides differences in acoustic features, the corresponding predicted texts also differ between languages. From a modeling granularity perspective, there are phoneme modeling, character modeling, sub-layer modeling, and word modeling. Therefore, simply adapting and fine-tuning the encoder is insufficient to achieve sufficient language adaptability.

[0034] In view of this, this application proposes a temporal sampling method. First, target speech data is acquired. Based on the target speech data, a sampling model is used to determine the temporal sampling result of the target speech data. The sampling model includes an l-layer downsampling network and an l-layer upsampling network. Each downsampling network layer includes parallel skip and downsampling modules, and each upsampling network layer includes parallel skip and upsampling modules. The sampling model is used to match the optimal sampling path in the sampling model for the target speech data based on the skip and downsampling modules of each downsampling network layer and each upsampling network layer. Through the scheme in this application, the input target speech data undergoes temporal downsampling and upsampling operations through a designed dynamic sub-sampling module. During the downsampling and upsampling operations, the method can select whether to skip the current downsampling layer based on the characteristics of the speech data, thereby adaptively learning the temporal sampling amplitude of the target speech data.

[0035] Exemplary scenario

[0036] The timing sampling method proposed in this application can be executed by an electronic device, which can be a terminal, such as a smartphone, tablet computer, desktop computer, etc.; or the electronic device can be a server, such as an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0037] Figure 1 The diagram illustrates an application scenario of the timing sampling method provided in an exemplary embodiment of this application. Figure 1 As shown in the embodiments of this application, the application scenarios provided include a voice data acquisition device 11, a terminal 12, and a server 13.

[0038] For example, the voice data acquisition device 11 can be a voice recorder or microphone, or other devices with voice acquisition capabilities. This application does not limit the specific structure of the voice data acquisition device. Specifically, the terminal 12 obtains target voice data from the voice data acquisition device 11. The terminal 12 can choose to perform time-series sampling on itself, or upload the target voice data to the server 13 and use the server 13 to perform time-series sampling, depending on the complexity and duration of the target voice data.

[0039] In one exemplary scenario, terminal 12 selects server 13 to downsample and upsample the target speech data. Specifically, server 13 is equipped with the sampling model described in this application. The target speech data is input into the sampling model, which matches the optimal sampling path based on the characteristics of the target speech data, ultimately obtaining the temporal sampling result of the target speech data, and then sending the temporal sampling result to terminal 12.

[0040] In another exemplary scenario, a speech recognition model is deployed in terminal 12, which recognizes the target speech data as text based on the received temporal sampling results of the target speech data.

[0041] Exemplary methods

[0042] Figure 2 The diagram shown is a flowchart illustrating a timing sampling method provided in an exemplary embodiment of this application. Figure 2 As shown in the embodiments of this application, the timing sampling method includes the following steps.

[0043] Step S210: Obtain target speech data.

[0044] Specifically, target speech data refers to the speech data that needs to be sampled. Target speech data can be speech data in any language, such as Chinese speech data, English speech data, Spanish speech data, French speech data, and so on.

[0045] Step S220: Based on the target speech data, use the sampling model to determine the temporal sampling results of the target speech data.

[0046] Specifically, the sampling model includes l-layer downsampling network and l-layer upsampling network. Each downsampling network includes parallel skip modules and downsampling modules, and each upsampling network includes parallel skip modules and upsampling modules. The sampling model is used to match the optimal sampling path in the sampling model for the target speech data based on the skip modules and downsampling modules of each downsampling network and the skip modules and upsampling modules of each upsampling network. l is a positive integer greater than 1. Figure 3The diagram shown is a schematic representation of the sampling model provided in an exemplary embodiment of this application. Exemplarily, l = 3, meaning the sampling model includes a 3-layer downsampling network and a 3-layer upsampling network. The skip module also extracts features from the input data, but it retains the enhancement and observation windows preceding the input data.

[0047] For example, the target speech data is input into the sampling model, and the sampling model then outputs the temporal sampling result of the target speech data.

[0048] In this embodiment, each downsampling network layer includes a parallel skip module and a downsampling module, and each upsampling network layer includes a parallel skip module and an upsampling module. The sampling model can match the skip module or downsampling module in each downsampling network layer to the target speech data based on the characteristics of the input target speech data, and also match the corresponding skip module or upsampling module in each upsampling network layer to the target speech data. Unlike a single static sampling model structure, this scheme can adjust the temporal sampling amplitude of the speech data based on the speech data of different languages ​​by using any module in the parallel network. Furthermore, this application includes at least one downsampling network layer and at least one upsampling network layer. For each downsampling network layer and each upsampling network layer, the skip module, downsampling module, and upsampling module can be adaptively selected based on the characteristics of the target speech data. This is beneficial for improving the feature learning ability for multilingual target speech data and for capturing the long-range correlation between frames in the temporal dimension of the target speech data.

[0049] Figure 4 The diagram shown is a schematic flowchart illustrating the determination of timing sampling results according to an exemplary embodiment of this application. Figure 2 Extending from the illustrated embodiment Figure 4 The illustrated embodiment will be described in detail below. Figure 4 The illustrated embodiments and Figure 2 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0050] like Figure 4 As shown in the embodiments of this application, the optimal sampling path includes the optimal downsampling path and the optimal upsampling path. Based on the target speech data, the temporal sampling result of the target speech data is determined using a sampling model, including the following steps.

[0051] Step S410: Determine the output of each downsampling network layer in the optimal downsampling path.

[0052] Continue Figure 3 As described in the embodiment, the sampling model includes a 3-layer downsampling network and a 3-layer upsampling network. In the downsampling network, the downsampling module is represented as b. l,sub The skip module is represented as b l,sikpFor example, the downsampling path is represented by the modules selected in each layer of the downsampling network. If, in the optimal downsampling path, the first layer of the downsampling network selects the downsampling module, the second layer selects the skip module, and the third layer selects the downsampling module, then the optimal downsampling path a = (b 1,sub b 2,sikp b 3,sub ).

[0053] Furthermore, in the optimal downsampling path a = (b 1,sub b 2,sikp b 3,sub In ), determine b 1,sub The output result b 1,sub (x1), b 2,skip The output result b 2,sikp (x2), b 3,sub The output result b 3,sub (x3).

[0054] Step S420: Based on the output results of each downsampling network in the optimal downsampling path, determine the output results of each upsampling network in the optimal upsampling path corresponding to the downsampling network.

[0055] Specifically, for each layer of the upsampling network in the optimal upsampling path, the initial processing result of the input data of the upsampling network is determined; the output result of the downsampling network corresponding to the upsampling network is determined; and the initial processing result of the input data of the upsampling network and the output result of the downsampling network are used as the output result of the upsampling network.

[0056] Specifically, following the example in step S410, the optimal downsampling path a = (b 1,sub b 2,sikp b 3,sub If ), then the optimal upsampling path b = (b 1,up b 2,sikp b 3,up That is, if the first-layer downsampling network selects the downsampling module, then the first-layer upsampling network selects the upsampling module; if the second-layer downsampling network selects the skip module, then the second-layer upsampling network selects the skip module; if the third-layer downsampling network selects the downsampling module, then the third-layer upsampling network selects the upsampling module.

[0057] Furthermore, in upsampling networks, b 1,up The output result is b 1,up (y1)+b 1,sub (x1); b 2,sikp The output result is b 2,skip (y2)+b 2,skip (x2); b3,up The output result is b 3,sub (x3)+b 3,up (y3).

[0058] Step S430: The output results of each downsampling network in the optimal downsampling path and the output results of each upsampling network in the optimal upsampling path are determined as the temporal sampling results of the target speech data.

[0059] In this embodiment, the output of the downsampling network and the initial processing result of the input data of the upsampling network at the same layer are used as the output of the upsampling network at that layer. This involves fusing the upsampling and downsampling results of the corresponding layer, or fusing the output of the skip module of the corresponding layer. By superimposing and fusing acoustic representations of the target speech data at different scales, information at different time scales of the corresponding speech data can be obtained, thereby improving the recognition results of the speech data.

[0060] Figure 5 The diagram shown is a flowchart illustrating the matching of the optimal sampling path according to an exemplary embodiment of this application. Figure 2 Extending from the illustrated embodiment Figure 5 The illustrated embodiment will be described in detail below. Figure 5 The illustrated embodiments and Figure 2 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0061] like Figure 5 As shown in the embodiments of this application, the optimal sampling path includes the optimal downsampling path and the optimal upsampling path. Based on the skip module and downsampling module of each downsampling network layer and the skip module and upsampling module of each upsampling network layer, the optimal sampling path in the sampling model is matched for the target speech data, including the following steps.

[0062] Step S510: For the l-layer downsampling network, based on the skip module and downsampling module of each layer downsampling network, match the optimal downsampling path for the target speech data.

[0063] For example, according to Figure 4 The illustrated embodiment matches the optimal downsampling path for the target speech data based on the features of the target speech data input to the sampling model. For example, the optimal downsampling path a = (b 1,sub b 2,sikp b 3,sub ).

[0064] Step S520: Based on the optimal downsampling path, for the l-layer upsampling network, match the optimal upsampling path for the target speech data from the skip module and upsampling module of each layer upsampling network.

[0065] In this embodiment, the selection of the upsampling path corresponds to the selection of the downsampling path. That is, if the first-layer downsampling network selects a downsampling module, then the first-layer upsampling network selects an upsampling module; if the second-layer downsampling network selects a skip module, then the second-layer upsampling network selects a skip module; if the third-layer downsampling network selects a downsampling module, then the third-layer upsampling network selects an upsampling module. In other words, if the optimal downsampling path a = (b... 1,sub b 2,sikp b 3,sub If ), then the optimal upsampling path b = (b 1,up b 2,sikp b 3,up ).

[0066] In this embodiment of the application, by determining the optimal downsampling path, the optimal upsampling path is determined. On the one hand, this simplifies the complexity of selecting the sampling path of the sampling model. On the other hand, based on the correspondence between the downsampling network and the upsampling network, the feature correlation between the target speech data in the same layer downsampling network and the upsampling network can be captured, so as to better perform temporal sampling of the target speech data.

[0067] Figure 6 The diagram shown is a flowchart illustrating a training method for a sampling model provided in an exemplary embodiment of this application. Figure 6 As shown, the training method for the sampling model provided in this application includes the following steps.

[0068] Step S610: Obtain the training corpus.

[0069] The training corpus can include data from multiple languages, such as German, Chinese, and Italian. Furthermore, to ensure the training effectiveness of the sampling model, the corpus for each language can be richer in terms of duration, content, and intonation. For example, for Chinese corpora, speech data from standard Mandarin and regional dialects can be obtained to improve the robustness of the sampling model. Additionally, if the amount of multilingual corpus data is limited, augmentation processing can be performed on the corpus to obtain corpora of the same languages ​​used to train the sampling model.

[0070] Step S620: Construct the sampling model to be trained.

[0071] The network structure of the sampling model to be trained is the same as that of the sampling model. Specifically, the network structure of the sampling model to be trained also includes l-layer downsampling network and l-layer upsampling network. Each downsampling network layer includes parallel skip modules and downsampling modules, and each upsampling network layer includes parallel skip modules and upsampling modules. The sampling model is used to match the optimal sampling path in the sampling model for the target speech data based on the skip modules and downsampling modules of each downsampling network layer and the skip modules and upsampling modules of each upsampling network layer. For example, when l=3, the network structure of the sampling model to be trained can be seen in [reference needed]. Figure 3 As shown.

[0072] Step S630: Determine the loss function based on the execution probability of the skip module and the execution probability of the downsampling module in each layer of the downsampling network in the model to be trained.

[0073] Specifically, step S630 includes: determining the sampling probability of the downsampling path corresponding to the l-layer downsampling network in the sampling model to be trained based on the execution probability of the skip module and the execution probability of the downsampling module; constructing a loss function for the weights and sampling parameters of the downsampling path based on the sampling probability of the downsampling path; and determining the loss function of the sampling model to be trained based on the loss function for the weights and sampling parameters of the downsampling path.

[0074] For example, in the parallel modules of each downsampling network, the probability of selecting and executing the downsampling module can be expressed as:

[0075]

[0076] Where, θ l,sub This indicates that layer l determines the parameters of the downsampling module to be used, θ l,skip This indicates the parameters θ that determine whether layer l uses the skip module. l It contains parameters that determine the sampling probability of each parallel module in layer l.

[0077] Furthermore, given that the number of layers in the search space is n, the sampling probability of the sampling model structure α can be expressed as:

[0078]

[0079] The vector θ represents the θ values ​​of all downsampling modules in the n-layer search space. l,sub Vectors and skip module θ l,skip Combination of vectors; m l It is a random parameter in {0, 1}. If the evaluation is 1, the downsampling module b l,sub Selected; if the evaluation is 0, skip module b l,skip Selected.

[0080] If the downsampling path corresponds to the downsampling model structure α, then the sampling probability of the downsampling path can also be expressed as:

[0081]

[0082] Based on the sampling probability P of the downsampling path θ (α), the constructed loss function can be expressed as:

[0083] L(α)=CE(α, w α θ l}

[0084] Where CE represents the Cross-Entropy (CE) loss function, w α This represents the sampling model weights.

[0085] In this embodiment, based on the network structure of the sampling model to be trained, the sampling probability of the downsampling path is first established, and then a loss function for the weights and sampling parameters of the downsampling path is constructed based on the sampling probability of the downsampling path. Through the scheme in this embodiment, the corresponding weights and sampling parameters can be adjusted for the same downsampling path during each training process of the sampling model to be trained, thereby improving the training accuracy of the sampling model to be trained.

[0086] Step S640: Based on the loss function and the training corpus, train the sampling model to be trained to obtain the sampling model.

[0087] Specifically, step S640 includes: determining the derivative of the loss function with respect to the weights and the derivative of the sampling parameters based on the loss function; and training the sampling model to be trained using the derivative of the loss function with respect to the weights and the derivative of the sampling parameters to obtain the sampling model.

[0088] Specifically, m l It is a random parameter in {0, 1}. If the evaluation is 1, the downsampling module b l,sub Selected, b l,sub (x l ) indicates that the l-th layer downsampling network operates on input x. l The output is below. If m l If the evaluation result is 0, then the data x is directly passed through the skip module without going through the downsampling module in the l-th layer downsampling network. l This application allows each layer of the downsampling network to independently select random parameters m. l That is, m l The final value of 0 or 1 is obtained through learning. Since it is a discrete variable, it cannot be directly learned using gradient backpropagation available for continuous variables. Therefore, this application rewrites the discrete optimization problem as follows:

[0089]

[0090] Obviously, based on the loss function constructed in step S630 above, relative to the model weights w α It is differentiable and can be optimized using stochastic gradient descent. However, this application cannot apply the gradient through a discrete random variable m. l Passed to θ l Therefore, the loss value cannot be directly related to the sampling parameter θ. l Differentiation; to avoid this situation, this case will use discrete mask variable m l It is transformed into a continuous random variable calculated by the Gumbel Softmax function.

[0091]

[0092] Where g l,sub ~Gumbel(0,1) represents random noise following a Gumbel distribution, where τ is a temperature parameter. When τ is close to 0, the function approximates discrete classification sampling. As τ increases, m... l It becomes a continuous random variable.

[0093] During training, the sampling model structure is dynamically and randomly adjusted. (Calculation) Update the parameters of each parallel layer based on the calculation results; calculate Update the sampling probability P based on the calculation results. θ After the sampling model structure converges, the sampling distribution P is used to... θ The optimal sampling path is obtained.

[0094] In this embodiment, by differentiating the weights and sampling parameters of the loss function and further optimizing the sampling model to be trained based on the differentiation results, the accuracy of training the sampling model to be trained is guaranteed.

[0095] Figure 7 The diagram shown is a flowchart illustrating a speech recognition method provided in an exemplary embodiment of this application. Figure 7 As shown in the embodiments of this application, the speech recognition method includes the following steps.

[0096] Step S710: Obtain the speech data to be recognized.

[0097] Specifically, the speech data to be recognized can be speech data in any language, and there are no restrictions on the duration and content of the speech data to be recognized.

[0098] Step S720: Perform time-series sampling on the speech data to be recognized to obtain the sampling result of the speech data to be recognized.

[0099] Specifically, the temporal sampling method described in any of the foregoing embodiments is used to perform temporal sampling on the speech data to be recognized, thereby obtaining the sampling result of the speech data to be recognized.

[0100] Step S730: Based on the sampling results of the speech data to be recognized, determine the recognition result of the speech data to be recognized.

[0101] For example, in the sampling model of the above embodiment, a decoder, a language adapter, and an activation function are added, etc., so as to determine the text recognition result of the speech data to be recognized based on the sampling result of the speech data to be recognized.

[0102] The solution in this application embodiment can obtain more accurate sampling results of the speech data to be recognized. The sampling results can include the correlation between long-distance frames at the temporal level of the speech data to be recognized, which further improves the recognition results of the speech data to be recognized.

[0103] Exemplary device

[0104] The above text combined Figures 2 to 7 The method embodiments of this application are described in detail below, in conjunction with... Figure 8 and Figure 9 The present application provides a detailed description of the apparatus embodiments. It should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be found in the foregoing method embodiments.

[0105] Figure 8 The diagram shown is a schematic representation of the timing sampling apparatus provided in an exemplary embodiment of this application. Figure 8 As shown, the timing sampling device 80 provided in this embodiment includes:

[0106] The acquisition module 810 is used to acquire target speech data;

[0107] The determination module 820 is used to determine the temporal sampling results of the target speech data based on the target speech data and using a sampling model. The sampling model includes an l-layer downsampling network and an l-layer upsampling network. Each downsampling network includes a parallel skip module and a downsampling module, and each upsampling network includes a parallel skip module and an upsampling module. The sampling model is used to match the optimal sampling path in the sampling model for the target speech data based on the skip modules and downsampling modules of each downsampling network and the skip modules and upsampling modules of each upsampling network. l is a positive integer greater than 1.

[0108] In one embodiment of this application, the determining module 820 is further configured to: determine the output result of each layer of the downsampling network in the optimal downsampling path; based on the output result of each layer of the downsampling network in the optimal downsampling path, determine the output result of each layer of the upsampling network in the optimal upsampling path corresponding to the downsampling network; and determine the output result of each layer of the downsampling network in the optimal downsampling path and the output result of each layer of the upsampling network in the optimal upsampling path as the temporal sampling result of the target speech data.

[0109] In one embodiment of this application, the determining module 820 is further configured to: determine the initial processing result of the input data of the upsampling network for each layer of the upsampling network in the optimal upsampling path; determine the output result of the downsampling network corresponding to the upsampling network; and use the initial processing result of the input data of the upsampling network and the output result of the downsampling network as the output result of the upsampling network.

[0110] In one embodiment of this application, the determining module 820 is further configured to: for a layer l downsampling network, match an optimal downsampling path for the target speech data based on the skip module and downsampling module of each layer downsampling network; and based on the optimal downsampling path, match an optimal upsampling path for the target speech data from the skip module and upsampling module of each layer upsampling network.

[0111] In one embodiment of this application, the determining module 820 is further configured to: acquire a training corpus set; construct a sampling model to be trained, wherein the network structure of the sampling model to be trained is the same as that of the sampling model; determine a loss function based on the execution probability of the skip module and the execution probability of the downsampling module in each layer of the downsampling network in the sampling model to be trained; and train the sampling model to be trained based on the loss function and the training corpus set to obtain the sampling model.

[0112] In one embodiment of this application, the determining module 820 is further configured to: determine the sampling probability of the downsampling path corresponding to the l-layer downsampling network in the sampling model to be trained based on the execution probability of the skip module and the execution probability of the downsampling module; construct a loss function for the weights and sampling parameters of the downsampling path based on the sampling probability of the downsampling path; and determine the loss function of the sampling model to be trained based on the loss function for the weights and sampling parameters of the downsampling path.

[0113] In one embodiment of this application, the determining module 820 is further configured to: determine the derivative of the loss function with respect to the weights and the derivative of the sampling parameters based on the loss function; and train the sampling model to be trained using the derivative of the loss function with respect to the weights and the derivative of the sampling parameters to obtain the sampling model.

[0114] Figure 9The diagram shown is a schematic flowchart of a speech recognition device provided in an exemplary embodiment of this application. Figure 9 As shown, the speech recognition device 90 provided in this application embodiment includes:

[0115] The acquisition module 910 is used to acquire the speech data to be recognized;

[0116] The sampling module 920 is used to perform time-series sampling on the speech data to be recognized using the time-series sampling method as described in the first aspect, and obtain the sampling result of the speech data to be recognized.

[0117] The determination module 930 is used to determine the recognition result of the speech data to be recognized based on the sampling results of the speech data to be recognized.

[0118] Below, for reference Figure 10 This describes an electronic device according to embodiments of the present application. Figure 10 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application.

[0119] like Figure 10 As shown, the electronic device 100 includes one or more processors 1001 and memory 1002.

[0120] The processor 1001 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0121] The memory 1002 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1001 may execute the program instructions to implement the methods of the various embodiments of this application described above and / or other desired functions. The computer-readable storage medium may also store various contents such as target speech data, sampling models, output results of each downsampling network layer, and output results of each upsampling network layer.

[0122] In one example, the electronic device 100 may also include an input device 1003 and an output device 1004, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0123] The input device 1003 may include, for example, a keyboard, a mouse, etc.

[0124] The output device 1004 can output various information to the outside, including target speech data, sampling models, output results of each downsampling network layer, and output results of each upsampling network layer. The output device 1004 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0125] Of course, for the sake of simplicity, Figure 10 Only some of the components of the electronic device 100 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 100 may include any other suitable components depending on the specific application.

[0126] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods according to the various embodiments of this application described above.

[0127] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0128] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the methods described above according to various embodiments of this application.

[0129] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0130] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0131] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0132] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0133] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0134] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A time-series sampling method, characterized in that, include: Acquire target speech data; Based on the target speech data, a sampling model is used to determine the temporal sampling result of the target speech data. The sampling model includes a l-layer downsampling network and a l-layer upsampling network. Each layer of the downsampling network includes a parallel skip module and a downsampling module, and each layer of the upsampling network includes a parallel skip module and an upsampling module. The sampling model is used to match the optimal sampling path in the sampling model for the target speech data based on the skip modules and downsampling modules of each layer of the downsampling network and the skip modules and upsampling modules of each layer of the upsampling network. l is a positive integer greater than 1.

2. The method according to claim 1, characterized in that, The optimal sampling path includes an optimal downsampling path and an optimal upsampling path. The step of determining the temporal sampling result of the target speech data using a sampling model, based on the target speech data, includes: Determine the output of each layer of the downsampling network in the optimal downsampling path; Based on the output results of each downsampling network in the optimal downsampling path, determine the output results of each upsampling network in the optimal upsampling path corresponding to the downsampling network; The output results of each downsampling network in the optimal downsampling path and the output results of each upsampling network in the optimal upsampling path are determined as the temporal sampling results of the target speech data.

3. The method according to claim 2, characterized in that, The step of determining the output of each layer of the upsampling network in the optimal upsampling path, based on the output of each layer of the downsampling network in the optimal downsampling path, includes: For each layer of the upsampling network in the optimal upsampling path, determine the initial processing result of the input data of the upsampling network; Determine the output result of the downsampling network corresponding to the upsampling network; The initial processing result of the input data of the upsampling network and the output result of the downsampling network are used as the output result of the upsampling network.

4. The method according to claim 1, characterized in that, The optimal sampling path includes an optimal downsampling path and an optimal upsampling path. The process of matching the optimal sampling path in the sampling model for the target speech data, based on the skip and downsampling modules of each layer of the downsampling network and the skip and upsampling modules of each layer of the upsampling network, includes: For the l-layer downsampling network, based on the skip module and downsampling module of each layer of the downsampling network, the optimal downsampling path is matched for the target speech data; Based on the optimal downsampling path, for the l-layer upsampling network, the optimal upsampling path is matched for the target speech data from the skip module and upsampling module of each layer of the upsampling network.

5. The method according to claim 1, characterized in that, The training method for the sampling model includes: Obtain the training corpus; Construct a sampling model to be trained, wherein the sampling model to be trained has the same network structure as the sampling model; The loss function is determined based on the execution probability of the skip module and the execution probability of the downsampling module in each layer of the downsampling network in the sampling model to be trained. Based on the loss function and the training corpus, the sampling model to be trained is trained to obtain the sampling model.

6. The method according to claim 5, characterized in that, The loss function is determined based on the execution probability of the skip module and the execution probability of the downsampling module in each layer of the downsampling network in the model to be trained, including: Based on the execution probability of the skip module and the execution probability of the downsampling module, the sampling probability of the downsampling path corresponding to the l-layer downsampling network in the sampling model to be trained is determined. Based on the sampling probability of the downsampling path, a loss function is constructed regarding the weights and sampling parameters of the downsampling path; The loss function of the sampling model to be trained is determined based on the weights of the downsampling path and the loss function of the sampling parameters.

7. The method according to claim 6, characterized in that, The step of training the sampling model to be trained based on the loss function to obtain the sampling model includes: Based on the loss function, determine the derivative of the loss function with respect to the weights and the derivative of the sampling parameters; The sampling model is trained using the derivatives of the weights and the derivatives of the sampling parameters to obtain the sampling model.

8. A speech recognition method, characterized in that, include: Acquire the speech data to be recognized; Using the temporal sampling method as described in any one of claims 1 to 7, the speech data to be recognized is temporally sampled to obtain the sampling result of the speech data to be recognized. Based on the sampling results of the speech data to be recognized, the recognition result of the speech data to be recognized is determined.

9. A timing sampling device, characterized in that, include: The acquisition module is used to acquire target speech data; A determination module is used to determine the temporal sampling result of the target speech data based on the target speech data and using a sampling model. The sampling model includes a l-layer downsampling network and a l-layer upsampling network. Each layer of the downsampling network includes a parallel skip module and a downsampling module, and each layer of the upsampling network includes a parallel skip module and an upsampling module. The sampling model is used to match the optimal sampling path in the sampling model for the target speech data based on the skip modules and downsampling modules of each layer of the downsampling network and the skip modules and upsampling modules of each layer of the upsampling network. l is a positive integer greater than 1.

10. A voice recognition device, characterized in that, include: The acquisition module is used to acquire the speech data to be recognized; A sampling module is used to perform time-series sampling on the speech data to be recognized using the time-series sampling method as described in any one of claims 1 to 7, and obtain the sampling result of the speech data to be recognized. The determination module is used to determine the recognition result of the speech data to be recognized based on the sampling result of the speech data to be recognized.

11. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for performing the method described in any one of claims 1 to 8.

12. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to perform the method described in any one of claims 1 to 8.

Images