Audio data processing method, device, equipment, medium and program product

Abstract

The embodiment of the invention provides an audio data processing method and device, equipment, a medium and a program product, which are used for accurately waking up a voice assistant. The method comprises the steps of determining first state information in response to received first audio data, and inputting the first state information to a to-be-trained DQN; in the to-be-trained DQN, the first exploration rate is updated based on the target rule, and a first type of action corresponding to the first state information is determined based on a second exploration rate obtained by updating the first exploration rate; wherein the target rule comprises the steps that if the average reward value of M continuous iteration steps is larger than a preset first threshold value, a preset exploration rate is used for updating the first exploration rate, the preset exploration rate is larger than the first exploration rate, and M is a preset experience value; determining a target DQN based on the first type of action and a loss function; and in response to the received second audio data, inputting second state information into the target DQN to obtain a second type of action.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an audio data processing method, apparatus, device, medium and program product. Background Art

[0002] Voice wake-up methods are widely recognized for their convenience, freeing users' hands. For example, when applied to the vehicle, users can activate or turn on the car's air conditioning through voice wake-up while focusing on the road ahead. However, voice wake-up technology is typically implemented with the help of deep learning models, and deep learning models used for voice wake-up generally rely on extremely large amounts of labeled data for training through supervised learning. However, actual application scenarios are complex and varied, and labeled data cannot actually cover all possible situations. As a result, the generalization ability is insufficient, and false wake-ups are prone to occur. Summary of the Invention

[0003] Based on this, it is necessary to provide an audio data processing method, device, equipment, medium and program product to address the above technical problems, so as to accurately wake up the voice assistant.

[0004] In a first aspect, an embodiment of the present application provides an audio data processing method, comprising:

[0005] In response to receiving the first audio data, determining first state information, and inputting the first state information into a DQN to be trained; wherein the DQN to be trained includes a loss function;

[0006] In the DQN to be trained, a first exploration rate is updated based on a target rule, and a first type of action corresponding to the first state information is determined based on a second exploration rate obtained by updating the first exploration rate; wherein the target rule includes updating the first exploration rate using a preset exploration rate if an average reward value of M consecutive iterations is greater than a preset first threshold, the preset exploration rate being greater than the first exploration rate, and M being a preset experience value;

[0007] Determining a target DQN based on the first type of action and the loss function;

[0008] In response to receiving the second audio data, second state information is input into the target DQN to obtain a second type of action; wherein the second state information corresponds to the second audio data.

[0009] In some embodiments, in the DQN to be trained, updating the first exploration rate based on the target rule, and determining the first type of action corresponding to the first state information based on the second exploration rate obtained by updating the first exploration rate, includes:

[0010] Processing the first state information by a to-be-trained Q network in the to-be-trained DQN to obtain a plurality of to-be-processed Q values;

[0011] Determining a target behavior strategy based on the second exploration rate; wherein the target behavior strategy is an exploration strategy or a greedy strategy;

[0012] Based on the target behavior strategy, a target Q value is selected from the multiple Q values ​​to be processed, and the first type of action corresponding to the target Q value is obtained.

[0013] In some embodiments, updating the first exploration rate based on the target rule includes:

[0014] Determine an iteration step sequence; wherein the iteration step sequence includes the iteration steps in order of iteration step sequence numbers and first reward values ​​corresponding to the iteration steps;

[0015] Processing the iteration step number and / or the first reward value through the target rule to determine an update strategy;

[0016] Based on the update strategy, the first exploration rate is updated.

[0017] In some embodiments, processing the iteration step number and / or the first reward value through the target rule to determine the update strategy includes:

[0018] In response to a maximum iteration step number in the iteration step sequence being less than a preset second threshold, determining that the update strategy is a first update strategy; wherein the first update strategy indicates that the first exploration rate decays as the iteration step number increases; or,

[0019] In response to the average reward value of M consecutive iteration steps in the iteration step sequence being greater than the first threshold, determining that the update strategy is a second update strategy; wherein the second update strategy indicates updating the first exploration rate based on the preset exploration rate, and the preset exploration rate is greater than 0.5 and less than 1; or,

[0020] In response to the fact that the number of positive first reward values ​​in K consecutive iteration steps in the iteration step sequence is greater than a preset third threshold, the update strategy is determined to be a third update strategy; wherein the third update strategy indicates that the first exploration rate decays with an increase in the iteration step number, and K is a preset experience value.

[0021] In some embodiments, updating the first exploration rate based on the update strategy includes:

[0022] In response to the update strategy being the first update strategy, the first exploration rate is processed using a first attenuation formula to obtain the second exploration rate; wherein the first attenuation formula is: ε' is the second exploration rate, ε is the first exploration rate, u1 is a preset first proportional factor, λ1 is a preset first exploration constant, and n is the iteration step number.

[0023] In some embodiments, a decay rate of the first exploration rate in the third update strategy is greater than a decay rate of the first exploration rate in the first update strategy.

[0024] In some embodiments, the average reward value is the arithmetic mean of the first reward values ​​corresponding to the first type of action in the M consecutive iteration steps;

[0025] The first reward value is obtained by the following method:

[0026] Determining second voice data based on a preset time window and a cutoff time of first voice data in the first audio data; wherein a start time of the second voice data is later than the cutoff time of the first voice data;

[0027] Converting the first voice data and the second voice data into first text data;

[0028] Performing natural language understanding processing on the first text data using a pre-trained language model to obtain a third type of action;

[0029] The first type of action is verified based on the third type of action to generate the first reward value.

[0030] In some embodiments, determining the first state information includes:

[0031] Acquiring historical status information; wherein the historical status information includes the cumulative number of positive first reward values ​​corresponding to the iteration steps within a preset time window;

[0032] determining first voice feature information of first voice data in the first audio data;

[0033] The first state information is generated based on the historical state information and the first voice feature information.

[0034] In some embodiments, determining the first voice feature information of the first voice data in the first audio data includes:

[0035] Performing denoising processing on the first audio data based on a preset reference noise signal to obtain first speech data;

[0036] Extracting Mel-spectrogram cepstral coefficients from the first speech data to obtain a first feature matrix;

[0037] performing perceptual linear prediction processing on the first speech data to obtain a second feature matrix;

[0038] Concatenating the first characteristic matrix and the second characteristic matrix to obtain a third characteristic matrix;

[0039] The third feature matrix is ​​processed through a preset attention mechanism to obtain the first speech feature information.

[0040] In a second aspect, an embodiment of the present application provides an audio data processing device, comprising:

[0041] A first input module, configured to determine first state information in response to receiving first audio data, and input the first state information to a DQN to be trained; wherein the DQN to be trained includes a loss function;

[0042] an action module, configured to update, in the DQN to be trained, a first exploration rate based on a target rule, and determine, based on a second exploration rate obtained by updating the first exploration rate, a first type of action corresponding to the first state information; wherein the target rule includes updating the first exploration rate using a preset exploration rate if an average reward value for M consecutive iterations is greater than a preset first threshold, the preset exploration rate being greater than the first exploration rate, and M being a preset experience value;

[0043] a loss module, configured to determine a target DQN based on the first type of action and the loss function;

[0044] The second input module is configured to input second state information into the target DQN in response to receiving the second audio data, so as to obtain a second type of action; wherein the second state information corresponds to the second audio data.

[0045] In some embodiments, the action module is specifically used to process the first state information through the Q network to be trained in the DQN to be trained to obtain multiple Q values ​​to be processed; determine the target behavior strategy based on the second exploration rate; wherein the target behavior strategy is an exploration strategy or a greedy strategy; based on the target behavior strategy, select the target Q value from the multiple Q values ​​to be processed to obtain the first type of action corresponding to the target Q value.

[0046] In some embodiments, the action module is further used to determine an iteration step sequence; wherein the iteration step sequence includes the iteration steps ordered by iteration step numbers, and the first reward values ​​corresponding to the iteration steps; the iteration step numbers and / or the first reward values ​​are processed by the target rule to determine an update strategy; and the first exploration rate is updated based on the update strategy.

[0047] In some embodiments, the action module is specifically used to determine that the update strategy is a first update strategy in response to the maximum iteration step number in the iteration step sequence being less than a preset second threshold; wherein the first update strategy indicates that the first exploration rate decays with an increase in the iteration step number; or, in response to the average reward value of M consecutive iteration steps in the iteration step sequence being greater than the first threshold, determine that the update strategy is a second update strategy; wherein the second update strategy indicates that the first exploration rate is updated based on the preset exploration rate, and the preset exploration rate is greater than 0.5 and less than 1; or, in response to the number of positive first reward values ​​of K consecutive iteration steps in the iteration step sequence being greater than a preset third threshold, determine that the update strategy is a third update strategy; wherein the third update strategy indicates that the first exploration rate decays with an increase in the iteration step number, and K is a preset experience value.

[0048] In some embodiments, the action module is specifically configured to, in response to the update strategy being the first update strategy, process the first exploration rate using a first attenuation formula to obtain the second exploration rate; wherein the first attenuation formula is: ε' is the second exploration rate, ε is the first exploration rate, u1 is a preset first proportional factor, λ1 is a preset first exploration constant, and n is the iteration step number.

[0049] In some embodiments, a decay rate of the first exploration rate in the third update strategy is greater than a decay rate of the first exploration rate in the first update strategy.

[0050] In some embodiments, the average reward value is the arithmetic mean of the first reward values ​​corresponding to the first type of action in the M consecutive iteration steps; the action module is also used to determine the second voice data based on a preset time window and the deadline of the first voice data in the first audio data; wherein the start time of the second voice data is later than the deadline of the first voice data; the first voice data and the second voice data are converted into first text data; the first text data is subjected to natural language understanding processing through a pre-trained language model to obtain a third type of action; the first type of action is verified based on the third type of action to generate the first reward value.

[0051] In some embodiments, the first input module is specifically used to obtain historical status information; wherein the historical status information includes the cumulative number of positive first reward values ​​corresponding to the iteration steps within a preset time window; determine the first voice feature information of the first voice data in the first audio data; and generate the first status information based on the historical status information and the first voice feature information.

[0052] In some embodiments, the first input module is specifically used to perform denoising processing on the first audio data based on a preset reference noise signal to obtain first speech data; extract Mel-spectrogram cepstral coefficients in the first speech data to obtain a first feature matrix; perform perceptual linear prediction processing on the first speech data to obtain a second feature matrix; concatenate the first feature matrix and the second feature matrix to obtain a third feature matrix; process the third feature matrix through a preset attention mechanism to obtain the first speech feature information.

[0053] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and for running on the processor, wherein when the processor executes the computer program, the method described in the first aspect and any embodiment is implemented.

[0054] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any one of the first aspect and any one of the embodiments.

[0055] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, it implements the method as described in any one of the first aspect and any embodiment.

[0056] In the audio data processing method provided by the embodiment of the present application, during the training of the DQN to be trained, the first exploration rate is updated by the target rule to determine the first action type. In the target rule for updating the first exploration rate, the average reward value of M consecutive iterative steps is used to judge whether the DQN to be trained is in a stage where it is easy to fall into the local optimum, and the first exploration rate is updated in time by the preset exploration rate, so as to prompt the DQN to be trained to increase the search range, thereby timely and effectively avoiding the DQN to be trained from falling into the local optimum during the training process, effectively improving the convergence speed of the loss function, and improving the accuracy of the output of the target DQN obtained by training. Thus, when the target DQN is used to process audio data, accurate recognition of user intentions can be achieved. In particular, when the target DQN is used to wake up a voice assistant, the target DQN can exhibit good performance and can effectively alleviate the problem of erroneous wake-up of the voice assistant.

[0057] Other features and advantages of the present application will be described in the following description and, in part, will become apparent from the description or may be learned through practice of the present application. The objectives and other advantages of the present application may be achieved and obtained through the structures particularly pointed out in the written description, claims, and drawings. It should be understood that the above general description and the detailed description that follow are merely exemplary and explanatory and do not limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0059] Figure 1 A flowchart of an audio data processing method provided in an embodiment of the present application;

[0060] Figure 2 A schematic diagram of the structure of the DQN to be trained provided in an embodiment of the present application;

[0061] Figure 3 A structural block diagram of an audio data processing device provided in an embodiment of the present application;

[0062] Figure 4 This is a diagram of the internal structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0064] It should be noted that the diagrams provided in the present embodiment are only schematic illustrations of the basic concept of the present application. The diagrams only show the components related to the present application rather than the number, shape and size of the components when actually implemented. The type, quantity and ratio of each component can be changed at will during actual implementation, and the component layout pattern may also be more complicated. The structures, ratios, sizes, etc. illustrated in the drawings of this specification are only used to match the content disclosed in the specification for people familiar with this technology to understand and read. They are not used to limit the restrictive conditions that can be implemented in this application. Therefore, they have no technical significance. Any modification of the structure, change of the proportional relationship or adjustment of the size should still fall within the scope of the technical content disclosed in this application without affecting the effect and purpose that can be achieved by this application. At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" quoted in this specification are only for the convenience of description and are not used to limit the scope of the implementation of this application. The change or adjustment of their relative relationship should also be considered as the scope of the implementation of this application without substantial change in the technical content.

[0065] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various places herein does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0066] As used herein, unless the context clearly indicates otherwise, the terms "a," "an," "an," and / or "the" are not intended to refer to the singular but may include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements specifically identified, and these steps and elements do not constitute an exclusive list. A method or apparatus may also include additional steps or elements.

[0067] The definition of inclusion herein, such as the terms “having”, “may have”, “include” or “may include” as used herein, indicates the existence of the corresponding functions, operations, elements, etc. herein, and does not limit the existence of one or more other functions, operations, elements, etc. In addition, it should be understood that the terms “including” or “having” as used herein indicate the existence of the features, numbers, steps, operations, elements, components or their combination described in the specification, and do not exclude the existence or addition of one or more other features, numbers, steps, operations, elements, components or their combination.

[0068] In the embodiments of the present application, prefixes such as "first" and "second" are used only to distinguish different description objects and have no limiting effect on the position, order, priority, quantity or content of the described objects. In the embodiments of the present application, the use of prefixes such as ordinal numbers to distinguish description objects does not constitute a restriction on the described objects. For the statement of the described + object, please refer to the description in the context of the claims or embodiments, and no unnecessary restrictions should be constituted due to the use of such prefixes. In addition, in the description of this embodiment, unless otherwise specified, the meaning of "plurality" is two or more.

[0069] In order to improve the accuracy of the voice assistant's response action and enhance the user experience, this application embodiment provides an audio data processing method, please refer to Figure 1 , the method comprises the following implementation steps:

[0070] Step 101: In response to receiving first audio data, determine first state information, and input the first state information to a DeepQ-Network (DQN) to be trained.

[0071] The first state information includes a first speech feature of the first speech data in the first audio data, and the DQN to be trained includes a loss function.

[0072] Specifically, the first audio data may include a preset keyword, or an approximate word with a similar semantic meaning to the preset keyword, or a sound-like word with a similar pronunciation to the preset keyword. For example, the preset keyword may be a preset wake-up word for a voice assistant. Therefore, the first audio data may be obtained by the following method:

[0073] First, audio data to be processed can be continuously collected and cached. The cached audio data to be processed is then framed to obtain multiple audio frames. Next, in response to detecting the first audio frame containing a preset keyword among the multiple audio frames, a first time stamp carried by the first audio frame is determined. Using the first time stamp as a reference, first target audio data of a first preset time length can be intercepted forward, and second target audio data of a second preset time length can be intercepted backward. The first target audio data and the second target audio data then constitute the first audio data.

[0074] The first audio data may include first voice data and noise data. The first voice data corresponds to the voice information sent by the user. The noise data may include background sound and / or audio data in the environment.

[0075] Furthermore, the DQN to be trained also includes a Q network to be trained. The Q network to be trained may include multiple hidden layers. The DQN to be trained can be understood as a reinforcement learning model to be trained with the Q network to be trained as its infrastructure. The loss function can be used to optimize the parameters of the multiple hidden layers. The parameters of the hidden layers may include the biases and weights of the hidden layers.

[0076] For ease of understanding, the following is an explanation of how to train DQN. Please refer to Figure 2 After the first state information s is input into the DQN to be trained, the Q-network to be trained in the DQN to be trained determines pending Q-values ​​for each type of action in the preset action set based on the first state information, obtaining multiple pending Q-values. The DQN to be trained can then select a target Q-value from the multiple pending Q-values ​​based on the target behavior strategy, thereby determining a first type of action corresponding to the target Q-value. The DQN to be trained performs the first type of action to obtain a first reward value through a reward function. Furthermore, by providing feedback on the first type of action to the user, new state information s' corresponding to the first type of action is obtained; that is, the new state information s' can be obtained through interaction with the user. This first reward value and new state information s' can be used to calculate a loss function value. Simultaneously, the parameters of the Q-network to be trained are tuned based on the loss function value, namely, the parameters of the hidden layer in the Q-network to be trained are updated and optimized. The hidden layer parameters include the weights and biases of the hidden layer. In this manner, the Q-network to be trained can be trained based on gradient descent and / or backpropagation to determine the direction and magnitude of the update of the hidden layer parameters.

[0077] The Q values ​​to be processed correspond one-to-one to the type actions in the preset action set, where the type actions in the preset action set include at least wake-up and non-wake-up.

[0078] Step 102: In the DQN to be trained, based on the target rule, update the first exploration rate, and determine the first type of action corresponding to the first state information based on the second exploration rate obtained by updating the first exploration rate.

[0079] The target rule includes, if the average reward value of M consecutive iteration steps is greater than a preset first threshold, updating the first exploration rate using a preset exploration rate, and the preset exploration rate is greater than the first exploration rate.

[0080] Specifically, the first exploration rate and the updated second exploration rate are both positive numbers less than 1.

[0081] The above average reward value can be obtained by calculating the arithmetic mean of the first reward value of each iteration step in M ​​iteration steps:

[0082] Among them, R ave is the average reward value mentioned above, M is the preset experience value, ncur is the most recent iteration step, r n is the first reward value of the nth iteration step.

[0083] The above-mentioned iteration step can be understood as a process of updating the Q network to be trained in the DQN to be trained, so each iteration step in the M iteration steps can correspond to a first reward value. The iteration step carries the iteration step number corresponding to the number of training times. The iteration step number increases with the number of training times, so n cur It is the iteration step with the largest iteration step number.

[0084] In this way, by comparing the relative size relationship between the average reward value of M iteration steps and the first threshold, it can be determined whether to search and optimize globally through a larger preset exploration rate to avoid falling into the local optimum and avoid the problem of insufficient generalization ability of the trained target DQN.

[0085] Furthermore, the first type of action can be selected from a preset action set. The preset action set includes multiple types of actions. Exemplarily, the preset action set may include at least wake-up and do not wake-up.

[0086] Alternatively, in one embodiment, the preset action set may include: immediate wake-up, delayed wake-up, and no wake-up. The first type of action can be selected from: immediate wake-up, delayed wake-up, and no wake-up. The delayed wake-up action may include delay information, which is used for timing so that the voice assistant wakes up on time when the delay information is met, improving the user experience.

[0087] Furthermore, during the training process of the DQN to be trained, as the number of iterations increases, the iteration step number accumulates with the iteration number, forming an iteration step sequence. The iteration steps in this iteration step sequence are arranged sequentially according to the iteration step number. Therefore, each iteration step corresponds to at least the iteration step number, and in each iteration step, after performing the first type of action, a first reward value corresponding to the first type of action is generated. Therefore, each iteration step also has a one-to-one correspondence with the first reward value. Furthermore, the first type of action also has a one-to-one correspondence with the first reward value; thus, each iteration step can have a one-to-one correspondence with the first type of action.

[0088] In one embodiment, the first reward value is obtained according to the following method: First, first audio data may be processed using ASR (Automatic Speech Recognition) technology to extract first speech data from the first audio data. Since the first speech data may be composed of continuous and / or discontinuous user speech, the cutoff time of the first speech data may be determined to facilitate the determination of second speech data. Then, the second speech data may be determined based on a preset time window and the cutoff time of the first speech data in the first audio data. The start time of the second speech data is later than the cutoff time of the first speech data. Next, the first and second speech data may be converted into first text data. Natural Language Understanding (NLU) processing may then be performed on the first text data using a pre-trained speech model to determine user intent in the first text data and obtain a third type of action. Finally, the first type of action may be verified based on the third type of action to generate the aforementioned first reward value. For example, if the first type of action is consistent with the third type of action, a positive first reward value may be generated. For another example, if the first type of action is inconsistent with the third type of action, a negative first reward value may be generated.

[0089] In this embodiment, a third type of action is generated through the first voice data and the second voice data. Since the second voice data is output by the user after the first voice data, the user intention in the first voice data can be further corrected through the second voice data, thereby improving the accuracy of the second action type, and then effectively improving the accuracy of the first reward value, thereby achieving the improvement of the accuracy of the DQN to be trained and the training efficiency.

[0090] In one embodiment, the first reward value may be generated by the following reward function:

[0091]

[0092] Here, a represents the first type of action, and a' represents the third type of action.

[0093] To further improve the training efficiency of the DQN to be trained and enhance the DQN's ability to understand user instructions, in one embodiment, a first reward value may be generated based on the degree of deviation between the first type of action and the third type of action.

[0094] For example, the first type of action and the third type of action can be processed by the reward function to obtain the first reward value. For example, the preset action set is represented as follows {A w ,A d ,A r}, where A w Indicates immediate wake-up, Ad Indicates delayed wake-up, A r = means refusing to wake up (i.e., not waking up); the first type of action and the third type of action are both selected from the aforementioned preset action set. Then the aforementioned reward function can be expressed as follows:

[0095]

[0096] Here, a represents the first type of action, and a' represents the third type of action.

[0097] Step 103: Determine a target DQN based on the first type of action and the loss function.

[0098] Specifically, a loss function value of a loss function may be determined based on the first type of action, and the loss function value is used to optimize parameters in the Q network to be trained.

[0099] In response to the loss function converging and / or the loss function value being less than a predetermined fourth threshold, training is determined to be complete, and a target Q network and a target DQN including the target Q network are obtained. The target DQN can be understood as a reinforcement learning model based on the target Q network.

[0100] In one embodiment, a first reward value corresponding to the first state information may be determined based on the first type of action, and updated state information obtained by updating the first state information may be determined. The updated state information includes information on the cumulative number of times the first reward value is positive.

[0101] Then, the first state information, the first type of action, the first reward value, and the updated state information are processed using a loss function to obtain a loss function value. The loss function value indicates the difference between the Q value in the current iteration step and the target Q value. The target Q value is the Q value selected from the Q values ​​generated by the Q network to be trained after the state information is updated.

[0102] Exemplarily, the target Q value may be generated based on the following Q value update rule:

[0103] Q G (s,a)←Q(s,a)+α[r+γmax a' Q(s',a')-Q(s,a)];

[0104] Among them, Q G(s, a) is the target Q value, Q(s, a) is the Q value in the iteration step with the largest iteration step number, α is the preset learning rate, 0<α<1, a is the first type of action corresponding to the first state information, r is the first reward value obtained by executing the first type of action a, γ is the preset discount factor, 0<γ<1, s is the first state information, s′ is the updated state information obtained by executing the first type of action, and a′ is the fourth type of action corresponding to the updated state information s′. The above (s, a, s', a') can also be obtained by random sampling in the experience replay pool. Similar to the first type of action, this fourth type of action is selected from the preset action set. For the setting of the preset action set, please refer to the setting in the aforementioned step 102 and will not be repeated here.

[0105] Finally, in response to the loss function value converging and / or the loss function value being less than a preset fourth threshold, a target Q network and a target DQN including the target Q network are obtained.

[0106] In one embodiment, the loss function L is expressed as follows:

[0107]

[0108] Where θ represents the parameters of the Q network to be trained in the DQN to be trained, s is the first state information, a is the first type of action, and s' is the updated state information. y is the target Q value, y = r + γmax a' Q(s',a'); a' is a fourth type of action corresponding to updating state information s'.

[0109] Furthermore, the updated state information may also include target feature information in the third audio data. The third audio data is user data received after the first audio data is received. The time difference between the expiration time of the third audio data and the expiration time of the first audio data is less than a preset time threshold. Thus, the user feedback (i.e., target feature information) after the DQN to be trained performs the first type of action can be determined by the third audio data whose time interval with the first audio data is less than the preset time threshold, and the user feedback is used as the updated state information.

[0110] Step 104: In response to receiving the second audio data, input the second state information into the target DQN to obtain a second type of action.

[0111] The second status information corresponds to the second audio data.

[0112] Specifically, the second audio data may contain a preset keyword, or an approximate word with a similar semantic meaning to the preset keyword, or a phonetic word with a similar pronunciation to the preset keyword. The second audio data is determined in the same manner as the first audio data.

[0113] The target behavior strategy in the target DQN is a greedy strategy. Therefore, in the target DQN, based on the second state information, the Q value to be processed corresponding to each type of action in the preset action set is determined;

[0114] Then, based on a greedy strategy, the type of action corresponding to the largest pending Q value can be determined as a second type of action. Similar to the first type of action, this second type of action is selected from a preset action set. The setting of this preset action set is described in step 102 above and will not be repeated here.

[0115] The method for determining the second state information based on the second audio data can be consistent with the method for determining the first state information based on the first audio data. In one embodiment, the second state information corresponding to the second voice data in the second audio data can be determined based on a preset attention mechanism.

[0116] In the audio data processing method described in steps 101-104 above, the target DQN is trained using the first audio data, and nodes with average reward values ​​greater than a first threshold are used to expand the search range. Global optimization is performed at a preset exploration rate greater than the first exploration rate, thereby avoiding local optima during training and improving the generalization of the target DQN. In this way, when the target DQN is used to wake up a voice assistant, the accuracy of the output second type of action can be effectively improved.

[0117] In one embodiment, the first state information is determined in the following manner: First, historical state information is obtained. The historical state information includes the cumulative number of positive first reward values ​​corresponding to iteration steps within a time window of a preset length. Then, first voice feature information of the first voice data in the first audio data is determined. Finally, the first state information is generated based on the historical state information and the first voice feature information. For example, the first historical state information and the first voice feature information can be concatenated to obtain the first state information. For example, the cumulative number of positive first reward values ​​corresponding to iteration steps within the time window of the preset length can be used as element a′ and added to a preset position in the first voice vector corresponding to the first voice feature information to obtain the first state information. Assuming that the first voice vector corresponding to the first voice feature information is A = (a, b ... c), after adding the element a′ corresponding to the cumulative value, the first state information can be expressed as A = (a, b ... c, a′).

[0118] The time window of the preset length may be determined based on the time stamp corresponding to the iteration step with the largest iteration step sequence number in the iteration step sequence.

[0119] In the above embodiment for determining the first state information, after obtaining the historical state information, by extracting the first voice data from the first audio data, and then extracting the first voice feature information from the first voice data, the purity of the user voice information is improved, the computational complexity and processing noise of the DQN to be trained are reduced, thereby improving the training efficiency and accuracy of the DQN to be trained.

[0120] Furthermore, to remove noise and redundant information while further enhancing the information richness of the first speech feature information, the first speech feature information can be determined in the following manner: first, the first audio data is denoised based on a preset reference noise signal to obtain the first speech data. Specifically, an endpoint detection algorithm can be used to calculate the short-term energy and zero-crossing rate of the denoised second audio data, and the first speech data can be extracted based on a preset threshold.

[0121] Then, MFCCs (Mel-Frequency Cepstral Coefficients) are extracted from the first speech data to obtain a first feature matrix. Perceptual Linear Prediction (PLP) processing can then be performed on the first speech data to obtain a second feature matrix. The first and second feature matrices are then concatenated to obtain a third feature matrix. Finally, the third feature matrix can be used as the first speech feature information.

[0122] In the aforementioned first feature matrix, second feature matrix and third feature matrix, the feature vectors contained in each can correspond to a speech frame in the first speech data respectively; that is, each row is a feature vector, corresponding one-to-one to a speech frame in the first speech data.

[0123] Furthermore, in one embodiment, after obtaining the third feature matrix, the third feature matrix may be processed through a preset attention mechanism to obtain the first speech feature information.

[0124] The preset attention mechanism can be a self-attention mechanism or a multi-head attention mechanism. In one embodiment, in order to fully capture the complex relationships between characters, words, and speech fragments in the aforementioned third feature matrix and provide rich and discriminative feature representations for the DQN to be trained and / or the target DQN, the third feature matrix can be extracted and processed by a multi-head attention mechanism. The specific instructions are as follows:

[0125] In the multi-head attention mechanism, first, the third feature matrix X is converted into Q (query) matrix, K (key) matrix, and V (value) matrix:

[0126] Q=XWQ ;

[0127] K=XW K ;

[0128] V=XW V ;

[0129] Among them, W Q 、W K 、W V Obtained through pre-training, and W Q 、W K 、W V The dimensions of correspond to the third characteristic matrix X.

[0130] Then, according to the number of heads h, the above Q matrix, K matrix, and V matrix are linearly transformed:

[0131]

[0132] Where i is an integer from 1 to h, h i represents the i-th attention head.

[0133] In this way, we get the Q matrix that each attention head needs to process K matrix V Matrix Afterwards, the attention matrix is ​​calculated separately: for The mth row, K n for The nth row, d k for The dimension of . Then we get the matrix corresponding to the attention head A in matrix A mn It represents the attention weight of the nth position to the mth position, reflecting the importance of the nth eigenvector to the mth eigenvector in the third feature matrix.

[0134] Then the output matrix in each attention head can be calculated

[0135] Finally, the output matrices of each attention head are spliced ​​along the feature dimension, and linear transformation is performed through the weight matrix W0 to map the spliced ​​matrix back to the original feature dimension to obtain the target feature matrix O G As the aforementioned first voice feature information:

[0136] Furthermore, in the DQN to be trained, updating a first exploration rate based on a target rule, and determining a first type of action corresponding to the first state information based on a second exploration rate obtained by updating the first exploration rate, includes:

[0137] First, the first state information is processed by the Q network to be trained in the aforementioned DQN to be trained to obtain a plurality of Q values ​​to be processed. The plurality of Q values ​​to be processed correspond to the type actions (for example, the first type actions) in the aforementioned preset action set. Then, based on the second exploration rate, the target behavior strategy is determined. The target behavior strategy is used to select a target Q value from a plurality of Q values ​​to be processed. The target behavior strategy is a greedy strategy or an exploration strategy. Finally, based on the target behavior strategy, a target Q value can be selected from a plurality of Q values ​​to be processed; then the type action corresponding to the target Q value can be determined as the first type action.

[0138] The above greedy strategy indicates that among multiple Q values ​​to be processed, the maximum value is selected as the target Q value.

[0139] The above exploration strategy indicates that, among multiple to-be-processed Q values, a to-be-processed Q value is randomly selected as the target Q value.

[0140] In one embodiment, the first exploration rate can be represented as ε1, and the second exploration rate can be represented as ε2. Thus, determining the target behavior strategy based on the second exploration rate includes: determining the target behavior strategy as an exploration strategy with a probability of ε2. Determining the target behavior strategy as a greedy strategy with a probability of (1-ε2).

[0141] Furthermore, in order to ensure that the loss function in the DQN to be trained can converge reasonably and effectively to achieve efficient training and prevent the DQN to be trained from falling into a local optimum during the iteration process, an embodiment is provided below to illustrate the update of the first exploration rate:

[0142] First, determine the iterative step sequence. The iterative step sequence includes iterative steps in order of iterative step numbers, and the first reward values ​​corresponding to the iterative steps. The iterative step sequence can be updated according to the number of iterations and the first reward value after each iteration generates the first reward value. That is, after each iteration generates the first reward value, the maximum iterative step number in the iterative step sequence is increased by 1, and the maximum iterative step number obtained by adding 1 corresponds one to one with the newly generated first reward value. Then, the iterative step number and / or the first reward value are processed by the target rule to determine the update strategy. Therefore, the target rule is to determine the rule of the update strategy by the iterative step number and / or the first reward value. Finally, the first exploration rate can be updated based on the update strategy to obtain the second exploration rate. The following provides an embodiment for determining the aforementioned update strategy:

[0143] In response to the maximum iteration step number in the iteration step sequence being less than the second threshold, it can be determined that the training is in the initial stage, so the update strategy is determined to be the first update strategy. The first update strategy indicates that the first exploration rate is decayed as the iteration step number increases.

[0144] Alternatively, in response to the average reward value of M consecutive iterations in the iteration sequence being greater than a first threshold, the training stage is determined to be an intermediate stage, and the update strategy is determined to be a second update strategy. The second update strategy indicates that the first exploration rate is updated based on a preset exploration rate, where M is a preset empirical value. The preset exploration rate is greater than 0.5. Exemplarily, the preset exploration rate is 0.6.

[0145] or,

[0146] In response to the number of positive first reward values ​​for K consecutive iterations in the iteration sequence being greater than a third threshold, it is determined that training has entered the final stage, and the update strategy can be determined to be a third update strategy. The third update strategy indicates that the first exploration rate is decayed to achieve the update. K is a preset empirical value. Specifically, the third update strategy may be to decay the first exploration rate as the iteration number increases.

[0147] The decay rate of the first exploration rate in the third update strategy may be greater than or equal to the decay rate of the first exploration rate in the first update strategy.

[0148] In one embodiment, the first update strategy can be implemented by a first attenuation formula. When the update strategy is determined to be the first update strategy, the first attenuation formula can be determined, and the first exploration rate can be processed by the first attenuation formula to obtain the second exploration rate.

[0149] The first attenuation formula can be: Wherein, ε' is the second exploration rate obtained by updating the first exploration rate, ε is the first exploration rate, u1 is the preset first proportional factor, λ1 is the preset first exploration constant, and n is the iterative step number.

[0150] For example, u1=1, λ1=0.001; ε'=ε·e -0.001n And the initial exploration rate when n=1 can be 0.3.

[0151] To further improve the training efficiency of the DQN to be trained, in one embodiment, the decay rate in the third update strategy is greater than the decay rate in the first update strategy.

[0152] The third update strategy can be implemented through a third attenuation formula. When the update strategy is determined to be the third update strategy, the first exploration rate can be processed through the third attenuation formula to obtain the second exploration rate, thereby implementing the update of the first exploration rate.

[0153] In one embodiment, the third attenuation formula is: Wherein, ε' is the second exploration rate obtained by updating the first exploration rate, ε is the first exploration rate, u2 is a preset second proportional factor, λ2 is a preset second exploration constant, and n is the iterative step number. λ2>λ1.

[0154] For example, the value of λ2 can be 0.002, 0.003, 0.004, 0.005, 0.006, 0.007, 0.008, 0.009, 0.01, or a range consisting of any two of these values. u2 can be selected from 0.5, 1.0, or a range therebetween.

[0155] In this way, by setting a larger λ2 in the third attenuation formula, the attenuation rate is increased, ensuring that the parameters in the DQN to be trained can continue to be effectively optimized while accelerating the convergence of the loss function in the DQN to be trained, and / or accelerating the speed at which the loss function value decreases until it is less than the aforementioned fourth threshold, thereby obtaining the target DQN.

[0156] In summary, the target DQN obtained through the aforementioned training process can effectively improve the accuracy of the second type of action output by processing the second audio data. Still taking the voice assistant as an example, the target DQN can effectively distinguish whether the second audio data containing the wake-up word contains the intention of the wake-up assistant by processing the second state information corresponding to the second audio data. In this way, it can avoid the problem that the user mentions the wake-up word during the chat and only uses it as chat content without involving the wake-up demand; or mentions something that sounds similar to the wake-up word but actually talks about other things, causing the voice assistant to be mistakenly awakened, thereby reducing the user experience.

[0157] It is understood that in the embodiments of the present application, both the first exploration rate and the second exploration rate are used to determine the target behavior strategy. The second exploration rate is obtained by updating the first exploration rate. Furthermore, to obtain the second exploration rate, an update strategy may be employed to update the first exploration rate. This update strategy may be implemented to update the first exploration rate by determining the training stage.

[0158] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0159] Based on the same inventive concept, Figure 3As shown, the embodiment of the present application provides an audio data processing device, including: a first input module 301, an action module 302, a loss module 303, and a second input module 303, wherein:

[0160] The first input module 301 is configured to determine first state information in response to receiving first audio data, and input the first state information to the DQN to be trained. The DQN to be trained includes a loss function

[0161] An action module 302 is configured to update a first exploration rate in the DQN to be trained based on a target rule, and determine a first type of action corresponding to the first state information based on a second exploration rate obtained by updating the first exploration rate; wherein the target rule includes updating the first exploration rate using a preset exploration rate if the average reward value of M consecutive iteration steps is greater than a preset first threshold, the preset exploration rate being greater than the first exploration rate, and M being a preset experience value.

[0162] The loss module 303 is configured to determine a target DQN based on the first type of action and the loss function.

[0163] The second input module 304 is configured to input second state information into the target DQN in response to receiving the second audio data, so as to obtain a second type of action; wherein the second state information corresponds to the second audio data.

[0164] In one embodiment, the action module 302 is specifically configured to:

[0165] The first state information is processed by the Q network to be trained in the DQN to be trained to obtain multiple Q values ​​to be processed; based on the second exploration rate, a target behavior strategy is determined; wherein the target behavior strategy is an exploration strategy or a greedy strategy; based on the target behavior strategy, a target Q value is selected from the multiple Q values ​​to be processed to obtain the first type of action corresponding to the target Q value.

[0166] In one embodiment, the action module 302 is further used to: determine an iteration step sequence; wherein the iteration step sequence includes iteration steps ordered by iteration step numbers, and the first reward value corresponding to the iteration steps; process the iteration step number and / or the first reward value through the target rule to determine an update strategy; and update the first exploration rate based on the update strategy.

[0167] In one embodiment, the action module 302 is specifically used to: in response to the maximum iteration step number in the iteration step sequence being less than a preset second threshold, determine that the update strategy is a first update strategy; wherein the first update strategy indicates that the first exploration rate decays with an increase in the iteration step number; or, in response to the average reward value of M consecutive iteration steps in the iteration step sequence being greater than the first threshold, determine that the update strategy is a second update strategy; wherein the second update strategy indicates that the first exploration rate is updated based on the preset exploration rate, and the preset exploration rate is greater than 0.5 and less than 1; or, in response to the number of positive first reward values ​​of K consecutive iteration steps in the iteration step sequence being greater than a preset third threshold, determine that the update strategy is a third update strategy; wherein the third update strategy indicates that the first exploration rate decays with an increase in the iteration step number, and K is a preset experience value.

[0168] In one embodiment, the action module 302 is specifically configured to:

[0169] In response to the update strategy being the first update strategy, the first exploration rate is processed using a first attenuation formula to obtain the second exploration rate; wherein the first attenuation formula is: Wherein, ε' is the second exploration rate, ε is the first exploration rate, u1 is a preset first proportional factor, λ1 is a preset first exploration constant, and n is the iteration step number.

[0170] In one embodiment, a decay rate of the first exploration rate in the third update strategy is greater than a decay rate of the first exploration rate in the first update strategy.

[0171] In one embodiment, the average reward value is the arithmetic mean of the first reward values ​​corresponding to the first type of action in the M consecutive iteration steps; the action module 302 is further configured to:

[0172] Determine second voice data based on a preset time window and a deadline of first voice data in the first audio data; wherein the start time of the second voice data is later than the deadline of the first voice data; convert the first voice data and the second voice data into first text data; perform natural language understanding processing on the first text data through a pre-trained language model to obtain a third type of action; verify the first type of action based on the third type of action to generate the first reward value.

[0173] In one embodiment, the first input module 301 is specifically configured to:

[0174] Obtain historical status information; wherein the historical status information includes the cumulative number of positive first reward values ​​corresponding to the iteration steps within a preset time window; determine the first voice feature information of the first voice data in the first audio data; generate the first status information based on the historical status information and the first voice feature information.

[0175] In one embodiment, the first input module 301 is specifically configured to:

[0176] Based on a preset reference noise signal, the first audio data is denoised to obtain first speech data; Mel-spectrogram cepstral coefficients in the first speech data are extracted to obtain a first feature matrix; perceptual linear prediction processing is performed on the first speech data to obtain a second feature matrix; the first feature matrix and the second feature matrix are concatenated to obtain a third feature matrix; and the third feature matrix is ​​processed through a preset attention mechanism to obtain the first speech feature information.

[0177] For the specific definition of the audio data processing device, please refer to the definition of the audio data processing method above and will not be repeated here. The various modules in the above-mentioned audio data processing device can be implemented in whole or in part by software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in hardware form, or can be stored in the memory of the electronic device in software form, so that the processor can call and execute the corresponding operations of the above-mentioned modules.

[0178] Based on the same inventive concept, see Figure 4 The present application also provides an electronic device. In one embodiment, the electronic device may include a memory 401, a communication module 403, and one or more processors 402 as shown in the figure.

[0179] The memory 401 is used to store computer programs executed by the processor 402. The memory 401 may mainly include a program storage area and a data storage area. The program storage area may store an operating system; the data storage area may store various operating instruction sets.

[0180] Memory 401 may be a volatile memory, such as random-access memory (RAM); a non-volatile memory, such as read-only memory, flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 401 may be a combination of the above memories.

[0181] The processor 402 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 402 is configured to implement the above-mentioned audio data processing method when calling the computer program stored in the memory 401 .

[0182] The communication module 403 is used to communicate with terminal devices, site devices or other network devices.

[0183] The specific connection medium between the memory 401, the communication module 403 and the processor 402 is not limited in the embodiment of the present application. Figure 4 In the embodiment, the memory 401 and the processor 402 are connected via a bus 404. Figure 4 The connections between the other components are shown in bold lines, which are only for illustration and are not intended to be limiting. The bus 404 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Figure 4 The diagram shows a single thick line, but this does not indicate that there is only one bus or one type of bus.

[0184] The memory 401 stores a computer storage medium, which stores computer-executable instructions for implementing the method for determining audio data processing according to an embodiment of the present application. The processor 402 is configured to execute the audio data processing method according to each embodiment of the computer-executable instructions.

[0185] In one embodiment, when the computer executable instructions are executed by the processor 402, the following steps are implemented:

[0186] In response to receiving the first audio data, determining first state information, and inputting the first state information into a DQN to be trained; wherein the DQN to be trained includes a loss function;

[0187] In the DQN to be trained, a first exploration rate is updated based on a target rule, and a first type of action corresponding to the first state information is determined based on a second exploration rate obtained by updating the first exploration rate; wherein the target rule includes updating the first exploration rate using a preset exploration rate if an average reward value of M consecutive iterations is greater than a preset first threshold, the preset exploration rate being greater than the first exploration rate, and M being a preset experience value;

[0188] Determining a target DQN based on the first type of action and the loss function;

[0189] In response to receiving the second audio data, second state information is input into the target DQN to obtain a second type of action; wherein the second state information corresponds to the second audio data.

[0190] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0191] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0192] In response to receiving the first audio data, determining first state information, and inputting the first state information into a DQN to be trained; wherein the DQN to be trained includes a loss function;

[0193] In the DQN to be trained, a first exploration rate is updated based on a target rule, and a first type of action corresponding to the first state information is determined based on a second exploration rate obtained by updating the first exploration rate; wherein the target rule includes updating the first exploration rate using a preset exploration rate if an average reward value of M consecutive iterations is greater than a preset first threshold, the preset exploration rate being greater than the first exploration rate, and M being a preset experience value;

[0194] Determining a target DQN based on the first type of action and the loss function;

[0195] In response to receiving the second audio data, second state information is input into the target DQN to obtain a second type of action; wherein the second state information corresponds to the second audio data.

[0196] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0197] The first state information is processed by the Q network to be trained in the DQN to be trained to obtain multiple Q values ​​to be processed; based on the second exploration rate, a target behavior strategy is determined; wherein the target behavior strategy is an exploration strategy or a greedy strategy; based on the target behavior strategy, a target Q value is selected from the multiple Q values ​​to be processed to obtain the first type of action corresponding to the target Q value.

[0198] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0199] Determine an iteration step sequence; wherein the iteration step sequence includes iteration steps ordered by iteration step numbers and the first reward values ​​corresponding to the iteration steps; process the iteration step numbers and / or the first reward values ​​through the target rule to determine an update strategy; and update the first exploration rate based on the update strategy.

[0200] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0201] In response to the maximum iteration step number in the iteration step sequence being less than a preset second threshold, the update strategy is determined to be the first update strategy; wherein the first update strategy indicates that the first exploration rate decays as the iteration step number increases; or, in response to the average reward value of M consecutive iteration steps in the iteration step sequence being greater than the first threshold, the update strategy is determined to be the second update strategy; wherein the second update strategy indicates that the first exploration rate is updated based on the preset exploration rate, and the preset exploration rate is greater than 0.5 and less than 1; or, in response to the number of positive first reward values ​​of K consecutive iteration steps in the iteration step sequence being greater than a preset third threshold, the update strategy is determined to be the third update strategy; wherein the third update strategy indicates that the first exploration rate decays as the iteration step number increases, and K is a preset experience value.

[0202] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0203] In response to the update strategy being the first update strategy, the first exploration rate is processed using a first attenuation formula to obtain the second exploration rate; wherein the first attenuation formula is: Wherein, ε' is the second exploration rate, ε is the first exploration rate, u1 is a preset first proportional factor, λ1 is a preset first exploration constant, and n is the iteration step number.

[0204] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0205] The average reward value is the arithmetic mean of the first reward values ​​corresponding to the first type of action in the M consecutive iteration steps; the first reward value is obtained by the following method: determining the second voice data based on a preset time window and the deadline of the first voice data in the first audio data; wherein the start time of the second voice data is later than the deadline of the first voice data; converting the first voice data and the second voice data into first text data; performing natural language understanding processing on the first text data through a pre-trained language model to obtain a third type of action; and verifying the first type of action based on the third type of action to generate the first reward value.

[0206] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0207] Obtain historical status information; wherein the historical status information includes the cumulative number of positive first reward values ​​corresponding to the iteration steps within a preset time window; determine the first voice feature information of the first voice data in the first audio data; generate the first status information based on the historical status information and the first voice feature information.

[0208] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented:

[0209] Based on a preset reference noise signal, the first audio data is denoised to obtain first speech data; Mel-spectrogram cepstral coefficients in the first speech data are extracted to obtain a first feature matrix; perceptual linear prediction processing is performed on the first speech data to obtain a second feature matrix; the first feature matrix and the second feature matrix are concatenated to obtain a third feature matrix; and the third feature matrix is ​​processed through a preset attention mechanism to obtain the first speech feature information.

[0210] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0211] Based on the same inventive concept, an embodiment of the present application further provides a computer program product, including a computer program, which implements any of the above-mentioned audio data processing methods when executed by a processor.

[0212] The program code for executing the computer program product of the present application may be written in any combination of one or more programming languages, and the program code may be executed entirely on the user device, partially on the user device, as an independent software package, partially on the user device and partially on a remote device, or entirely on the remote device.

[0213] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0214] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0215] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0216] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of user-operated steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0217] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A method for processing audio data, characterized in that: include: In response to receiving the first audio data, determining first state information, and inputting the first state information into a DQN to be trained; wherein the DQN to be trained includes a loss function; In the DQN to be trained, a first exploration rate is updated based on a target rule, and a first type of action corresponding to the first state information is determined based on a second exploration rate obtained by updating the first exploration rate; wherein the target rule includes updating the first exploration rate using a preset exploration rate if an average reward value of M consecutive iterations is greater than a preset first threshold, the preset exploration rate being greater than the first exploration rate, and M being a preset experience value; Determining a target DQN based on the first type of action and the loss function; In response to receiving the second audio data, second state information is input into the target DQN to obtain a second type of action; wherein the second state information corresponds to the second audio data.

2. The method according to claim 1, wherein The updating of the first exploration rate in the DQN to be trained based on the target rule, and determining the first type of action corresponding to the first state information based on the second exploration rate obtained by updating the first exploration rate, includes: Processing the first state information by a to-be-trained Q network in the to-be-trained DQN to obtain a plurality of to-be-processed Q values; Determining a target behavior strategy based on the second exploration rate; wherein the target behavior strategy is an exploration strategy or a greedy strategy; Based on the target behavior strategy, a target Q value is selected from the multiple Q values ​​to be processed, and the first type of action corresponding to the target Q value is obtained.

3. The method according to claim 1, wherein The updating of the first exploration rate based on the target rule includes: Determine an iteration step sequence; wherein the iteration step sequence includes the iteration steps in order of iteration step sequence numbers and first reward values ​​corresponding to the iteration steps; Processing the iteration step number and / or the first reward value through the target rule to determine an update strategy; Based on the update strategy, the first exploration rate is updated.

4. The method according to claim 3, wherein The step of processing the iteration step number and / or the first reward value by using the target rule to determine an update strategy includes: In response to a maximum iteration step number in the iteration step sequence being less than a preset second threshold, determining that the update strategy is a first update strategy; wherein the first update strategy indicates that the first exploration rate decays as the iteration step number increases; or, In response to the average reward value of M consecutive iteration steps in the iteration step sequence being greater than the first threshold, determining that the update strategy is a second update strategy; wherein the second update strategy indicates updating the first exploration rate based on the preset exploration rate, and the preset exploration rate is greater than 0.5 and less than 1; or, In response to the fact that the number of positive first reward values ​​in K consecutive iteration steps in the iteration step sequence is greater than a preset third threshold, the update strategy is determined to be a third update strategy; wherein the third update strategy indicates that the first exploration rate decays with an increase in the iteration step number, and K is a preset experience value.

5. The method according to claim 4, wherein Updating the first exploration rate based on the update strategy includes: In response to the update strategy being the first update strategy, the first exploration rate is processed using a first attenuation formula to obtain the second exploration rate; wherein the first attenuation formula is: ε ' is the second exploration rate, ε is the first exploration rate, u1 is the preset first proportional factor, λ1 is the preset first exploration constant, and n is the iteration step number.

6. The method according to claim 4, wherein The decay rate of the first exploration rate in the third update strategy is greater than the decay rate of the first exploration rate in the first update strategy.

7. The method according to any one of claims 1 to 6, wherein: The average reward value is the arithmetic mean of the first reward values ​​corresponding to the first type of action in the M consecutive iteration steps; The first reward value is obtained by the following method: Determining second voice data based on a preset time window and a cutoff time of first voice data in the first audio data; wherein a start time of the second voice data is later than the cutoff time of the first voice data; Converting the first voice data and the second voice data into first text data; Performing natural language understanding processing on the first text data using a pre-trained language model to obtain a third type of action; The first type of action is verified based on the third type of action to generate the first reward value.

8. The method according to claim 1, wherein The determining of the first state information includes: Acquiring historical status information; wherein the historical status information includes the cumulative number of positive first reward values ​​corresponding to the iteration steps within a preset time window; determining first voice feature information of first voice data in the first audio data; The first state information is generated based on the historical state information and the first voice feature information.

9. The method according to claim 8, wherein The determining the first voice feature information of the first voice data in the first audio data includes: Performing denoising processing on the first audio data based on a preset reference noise signal to obtain first speech data; Extracting Mel-spectrogram cepstral coefficients from the first speech data to obtain a first feature matrix; performing perceptual linear prediction processing on the first speech data to obtain a second feature matrix; Concatenating the first characteristic matrix and the second characteristic matrix to obtain a third characteristic matrix; The third feature matrix is ​​processed through a preset attention mechanism to obtain the first speech feature information.

10. An audio data processing device, characterized in that: include: A first input module, configured to determine first state information in response to receiving first audio data, and input the first state information to a DQN to be trained; wherein the DQN to be trained includes a loss function; an action module, configured to update, in the DQN to be trained, a first exploration rate based on a target rule, and determine, based on a second exploration rate obtained by updating the first exploration rate, a first type of action corresponding to the first state information; wherein the target rule includes updating the first exploration rate using a preset exploration rate if an average reward value for M consecutive iterations is greater than a preset first threshold, the preset exploration rate being greater than the first exploration rate, and M being a preset experience value; a loss module, configured to determine a target DQN based on the first type of action and the loss function; The second input module is configured to input second state information into the target DQN in response to receiving the second audio data, so as to obtain a second type of action; wherein the second state information corresponds to the second audio data.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and configured to run on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images