Audio data processing method, device, electronic device, storage medium and product

Abstract

The present application provides an audio data processing method, device, electronic device, storage medium and product, which belongs to the field of voice interaction technology. The method includes: receiving input audio data, performing recognition processing on the audio data, and outputting the recognition result; when the recognition result includes a wake-up word, obtaining a decoding graph of the audio data, the decoding graph includes a jump identifier sequence of the decoding path corresponding to the audio data, and the jump identifier sequence is used to indicate the phoneme changes between adjacent audio frames in the audio data; determining a target jump identifier from the jump identifier sequence, the target jump identifier satisfies a target condition, and the target condition indicates that the jump identifier in the jump identifier sequence belongs to the jump identifier corresponding to the wake-up phoneme sequence of the wake-up word; based on the target jump identifier, determining the first endpoint of the wake-up audio data corresponding to the wake-up word. This scheme implements endpoint detection at the phoneme level, and can accurately detect the first endpoint of the wake-up audio data.

Description

Technical Field

[0001] The present application relates to the field of voice interaction technology, and in particular to an audio data processing method, device, electronic device, storage medium and product. Background Art

[0002] Endpoint detection is an important step in audio processing. Through endpoint detection, the first endpoint (that is, the starting point of the target audio segment) and the last endpoint (that is, the ending point of the target audio segment) of the target audio segment in the audio can be determined, and then the target audio segment can be cut out from the large audio segment, so as to perform the next step of processing based on the cut target audio segment. For example, the wake-up audio (that is, the audio corresponding to the wake-up word) is cut out from a large segment of input audio, and then the wake-up audio can be applied to the model training. Therefore, how to accurately detect the endpoints of the wake-up audio has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide an audio data processing method, device, electronic device, storage medium and product, which can improve the accuracy of determining the first endpoint of the wake-up word audio. The technical solution is as follows:

[0004] In one aspect, a method for processing audio data is provided, the method comprising:

[0005] Receive input audio data, perform recognition processing on the audio data, and output a recognition result;

[0006] In a case where the recognition result includes a wake-up word, obtaining a decoding graph of the audio data, the decoding graph including a jump identification sequence of a decoding path corresponding to the audio data, the jump identification sequence being used to indicate a change in phonemes between adjacent audio frames in the audio data;

[0007] Determine a target jump identifier from the jump identifier sequence, wherein the target jump identifier satisfies a target condition, wherein the target condition indicates that the jump identifier in the jump identifier sequence belongs to a jump identifier corresponding to a wake-up phoneme sequence of a wake-up word;

[0008] Based on the target jump identifier, determine the first endpoint of the wake-up audio data corresponding to the wake-up word.

[0009] In a possible implementation manner, the decoding graph includes multiple decoding paths corresponding to the audio data and jump identification sequences of the multiple decoding paths;

[0010] The step of determining a target jump identifier from the jump identifier sequence comprises:

[0011] Based on the decoding parameters of the multiple decoding paths, selecting a decoding path that meets the parameter condition from the multiple decoding paths;

[0012] The target jump identifier is determined from the jump identifier sequence of the selected decoding path.

[0013] In another possible implementation manner, the selecting, based on the decoding parameters of the multiple decoding paths, a decoding path that satisfies a parameter condition from the multiple decoding paths includes:

[0014] Based on the decoding parameters of the multiple decoding paths, selecting a decoding path with the largest decoding parameter from the multiple decoding paths; or,

[0015] Based on the decoding parameters of the multiple decoding paths, selecting a decoding path whose decoding parameter exceeds a parameter threshold from the multiple decoding paths; or,

[0016] Based on the decoding parameters of the multiple decoding paths, a first target number of decoding paths are selected from the multiple decoding paths, and the decoding parameters of the first target number of decoding paths are greater than the decoding parameters of other decoding paths in the multiple decoding paths except the first target number of decoding paths.

[0017] In another possible implementation manner, the recognition result further includes an index of a decoding path, and the index indicates that the processing result is obtained by decoding the decoding path;

[0018] The step of determining a target jump identifier from the jump identifier sequence comprises:

[0019] Based on the index, determining a corresponding decoding path;

[0020] The target jump identifier is determined from the jump identifier sequence of the determined decoding path.

[0021] In another possible implementation manner, the target jump identifier is a jump identifier that meets the target condition and is queried from the jump identifier sequence, and determining the target jump identifier from the jump identifier sequence includes:

[0022] querying the jump identifiers in the jump identifier sequence in sequence;

[0023] When the queried jump identifier does not satisfy the target condition, the next jump identifier is queried continuously until a jump identifier satisfying the target condition is found from the jump identifier sequence.

[0024] In another possible implementation, sequentially querying the jump identifiers in the jump identifier sequence includes:

[0025] In the order of the decoding path from front to back, sequentially query the jump identifiers in the jump identifier sequence of the decoding path; or,

[0026] According to the order of the decoding path from back to front, the jump identifiers in the jump identifier sequence of the decoding path are queried in sequence.

[0027] In another possible implementation, the target jump identifier is a plurality of consecutive jump identifiers that meet the target condition and are queried from the jump identifier sequence, and sequentially querying the jump identifiers in the jump identifier sequence includes:

[0028] sequentially querying a plurality of continuous jump identifiers in the jump identifier sequence;

[0029] When the queried jump identifier does not meet the target condition, continue to query the next jump identifier until a jump identifier meeting the target condition is queried from the jump identifier sequence, including:

[0030] In the case that the multiple continuous jump identifiers queried do not meet the target condition, continue to query the next multiple continuous jump identifiers until multiple continuous jump identifiers that meet the target condition are queried from the jump identifier sequence, and the target condition indicates that each jump identifier of the multiple continuous jump identifiers in the jump identifier sequence belongs to the jump identifier corresponding to the wake-up phoneme sequence, and the jump path represented by the multiple continuous jump identifiers includes all or part of the jump path of the wake-up phoneme sequence.

[0031] In another possible implementation, determining the first endpoint of the wake-up audio data corresponding to the wake-up word based on the target jump identifier includes:

[0032] Based on the jump path represented by each target jump identifier, determine a first endpoint jump identifier from the multiple target jump identifiers, wherein the jump path represented by the first endpoint jump identifier includes the first phoneme in the wake-up phoneme sequence;

[0033] The audio data corresponding to the first endpoint jump identifier is determined as the first endpoint of the wake-up audio data.

[0034] In another possible implementation, the target jump identifier is a jump identifier whose corresponding phoneme queried from the jump identifier sequence belongs to the wake-up phoneme sequence, and when the queried jump identifier does not meet the target condition, continue to query the next jump identifier until a jump identifier that meets the target condition is queried from the jump identifier sequence, including:

[0035] Obtaining the mapping relationship between the queried jump identifier and the phoneme;

[0036] Based on the mapping relationship, determining the phoneme corresponding to the jump identifier;

[0037] In the case that the determined phoneme does not belong to the wake-up phoneme sequence, the next jump identifier is continuously queried until a jump identifier corresponding to the phoneme belonging to the wake-up phoneme sequence is found from the jump identifier sequence.

[0038] In another possible implementation, the target jump identifier is a jump identifier that is the same as any jump identifier in the jump identifier set corresponding to the wake-up phoneme sequence and is queried from the jump identifier sequence; when the queried jump identifier does not meet the target condition, continue to query the next jump identifier until a jump identifier that meets the target condition is queried from the jump identifier sequence, including:

[0039] Acquire a jump identifier set corresponding to the wake-up phoneme sequence, where the jump identifiers in the jump identifier set are used to indicate jump paths of adjacent phonemes in the wake-up phoneme sequence;

[0040] When a jump identifier different from any jump identifier in the jump identifier set is queried, the next jump identifier is queried continuously until a jump identifier identical to any jump identifier in the jump identifier set is queried from the jump identifier sequence.

[0041] In another possible implementation, the method further includes:

[0042] When the queried jump identifier does not satisfy the target condition, the audio data corresponding to the jump identifier is discarded from the audio data.

[0043] In another possible implementation, determining the first endpoint of the wake-up audio data corresponding to the wake-up word based on the target jump identifier includes:

[0044] The audio data corresponding to the target jump identifier is determined as the first endpoint of the wake-up audio data.

[0045] In another possible implementation, the receiving input audio data, performing recognition processing on the audio data, and outputting a recognition result includes:

[0046] receiving the audio data, wherein the audio data comprises a plurality of audio frames;

[0047] Whenever the received amount of audio frames in the audio data reaches a second target amount, performing recognition processing on the received audio data to obtain a recognition result of the received audio data;

[0048] The recognition result is outputted.

[0049] In another aspect, an audio data processing device is provided, the device comprising:

[0050] A processing module, used to receive input audio data, perform recognition processing on the audio data, and output a recognition result;

[0051] an acquisition module, configured to acquire, when the recognition result includes a wake-up word, a decoding graph of the audio data, wherein the decoding graph includes a jump identification sequence of a decoding path corresponding to the audio data, and the jump identification sequence is used to indicate a change in phonemes between adjacent audio frames in the audio data;

[0052] A first determination module is used to determine a target jump identifier from the jump identifier sequence, wherein the target jump identifier satisfies a target condition, wherein the target condition indicates that the jump identifier in the jump identifier sequence belongs to a jump identifier corresponding to a wake-up phoneme sequence of a wake-up word;

[0053] The second determination module is used to determine the first endpoint of the wake-up audio data corresponding to the wake-up word based on the target jump identifier.

[0054] In a possible implementation, the decoding graph includes multiple decoding paths corresponding to the audio data and jump identification sequences of the multiple decoding paths; and the first determining module includes:

[0055] A selection unit, configured to select a decoding path satisfying a parameter condition from the multiple decoding paths based on the decoding parameters of the multiple decoding paths;

[0056] The determination unit is used to determine the target jump identifier from the jump identifier sequence of the selected decoding path.

[0057] In another possible implementation, the selection unit is configured to select, based on the decoding parameters of the multiple decoding paths, a decoding path with a maximum decoding parameter from the multiple decoding paths; or

[0058] The selection unit is configured to select, based on the decoding parameters of the multiple decoding paths, a decoding path whose decoding parameter exceeds a parameter threshold from the multiple decoding paths; or

[0059] The selection unit is used to select a first target number of decoding paths from the multiple decoding paths based on the decoding parameters of the multiple decoding paths, and the decoding parameters of the first target number of decoding paths are greater than the decoding parameters of other decoding paths in the multiple decoding paths except the first target number of decoding paths.

[0060] In another possible implementation manner, the recognition result further includes an index of a decoding path, and the index indicates that the processing result is obtained by decoding the decoding path;

[0061] The first determination module is used to determine the corresponding decoding path based on the index; and determine the target jump identifier from the jump identifier sequence of the determined decoding path.

[0062] In another possible implementation, the target jump identifier is a jump identifier that satisfies the target condition and is queried from the jump identifier sequence, and the first determination module is used to query the jump identifiers in the jump identifier sequence in sequence; if the queried jump identifier does not satisfy the target condition, continue to query the next jump identifier until a jump identifier that satisfies the target condition is queried from the jump identifier sequence.

[0063] In another possible implementation, the first determining module is configured to query the jump identifiers in the jump identifier sequence of the decoding path in sequence from front to back of the decoding path; or

[0064] The first determining module is used to sequentially query the jump identifiers in the jump identifier sequence of the decoding path according to the order of the decoding path from back to front.

[0065] In another possible implementation, the target jump identifier is a plurality of consecutive jump identifiers that meet the target condition and are queried from the jump identifier sequence;

[0066] The first determination module is used to query multiple continuous jump identifiers in the jump identifier sequence in sequence; when the queried multiple continuous jump identifiers do not meet the target condition, continue to query the next multiple continuous jump identifiers until multiple continuous jump identifiers that meet the target condition are queried from the jump identifier sequence, and the target condition indicates that each jump identifier of the multiple continuous jump identifiers in the jump identifier sequence belongs to the jump identifier corresponding to the wake-up phoneme sequence, and the jump path represented by the multiple continuous jump identifiers includes all or part of the jump path of the wake-up phoneme sequence.

[0067] In another possible implementation, the second determination module is used to determine the jump path represented by the target jump identifier, and the first endpoint jump identifier is determined from the target jump identifier, and the jump path represented by the first endpoint jump identifier includes the first phoneme in the wake-up phoneme sequence; the audio data corresponding to the first endpoint jump identifier is determined as the first endpoint of the wake-up audio data.

[0068] In another possible implementation, the target jump identifier is a jump identifier whose corresponding phoneme queried from the jump identifier sequence belongs to the wake-up phoneme sequence, and the first determination module is used to obtain a mapping relationship between the queried jump identifier and the phoneme; based on the mapping relationship, determine the phoneme corresponding to the jump identifier; if the determined phoneme does not belong to the wake-up phoneme sequence, continue to query the next jump identifier until a jump identifier whose corresponding phoneme belongs to the wake-up phoneme sequence is queried from the jump identifier sequence.

[0069] In another possible implementation, the target jump identifier is a jump identifier that is identical to any jump identifier in the jump identifier set corresponding to the wake-up phoneme sequence and is queried from the jump identifier sequence; the first determination module is used to obtain the jump identifier set corresponding to the wake-up phoneme sequence, and the jump identifiers in the jump identifier set are used to represent the jump path of adjacent phonemes in the wake-up phoneme sequence; when a jump identifier different from any jump identifier in the jump identifier set is queried, continue to query the next jump identifier until a jump identifier that is identical to any jump identifier in the jump identifier set is queried from the jump identifier sequence.

[0070] In another possible implementation, the device further includes:

[0071] The discarding module is used to discard the audio data corresponding to the jump identifier from the audio data when the queried jump identifier does not meet the target condition.

[0072] In another possible implementation, the second determining module is configured to determine the audio data corresponding to the target jump identifier as the first endpoint of the wake-up audio data.

[0073] In another possible implementation, the processing module includes:

[0074] A receiving unit, configured to receive the audio data, wherein the audio data includes a plurality of audio frames;

[0075] a processing unit, configured to perform recognition processing on the received audio data to obtain a recognition result of the received audio data whenever the received amount of audio frames in the audio data reaches a second target amount;

[0076] An output unit is used to output the recognition result.

[0077] On the other hand, an electronic device is provided, comprising one or more processors and one or more memories, wherein at least one program code is stored in the one or more memories, and the at least one program code is loaded and executed by the one or more processors to implement the audio data processing method as described in any of the above implementation methods.

[0078] On the other hand, a computer-readable storage medium is provided, wherein at least one program code is stored in the computer-readable storage medium, and the at least one program code is loaded and executed by a processor to implement the audio data processing method as described in any of the above implementations.

[0079] On the other hand, a computer program product is provided, the computer program product comprising at least one program code, the at least one program code being loaded and executed by a processor to implement the audio data processing method as described in any of the above implementations.

[0080] The beneficial effects of the technical solution provided by the embodiments of the present application include at least:

[0081] An embodiment of the present application provides an audio data processing method that can find the starting endpoint of the wake-up audio based on the decoding path. Since the jump identifier in the decoding path can determine the unique phoneme, the scheme realizes the first endpoint detection at the phoneme level, which can more accurately detect the starting endpoint of the wake-up audio and improve the accuracy of the starting endpoint of the wake-up audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0083] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0084] Figure 2 is a flow chart of a method for processing audio data provided by an embodiment of the present application;

[0085] Figure 3 is a flow chart of a method for processing audio data provided by an embodiment of the present application;

[0086] Figure 4 is a block diagram of an audio data processing device provided in an embodiment of the present application;

[0087] Figure 5 is a block diagram of an audio data processing device provided in an embodiment of the present application;

[0088] Figure 6 It is a block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0089] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0090] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0091] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application. Figure 1 , the implementation environment includes an electronic device 101 and a server 102. The electronic device 101 is installed with a client provided by the server 102, and the user can implement functions such as data transmission and voice interaction between the client and the server 102 on the electronic device 101. The client has at least an audio recognition function, for example, identifying whether the input audio data wakes up the electronic device 101, and the client may also have a voice control function. Among them, the client may be a voice assistant or a voice control application.

[0092] In one possible implementation, the electronic device 101 receives input audio data, recognizes the audio data, finds the first endpoint of the wake-up word after recognizing the wake-up word, and reports the first endpoint to the server 102. In another possible implementation, the electronic device 101 receives input audio data, sends the audio data to the server 102, and the server 102 recognizes the audio data, finds the first endpoint of the wake-up word after recognizing the wake-up word.

[0093] The electronic device 101 may be a computer, a mobile phone, a stereo, an air conditioner, a television or other electronic devices. The server 102 may be a server, or a server cluster consisting of several servers, or a cloud computing service center.

[0094] Figure 2 is a flowchart of an audio data processing method provided in an embodiment of the present application, see Figure 2 , the method comprising:

[0095] 201. Receive input audio data, perform recognition processing on the audio data, and output a recognition result.

[0096] 202. When the recognition result includes a wake-up word, obtain a decoding graph of the audio data, where the decoding graph includes a jump identification sequence of a decoding path corresponding to the audio data, and the jump identification sequence is used to indicate a change in phonemes between adjacent audio frames in the audio data.

[0097] 203. Determine a target jump marker from the jump marker sequence, where the target jump marker satisfies a target condition, where the target condition indicates that the jump marker in the jump marker sequence belongs to a jump marker corresponding to a wake-up phoneme sequence of the wake-up word.

[0098] 204. Determine the first endpoint of the wake-up audio data corresponding to the wake-up word based on the target jump identifier.

[0099] The audio data processing method provided in the embodiment of the present application can find the starting endpoint of the wake-up audio based on the decoding path. Since the jump identifier in the decoding path can determine the unique phoneme, the scheme realizes the starting endpoint detection at the phoneme level, which can more accurately detect the starting endpoint of the wake-up audio and improve the accuracy of the starting endpoint of the wake-up audio.

[0100] In a possible implementation manner, the decoding graph includes multiple decoding paths corresponding to the audio data and jump identification sequences of the multiple decoding paths;

[0101] The step of determining a target jump identifier from the jump identifier sequence comprises:

[0102] Based on the decoding parameters of the multiple decoding paths, selecting a decoding path that meets the parameter condition from the multiple decoding paths;

[0103] The target jump identifier is determined from the jump identifier sequence of the selected decoding path.

[0104] In another possible implementation manner, the selecting, based on the decoding parameters of the multiple decoding paths, a decoding path that satisfies a parameter condition from the multiple decoding paths includes:

[0105] Based on the decoding parameters of the multiple decoding paths, selecting a decoding path with the largest decoding parameter from the multiple decoding paths; or,

[0106] Based on the decoding parameters of the multiple decoding paths, selecting a decoding path whose decoding parameter exceeds a parameter threshold from the multiple decoding paths; or,

[0107] Based on the decoding parameters of the multiple decoding paths, a first target number of decoding paths are selected from the multiple decoding paths, and the decoding parameters of the first target number of decoding paths are greater than the decoding parameters of other decoding paths in the multiple decoding paths except the first target number of decoding paths.

[0108] In another possible implementation manner, the recognition result further includes an index of a decoding path, and the index indicates that the processing result is obtained by decoding the decoding path;

[0109] The step of determining a target jump identifier from the jump identifier sequence comprises:

[0110] Based on the index, determining a corresponding decoding path;

[0111] The target jump identifier is determined from the jump identifier sequence of the determined decoding path.

[0112] In another possible implementation manner, the target jump identifier is a jump identifier that meets the target condition and is queried from the jump identifier sequence, and determining the target jump identifier from the jump identifier sequence includes:

[0113] querying the jump identifiers in the jump identifier sequence in sequence;

[0114] When the queried jump identifier does not satisfy the target condition, the next jump identifier is queried continuously until a jump identifier satisfying the target condition is found from the jump identifier sequence.

[0115] In another possible implementation, sequentially querying the jump identifiers in the jump identifier sequence includes:

[0116] In the order of the decoding path from front to back, sequentially query the jump identifiers in the jump identifier sequence of the decoding path; or,

[0117] According to the order of the decoding path from back to front, the jump identifiers in the jump identifier sequence of the decoding path are queried in sequence.

[0118] In another possible implementation, the target jump identifier is a plurality of consecutive jump identifiers that meet the target condition and are queried from the jump identifier sequence, and sequentially querying the jump identifiers in the jump identifier sequence includes:

[0119] sequentially querying a plurality of continuous jump identifiers in the jump identifier sequence;

[0120] When the queried jump identifier does not meet the target condition, continue to query the next jump identifier until a jump identifier meeting the target condition is queried from the jump identifier sequence, including:

[0121] In the case that the multiple continuous jump identifiers queried do not meet the target condition, continue to query the next multiple continuous jump identifiers until multiple continuous jump identifiers that meet the target condition are queried from the jump identifier sequence, and the target condition indicates that each jump identifier of the multiple continuous jump identifiers in the jump identifier sequence belongs to the jump identifier corresponding to the wake-up phoneme sequence, and the jump path represented by the multiple continuous jump identifiers includes all or part of the jump path of the wake-up phoneme sequence.

[0122] In another possible implementation, determining the first endpoint of the wake-up audio data corresponding to the wake-up word based on the target jump identifier includes:

[0123] Based on the jump path represented by the target jump identifier, determining a first endpoint jump identifier from the target jump identifier, wherein the jump path represented by the first endpoint jump identifier includes the first phoneme in the wake-up phoneme sequence;

[0124] The audio data corresponding to the first endpoint jump identifier is determined as the first endpoint of the wake-up audio data.

[0125] In another possible implementation, the target jump identifier is a jump identifier whose corresponding phoneme queried from the jump identifier sequence belongs to the wake-up phoneme sequence, and when the queried jump identifier does not meet the target condition, continue to query the next jump identifier until a jump identifier that meets the target condition is queried from the jump identifier sequence, including:

[0126] Obtaining the mapping relationship between the queried jump identifier and the phoneme;

[0127] Based on the mapping relationship, determining the phoneme corresponding to the jump identifier;

[0128] In the case that the determined phoneme does not belong to the wake-up phoneme sequence, the next jump identifier is continuously queried until a jump identifier corresponding to the phoneme belonging to the wake-up phoneme sequence is found from the jump identifier sequence.

[0129] In another possible implementation, the target jump identifier is a jump identifier that is the same as any jump identifier in the jump identifier set corresponding to the wake-up phoneme sequence and is queried from the jump identifier sequence; when the queried jump identifier does not meet the target condition, continue to query the next jump identifier until a jump identifier that meets the target condition is queried from the jump identifier sequence, including:

[0130] Acquire a jump identifier set corresponding to the wake-up phoneme sequence, where the jump identifiers in the jump identifier set are used to indicate jump paths of adjacent phonemes in the wake-up phoneme sequence;

[0131] When a jump identifier different from any jump identifier in the jump identifier set is queried, the next jump identifier is queried continuously until a jump identifier identical to any jump identifier in the jump identifier set is queried from the jump identifier sequence.

[0132] In another possible implementation, the method further includes:

[0133] When the queried jump identifier does not satisfy the target condition, the audio data corresponding to the jump identifier is discarded from the audio data.

[0134] In another possible implementation, determining the first endpoint of the wake-up audio data corresponding to the wake-up word based on the target jump identifier includes:

[0135] The audio data corresponding to the target jump identifier is determined as the first endpoint of the wake-up audio data.

[0136] In another possible implementation, the receiving input audio data, performing recognition processing on the audio data, and outputting a recognition result includes:

[0137] receiving the audio data, wherein the audio data comprises a plurality of audio frames;

[0138] Whenever the received amount of audio frames in the audio data reaches a second target amount, performing recognition processing on the received audio data to obtain a recognition result of the received audio data;

[0139] The recognition result is outputted.

[0140] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0141] Figure 3 is a flow chart of an audio data processing method provided in an embodiment of the present application. The present application embodiment is illustrated by taking an electronic device as an example. Figure 3 , the method comprising:

[0142] 301. An electronic device receives input audio data, performs recognition processing on the audio data, and outputs a recognition result.

[0143] In the embodiment of the present application, the audio data can be audio data collected by the electronic device, or can be audio data obtained from other devices such as a server, or can be audio data stored locally. The embodiment of the present application does not limit the audio data. Optionally, the electronic device has an audio collection function, and the audio data is audio data collected by the electronic device in real time.

[0144] The recognition result includes text data, which represents the content expressed by the human voice in the audio data. Therefore, the electronic device processes the audio data and can recognize the sentences spoken by the human voice in the audio data. For example, when a user says "It's a nice day today" to the electronic device, the electronic device obtains the audio data through the audio acquisition function, recognizes and processes the audio data, and outputs the recognition result "It's a nice day today".

[0145] In the embodiment of the present application, the process of the electronic device identifying and processing the audio data is not limited, and the process of the electronic device identifying and processing the audio data is only exemplified in the following manner.

[0146] In a possible implementation, the recognition processing of the audio data can be completed by a decoder. Optionally, the electronic device includes a decoder, the electronic device receives input audio data, performs recognition processing on the audio data, and outputs a recognition result, including: the decoder receives input audio data, performs recognition processing on the audio data, and outputs a recognition result.

[0147] The decoder may be a decoder using a chain model. Optionally, the decoder performs recognition processing on the audio data through a generated decoding graph to obtain a recognition result. The generated decoding graph may be a WFST (Weighted Finite State Transducers) decoding graph. Furthermore, the generated decoding graph may be obtained by training the chain model.

[0148] For example, the electronic device divides the audio data into multiple audio frames, extracts features from each audio frame, obtains audio frame features of each audio frame, processes the audio frame features of each audio frame, and obtains the probability distribution corresponding to each audio frame, which represents the probability of the pronunciation action corresponding to the audio frame being each phoneme. The change of phonemes between adjacent audio frames can be represented by a transition id, and the decoder can score the transition path of the phoneme according to the probability of the phoneme itself, the probability of other phonemes before the phoneme, and the probability of other phonemes after the phoneme, and finally decodes based on the path with the highest score to obtain the recognition result.

[0149] In addition, in an embodiment of the present application, when the audio data is recognized and processed, the audio data may be recognized and processed after the audio data is completely acquired, or the acquired audio data may be recognized and processed in real time during the process of acquiring the audio data. Optionally, the electronic device receives input audio data, recognizes and processes the audio data, and outputs a recognition result, including: receiving audio data, the audio data including multiple audio frames; whenever the received amount of audio frames in the audio data reaches a second target amount, recognizing and processing the received audio data to obtain a recognition result of the received audio data; and outputting the recognition result. Therefore, the embodiment of the present application can process the acquired audio data without waiting for the audio data to be completely received.

[0150] The second target number can be any number, for example, the second target number is 1, 3, 5, 10, etc. The second target number can be an empirical value, or any value set by a technician, or a system default value. The second target number is not limited in the embodiment of the present application.

[0151] In addition, in an embodiment of the present application, the electronic device may identify and process the audio data through one decoder, or may identify and process the audio data through multiple decoders. In one possible implementation, the electronic device identifies and processes the audio data through one decoder and outputs a recognition result. In another possible implementation, the audio data is divided into multiple audio data segments, and the electronic device identifies and processes each audio data segment respectively through the multiple decoders to obtain a recognition result output by each decoder. In order to avoid the wake-up audio being divided into two audio data segments, when the audio data is divided, the division point of the audio data can also be divided into other audio data segments.

[0152] For example, the audio data from 0 seconds to 10 seconds is divided into one audio data segment, and the audio data from 10 seconds to 20 seconds is divided into one audio data segment. In order to avoid the wake-up audio at the 10th second, the audio data from 5 seconds to 15 seconds can also be divided into one audio data segment.

[0153] 302. When the recognition result includes a wake-up word, the electronic device obtains a decoding graph of the audio data, where the decoding graph includes a jump identification sequence of a decoding path corresponding to the audio data, and the jump identification sequence is used to indicate a change in phonemes between adjacent audio frames in the audio data.

[0154] If the recognition result includes the wake-up word, it means that the user has said the wake-up word. Since the audio data may include silent audio, audio corresponding to other sentences, etc. in addition to the wake-up audio corresponding to the wake-up word, in an embodiment of the present application, the electronic device can also detect the starting point of the wake-up audio from the audio data, so as to subsequently extract the wake-up audio from the audio data. Among them, the extracted wake-up audio can be used for storage, transmission, model training, etc., and the embodiment of the present application does not limit the use of the wake-up audio.

[0155] The wake-up word is a word used to wake up a client, a function, or a device. For example, the wake-up word is "Hello", and the electronic device wakes up the voice assistant after recognizing "Hello". It should be noted that the wake-up word can be set by the user or by the system default, and the embodiment of the present application does not limit the wake-up word.

[0156] The decoding graph of the audio data in step 302 is obtained by inputting the relevant information of the audio data into the decoding graph generated by the electronic device. The decoding graph includes nodes and edges connecting the nodes. The nodes are used to represent the states of the phonemes, and one node corresponds to one phoneme state. The edges connecting the nodes represent the jumps between the phoneme states.

[0157] After the electronic device divides the audio data into multiple audio frames, the audio frame features corresponding to each audio frame can be extracted, and the audio frame features can be MFCC (Mel Frequency Cepstrum Coefficient, Mel Frequency Cepstrum Coefficient) features, FBank (Filter Bank, filter bank) features or energy features, etc. The audio frame features of each audio frame are processed to obtain the probability distribution corresponding to each audio frame, and the probability distribution represents the probability of the pronunciation action corresponding to the audio frame being each phoneme. When decoding, the decoder will score the jump path between phonemes based on the probability of each phoneme in the current video frame, the probability of each phoneme in the aforementioned audio frame, and the probability of each phoneme in the subsequent audio frame. The jump path with the highest score can be regarded as the most likely jump path. Therefore, the decoder can decode based on the decoding path with the highest score and output the recognition result.

[0158] The decoding path is any jump path from the start node to the end node of the decoding graph. The jump identifier sequence of the decoding path is the jump identifier corresponding to each edge in the decoding path, which is used to represent the jump path of multiple nodes in the decoding path and is determined based on the phoneme changes between adjacent audio frames in the audio data.

[0159] In the process of decoding the decoding graph to obtain the recognition result, the phoneme state corresponding to each audio frame can be input into the decoding graph, and then the input data can be decoded based on the decoding graph to obtain the jump mark sequence, phoneme sequence, word sequence and sentence in turn.

[0160] 303. The electronic device sequentially queries the jump identifiers in the jump identifier sequence.

[0161] Since the jump identifier can be used to determine a unique phoneme, the embodiment of the present application can use the jump identifier to implement endpoint detection at the phoneme level.

[0162] In one possible implementation, the electronic device sequentially queries the jump identifiers in the jump identifier sequence, including: the electronic device sequentially queries the jump identifiers in the jump identifier sequence of the decoding path in order from the front to the back of the decoding path. In another possible implementation, the electronic device sequentially queries the jump identifiers in the jump identifier sequence, including: the electronic device sequentially queries the jump identifiers in the jump identifier sequence of the decoding path in order from the back to the front of the decoding path.

[0163] In the embodiment of the present application, the wake-up audio start point detection is performed when the output recognition result includes the wake-up word, and the electronic device in the embodiment of the present application can also process the input audio data in real time during the input of the audio data. Therefore, the electronic device can stop processing the audio data when the output recognition result includes the wake-up word, and query the jump identifiers in the jump identifier sequence in sequence from back to front in the decoding path, so as to detect the start point of the wake-up audio more quickly.

[0164] It should be noted that the decoding graph includes multiple decoding paths, and the electronic device may perform the first endpoint detection based on only one decoding path, or may perform the first endpoint detection based on multiple decoding paths. This embodiment of the application does not limit this.

[0165] In a possible implementation, the decoding diagram includes multiple decoding paths corresponding to the audio data and jump identification sequences of the multiple decoding paths. The electronic device selects a decoding path that meets the conditions from the multiple decoding paths according to certain conditions. The electronic device sequentially queries the jump identifications in the jump identification sequence, including: the electronic device selects a decoding path that meets the parameter conditions from the multiple decoding paths based on the decoding parameters of the multiple decoding paths; the electronic device sequentially queries the jump identifications in the jump identification sequence of the decoding path in the selected decoding path.

[0166] Optionally, the electronic device selects a decoding path that meets a parameter condition from the multiple decoding paths based on the decoding parameters of the multiple decoding paths, including: selecting a decoding path with the largest decoding parameter from the multiple decoding paths based on the decoding parameters of the multiple decoding paths; or, selecting a decoding path with a decoding parameter exceeding a parameter threshold from the multiple decoding paths based on the decoding parameters of the multiple decoding paths; or, selecting a first target number of decoding paths from the multiple decoding paths based on the decoding parameters of the multiple decoding paths, the decoding parameters of the first target number of decoding paths being greater than the decoding parameters of other decoding paths in the multiple decoding paths except the first target number of decoding paths.

[0167] Among them, the first target number can be any number, for example, 2, 3, 5, etc. The embodiment of the present application does not limit the first target number. The first target number can be an empirical value, or any value set by a technician, or a default value of an electronic device, etc. The parameter threshold can be any value, for example, 10, 20, 30, etc. The embodiment of the present application does not limit the parameter threshold. The parameter threshold can be an empirical value, or any value set by a technician, or a default value of an electronic device, etc.

[0168] Based on the decoding parameters and parameter conditions of the multiple decoding paths, the electronic device can select a decoding path with a higher score from the multiple decoding paths, thereby ensuring the reliability of the decoding path and further ensuring the reliability of the detected head endpoint.

[0169] In one possible implementation, the recognition result output by the electronic device also includes an index of a decoding path, and the index indicates that the processing result is obtained by decoding the decoding path. The electronic device decodes based on the decoding path to obtain a recognition result, and when the recognition result includes a wake-up word, the electronic device detects the first endpoint based on the decoding path. Optionally, the electronic device sequentially queries the jump identifiers in the jump identifier sequence, including: the electronic device determines the corresponding decoding path based on the index; the electronic device sequentially queries the jump identifiers in the jump identifier sequence of the decoding path in the decoding path.

[0170] 304. When the queried jump identifier does not meet the target condition, the electronic device continues to query the next jump identifier until a jump identifier that meets the target condition is queried from the jump identifier sequence, and the target condition indicates that the jump identifier in the jump identifier sequence belongs to the jump identifier corresponding to the wake-up phoneme sequence of the wake-up word.

[0171] In the embodiment of the present application, after the electronic device queries the jump identifier, if the jump identifier does not meet the target condition, it means that the audio data corresponding to the jump path of the jump identifier must not be the wake-up audio data. Therefore, the next jump identifier can be queried until the jump identifier that meets the target condition is queried, and then the first endpoint of the wake-up audio data is determined based on the jump identifier that meets the target condition. Among them, the jump identifier that meets the target condition can be regarded as the target jump identifier in the jump identifier sequence, and querying the jump identifier that meets the target condition from the jump identifier sequence is to determine the target jump identifier from the jump identifier sequence.

[0172] In the embodiment of the present application, the target condition indicates that the jump identifier in the jump identifier sequence belongs to the jump identifier corresponding to the wake-up phoneme sequence of the wake-up word. Among them, the jump identifier corresponding to the wake-up phoneme sequence of the wake-up word can be the identifier of the jump path of any two or more adjacent phonemes in the wake-up phoneme sequence; it can also be the identifier of the jump path between any phoneme and the first or the first N (N is an integer greater than 1) phonemes of the wake-up phoneme sequence; it can also be the identifier of the jump path between the last or the last N (N is an integer greater than 1) phonemes of the wake-up phoneme sequence and any phoneme.

[0173] In addition, in the embodiment of the present application, the jump identifier in the jump identifier sequence of the decoding path can be a jump identifier corresponding to a single phoneme or a jump identifier corresponding to a triphoneme, which is not limited in the embodiment of the present application. Among them, a triphoneme refers to a combination of three phonemes.

[0174] Since the jump identifier can determine a unique phoneme, the embodiment of the present application can determine the phoneme corresponding to the jump identifier through the jump identifier in the jump identifier sequence, and then determine whether the phoneme is a phoneme in the wake-up phoneme sequence to detect the starting endpoint of the wake-up audio data. In a possible implementation, the target jump identifier is a jump identifier whose corresponding phoneme queried from the jump identifier sequence belongs to the wake-up phoneme sequence. When the queried jump identifier does not meet the target condition, the electronic device continues to query the next jump identifier until a jump identifier that meets the target condition is queried from the jump identifier sequence, including: obtaining a mapping relationship between the queried jump identifier and the phoneme; based on the mapping relationship, determining the phoneme corresponding to the jump identifier; when the determined phoneme does not belong to the wake-up phoneme sequence, continue to query the next jump identifier until a jump identifier whose corresponding phoneme belongs to the wake-up phoneme sequence is queried from the jump identifier sequence.

[0175] It should be noted that when the user wakes up with the wake-up word, he or she rarely says other sentences except the wake-up word. Therefore, in one possible implementation, after the electronic device queries the jump identifier of the corresponding phoneme belonging to the wake-up phoneme sequence, it can be considered that the wake-up audio data has been found. In another possible implementation, in order to more accurately detect the starting endpoint of the wake-up audio data, after the electronic device queries the jump identifier of the corresponding phoneme belonging to the wake-up phoneme sequence, it will continue to query until it finds multiple continuous jump identifiers of the corresponding phoneme as the wake-up phoneme sequence, and then determines the starting endpoint of the wake-up audio data based on the multiple continuous jump identifiers.

[0176] In addition, the electronic device can also obtain a jump identifier set corresponding to the wake-up phoneme sequence, and compare the queried jump identifier with each jump identifier in the jump identifier set. If the queried jump identifier is the same as any jump identifier in the jump identifier set, it means that the jump path represented by the queried jump identifier includes the wake-up phoneme, and the jump path represented by the queried jump identifier includes the wake-up phoneme, and the electronic device has found audio data related to the wake-up audio data. In another possible implementation, the target jump identifier is a jump identifier that is identical to any jump identifier in a jump identifier set corresponding to the wake-up phoneme sequence and is queried from a jump identifier sequence; when the queried jump identifier does not meet the target condition, the electronic device continues to query the next jump identifier until a jump identifier that meets the target condition is queried, including: obtaining a jump identifier set corresponding to the wake-up phoneme sequence, the jump identifiers in the jump identifier set being used to represent the jump path of adjacent phonemes in the wake-up phoneme sequence; when a jump identifier that is different from any jump identifier in the jump identifier set is queried, continuing to query the next jump identifier until a jump identifier that is identical to any jump identifier in the jump identifier set is queried from the jump identifier sequence.

[0177] Among them, the jump identifier set can be obtained by the electronic device querying the mapping relationship between the jump identifier and the phoneme, or can be input by a technician, and the embodiment of the present application does not limit this.

[0178] It should be noted that when the user wakes up with the wake-up word, he or she rarely says other sentences except the wake-up word. Therefore, in one possible implementation, after the electronic device queries for a jump identifier that is identical to any jump identifier in the jump identifier set, it can be considered that the wake-up audio data has been found. In another possible implementation, in order to more accurately detect the starting endpoint of the wake-up audio data, after the electronic device queries for a jump identifier that is identical to any jump identifier in the jump identifier set, it will continue to query until multiple consecutive jump identifiers that are identical to the jump identifiers in the jump identifier set are found.

[0179] 305. The electronic device determines the first endpoint of the wake-up audio data corresponding to the wake-up word based on the target jump identifier, where the target jump identifier is a jump identifier that meets the target condition.

[0180] In a possible implementation, the electronic device determines the starting endpoint of the wake-up audio data corresponding to the wake-up word based on the target jump identifier, including: determining the audio data corresponding to the target jump identifier as the starting endpoint of the wake-up audio data.

[0181] Optionally, the audio data corresponding to the target jump identifier is an audio frame, and determining the audio data corresponding to the target jump identifier as the starting endpoint of the wake-up audio data is to determine the audio frame corresponding to the target jump identifier as the first frame of the wake-up audio data. Optionally, the starting endpoint of the wake-up audio data is represented by a time point. The electronic device determines the starting endpoint of the wake-up audio data corresponding to the wake-up word based on the target jump identifier, including: the electronic device determines the starting time point of the audio data corresponding to the target jump identifier as the starting endpoint of the wake-up audio data.

[0182] It should be noted that the above steps 303 to 305 are only an example of the detection process of the first endpoint, taking the target jump identifier as one, and in another embodiment, the target jump identifier is multiple. In a possible implementation, the electronic device sequentially queries the jump identifiers in the jump identifier sequence, including: sequentially querying multiple continuous jump identifiers in the jump identifier sequence.

[0183] When the electronic device sequentially queries multiple consecutive jump identifiers in the jump identifier sequence, it may move back one jump identifier each time, or may move back multiple jump identifiers each time. For example, when the electronic device queries the 1st to 5th jump identifiers in the jump identifier sequence, if the 1st to 5th jump identifiers do not meet the target condition, the electronic device may query the 2nd to 6th jump identifiers, or may query the 6th to 10th jump identifiers.

[0184] In one possible implementation, the target jump identifier is a plurality of consecutive jump identifiers that meet the target condition and are queried in a jump identifier sequence; when the queried jump identifier does not meet the target condition, the electronic device continues to query the next jump identifier until a jump identifier that meets the target condition is queried from the jump identifier sequence, including: when the queried multiple consecutive jump identifiers do not meet the target condition, continue to query the next multiple consecutive jump identifiers until multiple consecutive jump identifiers that meet the target condition are queried from the jump identifier sequence, the target condition indicating that each of the multiple consecutive jump identifiers in the jump identifier sequence belongs to the jump identifier corresponding to the wake-up phoneme sequence, and the multiple consecutive jump identifiers represent the jump path of the wake-up phoneme sequence.

[0185] Among them, the target condition indicates that multiple continuous jump identifiers in the jump identifier sequence represent the jump path of the wake-up phoneme sequence, which can be: the jump path represented by multiple continuous jumps in the jump identifier sequence contains the complete jump path of the wake-up phoneme sequence; it can also be: the jump path represented by multiple continuous jumps in the jump identifier sequence includes a partial jump path of the wake-up phoneme sequence.

[0186] For example, the wake-up phoneme sequence is "nihao", and the jump path represented by the multiple target jump identifiers is "h→ai→n→i→h", then it is considered that the multiple target jump identifiers meet the target condition.

[0187] In one possible implementation, the electronic device determines the starting endpoint of the wake-up audio data corresponding to the wake-up word based on the target jump identifier, including: the electronic device determines the starting endpoint jump identifier from the target jump identifier based on the jump path represented by the target jump identifier, the jump path represented by the starting endpoint jump identifier includes the first phoneme in the wake-up phoneme sequence; and determines the audio data corresponding to the starting endpoint jump identifier as the starting endpoint of the wake-up audio data.

[0188] Optionally, the audio data corresponding to the first endpoint jump identifier is an audio frame, and determining the audio data corresponding to the first endpoint jump identifier as the first endpoint of the wake-up audio data is to determine the audio frame corresponding to the first endpoint jump identifier as the first frame of the wake-up audio data. Optionally, the first endpoint of the wake-up audio data is represented by a time point. The electronic device determines the audio data corresponding to the first endpoint jump identifier as the first endpoint of the wake-up audio data, including: the electronic device determines the starting time point of the audio data corresponding to the first endpoint jump identifier as the first endpoint of the wake-up audio data.

[0189] It should be noted that in an embodiment of the present application, in a possible implementation, the audio data processing method further includes: when the queried jump identifier does not meet the target condition, discarding the audio data corresponding to the jump identifier from the audio data. In this way, other audio data before the wake-up audio data in the audio data can be discarded, so as to obtain the wake-up audio data later.

[0190] Another point that needs to be explained is that the embodiment of the present application only takes "the electronic device queries the jump identifiers in the jump identifier sequence in sequence, and when the queried jump identifier does not meet the target condition, continues to query the next jump identifier until the target jump identifier that meets the target condition is queried" as an example to exemplify "determining the target jump identifier from the jump identifier sequence".

[0191] In another embodiment, after obtaining the decoding diagram of the audio data, the electronic device executes the steps of "determining a target jump identifier from a jump identifier sequence; and determining the starting endpoint of the wake-up audio data corresponding to the wake-up word based on the target jump identifier."

[0192] It should be noted that the decoding graph includes multiple decoding paths, and the electronic device may perform the first endpoint detection based on only one decoding path, or may perform the first endpoint detection based on multiple decoding paths. This embodiment of the application does not limit this.

[0193] In one possible implementation, the decoding diagram includes multiple decoding paths corresponding to the audio data and a jump identifier sequence of the multiple decoding paths; determining a target jump identifier from the jump identifier sequence includes: based on decoding parameters of the multiple decoding paths, selecting a decoding path that meets the parameter conditions from the multiple decoding paths; and determining the target jump identifier from the jump identifier sequence of the selected decoding path.

[0194] Based on the decoding parameters and parameter conditions of the multiple decoding paths, the electronic device can select a decoding path with a higher score from the multiple decoding paths, thereby ensuring the reliability of the decoding path and further ensuring the reliability of the detected head endpoint.

[0195] Optionally, based on the decoding parameters of the multiple decoding paths, a decoding path that meets the parameter condition is selected from the multiple decoding paths, including: based on the decoding parameters of the multiple decoding paths, selecting a decoding path with the largest decoding parameter from the multiple decoding paths; or, based on the decoding parameters of the multiple decoding paths, selecting a decoding path with decoding parameters exceeding a parameter threshold from the multiple decoding paths; or, based on the decoding parameters of the multiple decoding paths, selecting a first target number of decoding paths from the multiple decoding paths, the decoding parameters of the first target number of decoding paths being greater than the decoding parameters of other decoding paths in the multiple decoding paths except the first target number of decoding paths.

[0196] In another possible implementation, the recognition result output by the electronic device also includes an index of a decoding path, and the index indicates that the processing result is obtained by decoding the decoding path. The electronic device decodes based on the decoding path to obtain a recognition result, and when the recognition result includes a wake-up word, the electronic device detects the first endpoint based on the decoding path. Optionally, the recognition result also includes an index of a decoding path, and the index indicates that the processing result is obtained by decoding the decoding path; the electronic device determines a target jump identifier from the jump identifier sequence, including: determining a corresponding decoding path based on the index; and determining the target jump identifier from the jump identifier sequence of the determined decoding path.

[0197] An embodiment of the present application provides an audio data processing method that can find the starting endpoint of the wake-up audio based on the decoding path. Since the jump identifier in the decoding path can determine the unique phoneme, the scheme realizes the first endpoint detection at the phoneme level, which can more accurately detect the starting endpoint of the wake-up audio and improve the accuracy of the starting endpoint of the wake-up audio.

[0198] In addition, in the embodiment of the present application, a single decoder is used to process the audio data, which can not only save more computing resources, but also avoid the wake-up word from being divided into different audio segments, thereby more accurately detecting the starting point of the wake-up audio.

[0199] Figure 4 is a block diagram of an audio data processing device provided in an embodiment of the present application, see Figure 4 , the device comprises:

[0200] The processing module 401 is used to receive input audio data, perform recognition processing on the audio data, and output a recognition result;

[0201] An acquisition module 402 is used to acquire a decoding graph of the audio data when the recognition result includes a wake-up word, the decoding graph including a jump identification sequence of a decoding path corresponding to the audio data, and the jump identification sequence is used to indicate a change in phonemes between adjacent audio frames in the audio data;

[0202] A first determination module 403 is used to determine a target jump identifier from the jump identifier sequence, wherein the target jump identifier satisfies a target condition, wherein the target condition indicates that the jump identifier in the jump identifier sequence belongs to a jump identifier corresponding to a wake-up phoneme sequence of a wake-up word;

[0203] The second determination module 404 is used to determine the first endpoint of the wake-up audio data corresponding to the wake-up word based on the target jump identifier.

[0204] An embodiment of the present application provides an audio data processing device that can find the starting endpoint of the wake-up audio based on the decoding path. Since the jump identifier in the decoding path can determine the unique phoneme, the scheme realizes the first endpoint detection at the phoneme level, and can more accurately detect the starting endpoint of the wake-up audio, thereby improving the accuracy of the starting endpoint of the wake-up audio.

[0205] like Figure 5 As shown, in a possible implementation, the decoding graph includes multiple decoding paths corresponding to the audio data and jump identification sequences of the multiple decoding paths; the first determining module 403 includes:

[0206] A selection unit 4031 is configured to select a decoding path satisfying a parameter condition from the multiple decoding paths based on the decoding parameters of the multiple decoding paths;

[0207] The determination unit 4032 is configured to determine the target jump identifier from the jump identifier sequence of the selected decoding path.

[0208] In another possible implementation, the selection unit 4031 is configured to select, based on the decoding parameters of the multiple decoding paths, a decoding path with a maximum decoding parameter from the multiple decoding paths; or,

[0209] The selection unit 4031 is configured to select, based on the decoding parameters of the multiple decoding paths, a decoding path whose decoding parameter exceeds a parameter threshold from the multiple decoding paths; or

[0210] The selection unit 4031 is used to select a first target number of decoding paths from the multiple decoding paths based on the decoding parameters of the multiple decoding paths, and the decoding parameters of the first target number of decoding paths are greater than the decoding parameters of other decoding paths in the multiple decoding paths except the first target number of decoding paths.

[0211] In another possible implementation manner, the recognition result further includes an index of a decoding path, and the index indicates that the processing result is obtained by decoding the decoding path;

[0212] The first determination module 403 is configured to determine a corresponding decoding path based on the index; and determine the target jump identifier from a jump identifier sequence of the determined decoding path.

[0213] In another possible implementation, the target jump identifier is a jump identifier that satisfies the target condition and is queried from the jump identifier sequence, and the first determination module 403 is used to query the jump identifiers in the jump identifier sequence in sequence; if the queried jump identifier does not satisfy the target condition, continue to query the next jump identifier until a jump identifier that satisfies the target condition is queried from the jump identifier sequence.

[0214] In another possible implementation, the first determining module 403 is configured to query the jump identifiers in the jump identifier sequence of the decoding path in sequence from front to back of the decoding path; or

[0215] The first determining module 403 is used to sequentially query the jump identifiers in the jump identifier sequence of the decoding path according to the order of the decoding path from back to front.

[0216] In another possible implementation, the target jump identifier is a plurality of consecutive jump identifiers that meet the target condition and are queried from the jump identifier sequence;

[0217] The first determination module 403 is used to query multiple continuous jump identifiers in the jump identifier sequence in sequence; when the queried multiple continuous jump identifiers do not meet the target condition, continue to query the next multiple continuous jump identifiers until multiple continuous jump identifiers that meet the target condition are queried from the jump identifier sequence, and the target condition indicates that each jump identifier of the multiple continuous jump identifiers in the jump identifier sequence belongs to the jump identifier corresponding to the wake-up phoneme sequence, and the jump path represented by the multiple continuous jump identifiers includes all or part of the jump path of the wake-up phoneme sequence.

[0218] In another possible implementation, the second determination module 404 is used to determine the first endpoint jump identifier from the target jump identifier based on the jump path represented by the target jump identifier, and the jump path represented by the first endpoint jump identifier includes the first phoneme in the wake-up phoneme sequence; and determine the audio data corresponding to the first endpoint jump identifier as the first endpoint of the wake-up audio data.

[0219] In another possible implementation, the target jump identifier is a jump identifier whose corresponding phoneme queried from the jump identifier sequence belongs to the wake-up phoneme sequence, and the first determination module 403 is used to obtain a mapping relationship between the queried jump identifier and the phoneme; based on the mapping relationship, determine the phoneme corresponding to the jump identifier; when the determined phoneme does not belong to the wake-up phoneme sequence, continue to query the next jump identifier until a jump identifier whose corresponding phoneme belongs to the wake-up phoneme sequence is queried from the jump identifier sequence.

[0220] In another possible implementation, the target jump identifier is a jump identifier that is identical to any jump identifier in the jump identifier set corresponding to the wake-up phoneme sequence and is queried from the jump identifier sequence; the first determination module 403 is used to obtain the jump identifier set corresponding to the wake-up phoneme sequence, and the jump identifiers in the jump identifier set are used to represent the jump path of adjacent phonemes in the wake-up phoneme sequence; when a jump identifier different from any jump identifier in the jump identifier set is queried, continue to query the next jump identifier until a jump identifier that is identical to any jump identifier in the jump identifier set is queried from the jump identifier sequence.

[0221] In another possible implementation, the device further includes:

[0222] The discarding module 405 is configured to discard the audio data corresponding to the jump identifier from the audio data if the queried jump identifier does not meet the target condition.

[0223] In another possible implementation, the second determining module 404 is configured to determine the audio data corresponding to the target jump identifier as the first endpoint of the wake-up audio data.

[0224] In another possible implementation, the processing module 401 includes:

[0225] The receiving unit 4011 is used to receive the audio data, where the audio data includes a plurality of audio frames;

[0226] The processing unit 4012 is configured to perform recognition processing on the received audio data to obtain a recognition result of the received audio data whenever the received amount of audio frames in the audio data reaches a second target amount;

[0227] The output unit 4013 is used to output the recognition result.

[0228] Figure 6 The block diagram of the structure of an electronic device 600 provided by an exemplary embodiment of the present invention is shown. The electronic device 600 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The electronic device 600 may also be called a user device, a portable electronic device, a laptop electronic device, a desktop electronic device, or other names.

[0229] Typically, the electronic device 600 includes a processor 601 and a memory 602 .

[0230] The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 601 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 601 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0231] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction, which is used to be executed by the processor 601 to implement the audio data processing method provided in the method embodiment of the present application.

[0232] In some embodiments, the electronic device 600 may further optionally include: a peripheral device interface 603 and at least one peripheral device. The processor 601, the memory 602 and the peripheral device interface 603 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 603 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 604, a display screen 605, a camera 606, an audio circuit 607, a positioning component 608 and a power supply 609.

[0233] The peripheral device interface 603 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 601 and the memory 602. In some embodiments, the processor 601, the memory 602, and the peripheral device interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 601, the memory 602, and the peripheral device interface 603 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0234] The radio frequency circuit 604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 604 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 604 can communicate with other electronic devices through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 604 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.

[0235] The display screen 605 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos and any combination thereof. When the display screen 605 is a touch display screen, the display screen 605 also has the ability to collect touch signals on the surface or above the surface of the display screen 605. The touch signal can be input to the processor 601 as a control signal for processing. At this time, the display screen 605 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 605 can be one, and the front panel of the electronic device 600 is set; in other embodiments, the display screen 605 can be at least two, which are respectively set on different surfaces of the electronic device 600 or are folded; in some other embodiments, the display screen 605 can be a flexible display screen, which is set on the curved surface or folded surface of the electronic device 600. Even, the display screen 605 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 605 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0236] The camera assembly 606 is used to capture images or videos. Optionally, the camera assembly 606 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the electronic device, and the rear camera is arranged on the back of the electronic device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0237] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 601 for processing, or input them into the radio frequency circuit 604 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the electronic device 600. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 607 may also include a headphone jack.

[0238] Positioning component 608 is used to locate the current geographic location of electronic device 600 to implement navigation or LBS (Location Based Service). Positioning component 608 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Grenas system or the European Union's Galileo system.

[0239] The power supply 609 is used to power various components in the electronic device 600. The power supply 609 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 609 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0240] In some embodiments, the electronic device 600 further includes one or more sensors 610 , including but not limited to: an acceleration sensor 611 , a gyroscope sensor 612 , a pressure sensor 613 , a fingerprint sensor 614 , an optical sensor 615 , and a proximity sensor 616 .

[0241] The acceleration sensor 611 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the electronic device 600. For example, the acceleration sensor 611 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 601 can control the display screen 605 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 611. The acceleration sensor 611 can also be used for collecting game or user motion data.

[0242] The gyro sensor 612 can detect the body direction and rotation angle of the electronic device 600, and the gyro sensor 612 can cooperate with the acceleration sensor 611 to collect the user's 3D actions on the electronic device 600. The processor 601 can implement the following functions based on the data collected by the gyro sensor 612: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0243] The pressure sensor 613 can be set on the side frame of the electronic device 600 and / or the lower layer of the display screen 605. When the pressure sensor 613 is set on the side frame of the electronic device 600, it can detect the user's holding signal of the electronic device 600, and the processor 601 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 613. When the pressure sensor 613 is set on the lower layer of the display screen 605, the processor 601 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 605. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0244] The fingerprint sensor 614 is used to collect the user's fingerprint, and the processor 601 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 614, or the fingerprint sensor 614 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 601 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, paying, and changing settings. The fingerprint sensor 614 can be set on the front, back, or side of the electronic device 600. When a physical button or a manufacturer logo is set on the electronic device 600, the fingerprint sensor 614 can be integrated with the physical button or the manufacturer logo.

[0245] The optical sensor 615 is used to collect the ambient light intensity. In one embodiment, the processor 601 can control the display brightness of the display screen 605 according to the ambient light intensity collected by the optical sensor 615. Specifically, when the ambient light intensity is high, the display brightness of the display screen 605 is increased; when the ambient light intensity is low, the display brightness of the display screen 605 is reduced. In another embodiment, the processor 601 can also dynamically adjust the shooting parameters of the camera assembly 606 according to the ambient light intensity collected by the optical sensor 615.

[0246] The proximity sensor 616, also called a distance sensor, is usually disposed on the front panel of the electronic device 600. The proximity sensor 616 is used to collect the distance between the user and the front of the electronic device 600. In one embodiment, when the proximity sensor 616 detects that the distance between the user and the front of the electronic device 600 is gradually decreasing, the processor 601 controls the display screen 605 to switch from the screen-on state to the screen-off state; when the proximity sensor 616 detects that the distance between the user and the front of the electronic device 600 is gradually increasing, the processor 601 controls the display screen 605 to switch from the screen-off state to the screen-on state.

[0247] Those skilled in the art will understand that Figure 6 The structure shown in the figure does not constitute a limitation on the electronic device 600, and may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0248] An embodiment of the present application also provides a computer-readable storage medium, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the audio data processing method as described in any of the above implementations.

[0249] An embodiment of the present application also provides a computer program product, which includes at least one program code, and the at least one program code is loaded and executed by a processor to implement the audio data processing method as described in any of the above implementations.

[0250] In some embodiments, the computer program involved in the embodiments of the present application may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain system.

[0251] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. An audio data processing method, characterized in that: The method comprises: Receive input audio data, perform recognition processing on the audio data, and output a recognition result; In a case where the recognition result includes a wake-up word, obtaining a decoding graph of the audio data, the decoding graph including a jump identification sequence of a decoding path corresponding to the audio data, the jump identification sequence being used to indicate a change in phonemes between adjacent audio frames in the audio data; Determine a target jump identifier from the jump identifier sequence, wherein the target jump identifier satisfies a target condition, wherein the target condition indicates that the jump identifier in the jump identifier sequence belongs to the jump identifier corresponding to the wake-up phoneme sequence of the wake-up word; Based on the jump path represented by the target jump identifier, determining a first endpoint jump identifier from the target jump identifier, wherein the jump path represented by the first endpoint jump identifier includes the first phoneme in the wake-up phoneme sequence; Determine the audio data corresponding to the first endpoint jump identifier as the first endpoint of the wake-up audio data corresponding to the wake-up word; The step of determining a target jump identifier from the jump identifier sequence includes: The decoding graph includes multiple decoding paths corresponding to the audio data and jump identification sequences of the multiple decoding paths; based on decoding parameters of the multiple decoding paths, a decoding path that meets the parameter conditions is selected from the multiple decoding paths; the target jump identification is determined from the jump identification sequence of the selected decoding path; or, The recognition result also includes an index of a decoding path, wherein the index indicates that the recognition result is obtained by decoding the decoding path; based on the index, the corresponding decoding path is determined; and the target jump identifier is determined from a jump identifier sequence of the determined decoding path.

2. The method according to claim 1, characterized in that: The selecting a decoding path satisfying a parameter condition from the multiple decoding paths based on the decoding parameters of the multiple decoding paths comprises: Based on the decoding parameters of the multiple decoding paths, selecting a decoding path with the largest decoding parameter from the multiple decoding paths; or, Based on the decoding parameters of the multiple decoding paths, selecting a decoding path whose decoding parameter exceeds a parameter threshold from the multiple decoding paths; or, Based on the decoding parameters of the multiple decoding paths, a first target number of decoding paths are selected from the multiple decoding paths, and the decoding parameters of the first target number of decoding paths are greater than the decoding parameters of other decoding paths in the multiple decoding paths except the first target number of decoding paths.

3. The method according to claim 1, characterized in that The target jump identifier is a jump identifier that meets the target condition and is found from the jump identifier sequence. The step of determining the target jump identifier from the jump identifier sequence includes: querying the jump identifiers in the jump identifier sequence in sequence; When the queried jump identifier does not satisfy the target condition, the next jump identifier is queried continuously until a jump identifier satisfying the target condition is found from the jump identifier sequence.

4. The method according to claim 3, characterized in that: The step of sequentially querying the jump identifiers in the jump identifier sequence includes: In the order of the decoding path from front to back, sequentially query the jump identifiers in the jump identifier sequence of the decoding path; or, According to the order of the decoding path from back to front, the jump identifiers in the jump identifier sequence of the decoding path are queried in sequence.

5. The method according to claim 3 or 4, characterized in that: The target jump identifier is a plurality of continuous jump identifiers that meet the target condition and are queried from the jump identifier sequence, and sequentially querying the jump identifiers in the jump identifier sequence includes: sequentially querying a plurality of continuous jump identifiers in the jump identifier sequence; When the queried jump identifier does not meet the target condition, continue to query the next jump identifier until a jump identifier meeting the target condition is queried from the jump identifier sequence, including: In the case that the multiple continuous jump identifiers queried do not meet the target condition, continue to query the next multiple continuous jump identifiers until multiple continuous jump identifiers that meet the target condition are queried from the jump identifier sequence, and the target condition indicates that each jump identifier of the multiple continuous jump identifiers in the jump identifier sequence belongs to the jump identifier corresponding to the wake-up phoneme sequence, and the jump path represented by the multiple continuous jump identifiers includes all or part of the jump path of the wake-up phoneme sequence.

6. The method according to claim 3, characterized in that The target jump identifier is a jump identifier whose corresponding phoneme queried from the jump identifier sequence belongs to the wake-up phoneme sequence, and when the queried jump identifier does not meet the target condition, continue to query the next jump identifier until a jump identifier that meets the target condition is queried from the jump identifier sequence, including: Obtaining the mapping relationship between the queried jump identifier and the phoneme; Based on the mapping relationship, determining the phoneme corresponding to the jump identifier; In the case that the determined phoneme does not belong to the wake-up phoneme sequence, the next jump identifier is continuously queried until a jump identifier corresponding to the phoneme belonging to the wake-up phoneme sequence is found from the jump identifier sequence.

7. The method according to claim 3, characterized in that The target jump identifier is a jump identifier that is the same as any jump identifier in the jump identifier set corresponding to the wake-up phoneme sequence and is found from the jump identifier sequence; When the queried jump identifier does not meet the target condition, continue to query the next jump identifier until a jump identifier meeting the target condition is queried from the jump identifier sequence, including: Acquire a jump identifier set corresponding to the wake-up phoneme sequence, where the jump identifiers in the jump identifier set are used to indicate jump paths of adjacent phonemes in the wake-up phoneme sequence; When a jump identifier different from any jump identifier in the jump identifier set is queried, the next jump identifier is queried continuously until a jump identifier identical to any jump identifier in the jump identifier set is queried from the jump identifier sequence.

8. The method according to claim 3, characterized in that The method further comprises: When the queried jump identifier does not satisfy the target condition, the audio data corresponding to the jump identifier is discarded from the audio data.

9. The method according to claim 1, characterized in that: The method further comprises: The audio data corresponding to the target jump identifier is determined as the first endpoint of the wake-up audio data.

10. The method according to claim 1, characterized in that The receiving of input audio data, performing recognition processing on the audio data, and outputting a recognition result includes: receiving the audio data, wherein the audio data comprises a plurality of audio frames; Whenever the received amount of audio frames in the audio data reaches a second target amount, performing recognition processing on the received audio data to obtain a recognition result of the received audio data; The recognition result is outputted.

11. An audio data processing device, characterized in that: The device comprises: A processing module, used to receive input audio data, perform recognition processing on the audio data, and output a recognition result; an acquisition module, configured to acquire, when the recognition result includes a wake-up word, a decoding graph of the audio data, wherein the decoding graph includes a jump identification sequence of a decoding path corresponding to the audio data, and the jump identification sequence is used to indicate a change in phonemes between adjacent audio frames in the audio data; A first determination module is used to determine a target jump identifier from the jump identifier sequence, wherein the target jump identifier satisfies a target condition, wherein the target condition indicates that the jump identifier in the jump identifier sequence belongs to a jump identifier corresponding to a wake-up phoneme sequence of a wake-up word; A second determination module is used to determine a first endpoint jump identifier from the target jump identifier based on the jump path represented by the target jump identifier, wherein the jump path represented by the first endpoint jump identifier includes the first phoneme in the wake-up phoneme sequence; and determine the audio data corresponding to the first endpoint jump identifier as the first endpoint of the wake-up audio data corresponding to the wake-up word; The first determining module is used to determine a target jump identifier from the jump identifier sequence, including: The decoding graph includes multiple decoding paths corresponding to the audio data and jump identification sequences of the multiple decoding paths; the first determination module includes a selection unit and a determination unit, the selection unit is used to select a decoding path that meets the parameter condition from the multiple decoding paths based on the decoding parameters of the multiple decoding paths; the determination unit is used to determine the target jump identification from the jump identification sequence of the selected decoding path; or, The recognition result also includes an index of a decoding path, wherein the index indicates that the recognition result is obtained by decoding the decoding path; the first determination module is used to determine the corresponding decoding path based on the index; and to determine the target jump identifier from the jump identifier sequence of the determined decoding path.

12. An electronic device, characterized in that: The electronic device comprises: A processor and a memory, wherein at least one program code is stored in the memory, and the at least one program code is loaded and executed by the processor to implement the audio data processing method according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that: The storage medium stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the audio data processing method according to any one of claims 1 to 11.

14. A computer program product, characterized in that The computer program product comprises at least one program code, and the at least one program code is loaded and executed by a processor to implement the audio data processing method according to any one of claims 1 to 11.

Images