Speech detection method, apparatus and storage medium

Abstract

The application relates to the field of speech detection in the field of artificial intelligence technology, in particular to a speech detection method and device and a storage medium, the method comprising the following steps: determining a second text symbol according to a first text symbol and a first audio signal in an audio signal sequence; determining whether the semantic of the second text symbol reaches a rear end point according to the first text symbol; in the case that the semantic of the second text symbol does not reach the rear end point, taking the second text symbol as a new first text symbol, taking an audio signal after the first audio signal in the audio signal sequence as a new first audio signal, and repeatedly executing the steps of determining the second text symbol according to the first text symbol and the first audio signal in the audio signal sequence and the following steps. According to the embodiment of the application, the rear end point of the input audio can be accurately determined based on the semantic, misjudgment can be prevented, the number and deployment process of the model are simplified, and the precision of the model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a speech detection method and device and a storage medium. BACKGROUND

[0002] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. That is, artificial intelligence studies the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0003] Speech detection is an important field of AI. With the increasingly widespread application of speech wake-up and speech recognition technologies in life, speech detection is valued as a necessary front-end processing. The role of speech detection is to detect whether there is speech in the current environment and locate the start and end positions of the speech, so as to separate the speech segment from the noise and send it to the back end for speech recognition, wake-up and other processing. At present, there is a lack of speech detection model that can accurately locate the start and end positions of the speech, especially the end position of the speech, which is easily affected by speech pauses and the like, resulting in the risk of false judgment, and there is an urgent need for a more efficient and more accurate speech detection method. SUMMARY

[0004] Therefore, a speech detection method, device and storage medium are provided.

[0005] In a first aspect, an embodiment of the present application provides a speech detection method, which comprises: determining a second text symbol according to a first text symbol and a first audio signal in an audio signal sequence, the initial value of the first text symbol being an empty character, the second text symbol corresponding to the content of the first audio signal; determining whether the semantics of the second text symbol reaches a rear end point according to the first text symbol, the rear end point indicating the end of speech in the audio signal sequence; in the case where the semantics of the second text symbol does not reach the rear end point, taking the second text symbol as a new first text symbol, taking an audio signal after the first audio signal in the audio signal sequence as a new first audio signal, and repeating the steps of determining a second text symbol according to a first text symbol and a first audio signal in an audio signal sequence and the subsequent steps.

[0006] According to the embodiment of the present application, by determining the second text symbol according to the first text symbol and the first audio signal of the audio signal sequence, determining whether the semantic of the second text symbol reaches the back end point according to the first text symbol, the semantic judgment of whether the text symbol reaches the back end point can be realized, the audio reception is prevented from being ended in advance when the user speaks stutters, the misjudgment is prevented, the detection of the back end point is more accurate, and the user experience is improved. By taking the second text symbol as a new first text symbol, taking the audio signal after the first audio signal in the audio signal sequence as a new first audio signal, and repeatedly performing the steps of determining the second text symbol according to the first text symbol and the first audio signal of the audio signal sequence and the subsequent steps under the condition that the semantic of the second text symbol does not reach the back end point, the back end point detection of the audio does not need to rely on a complex and artificially set multi-model decision process, the speech detection method is more flexible, and the accuracy of the speech detection method is improved.

[0007] According to the first aspect, in a first possible implementation manner of the speech detection method, the method further includes: sequentially detecting whether an audio frame contained in each audio signal in the audio signal sequence reaches a front end point, the front end point indicating that speech starts in the audio signal sequence; when a first audio frame reaching the front end point is detected, determining an audio signal in which the audio frame is located as the first audio signal, and stopping the detection.

[0008] According to the embodiment of the present application, by sequentially detecting whether an audio frame contained in each audio signal in the audio signal sequence reaches a front end point, determining the first audio signal in which the first audio frame reaching the front end point is located as the first audio signal when the first audio frame reaching the front end point is detected, and stopping the detection, the front end point of the audio can be detected on the basis of the detection of the back end point, so that the human voice interval can be more accurately determined, the detection is stopped when the audio signal reaching the front end point is detected, and the calculation amount and the resource consumption can be reduced.

[0009] In a second possible implementation form of the speech detection method according to the first aspect, the method is used for a target detection model, the target detection model comprising a speech behavior detection module, an association network module, a semantic endpoint detection module, an encoder module and a prediction network module, the encoder module being configured to obtain a first feature vector of an audio frame contained in an audio signal in an audio signal sequence; the speech behavior detection module being configured to determine whether the audio frame contained in the audio signal in the audio signal sequence reaches a front endpoint according to the first feature vector; the prediction network module being configured to obtain a second feature vector of a first text symbol; the association network module being configured to obtain the second text symbol according to the first feature vector and the second feature vector; and the semantic endpoint detection module being configured to determine whether a semantic of the second text symbol reaches a back endpoint according to the second feature vector.

[0010] According to the embodiments of the present application, by using the target detection model comprising the speech behavior detection module, the association network module, the semantic endpoint detection module, the encoder module and the prediction network module, the VAD model and the ASR model can be integrated in one model, the front endpoint can be detected while the back endpoint is detected based on the semantic, the model quantity and the deployment process are greatly simplified, a large amount of resources is saved, and the accuracy of speech detection is improved based on the semantic back endpoint.

[0011] In a third possible implementation form of the speech detection method according to the second aspect, the method further comprises: pre-training the prediction network module and the semantic endpoint detection module; and training the target detection model comprising the speech behavior detection module, the association network module, the encoder module, the pre-trained prediction network module and the pre-trained semantic endpoint detection module to obtain a trained target detection model.

[0012] According to the embodiments of the present application, by pre-training the prediction network module and the semantic endpoint detection module, and training the target detection model comprising the speech behavior detection module, the association network module, the encoder module, the pre-trained prediction network module and the pre-trained semantic endpoint detection module, the training process can be more targeted, the back endpoint based on the semantic judgment can be more accurate, the subsequent training model can be better trained by pre-training, and the prediction accuracy of the trained target detection model can be higher.

[0013] In a fourth possible implementation form of the voice detection method according to the second possible implementation form of the first aspect, the pre-training of the prediction network module and the semantic endpoint detection module comprises: inputting a previous text symbol in a text sample into the prediction network module, the prediction network module predicting a current text symbol according to a feature vector of the previous text symbol; inputting the current text symbol into the semantic endpoint detection module to determine whether a semantic of the current text symbol reaches a back endpoint; calculating a first loss value of the prediction network module according to a label of the current text symbol, and calculating a second loss value of the semantic endpoint detection module according to a label of whether the current text symbol reaches the back endpoint, and adjusting parameters of the prediction network module and the semantic endpoint detection module according to the first loss value and the second loss value.

[0014] According to the embodiments of the present application, by inputting a previous text symbol in a text sample into the prediction network module, the prediction network module predicting a current text symbol according to a feature vector of the previous text symbol, inputting the current text symbol into the semantic endpoint detection module to determine whether a semantic of the current text symbol reaches a back endpoint, calculating a first loss value of the prediction network module according to a label of the current text symbol, and calculating a second loss value of the semantic endpoint detection module according to a label of whether the current text symbol reaches the back endpoint, and adjusting parameters of the prediction network module and the semantic endpoint detection module according to the first loss value and the second loss value, the function of the model detecting a back endpoint based on semantics can be more targetedly trained in the pre-training stage, and the accuracy of the model in voice detection is improved. In the pre-training stage, the model is trained by using the text sample which is easy to obtain, without using the audio sample, so that the related resources can be saved, and the accuracy of the model in voice detection based on semantics is further improved.

[0015] According to the fifth possible implementation form of the voice detection method according to the second possible implementation form of the first aspect, the voice behavior detection module stops running when it is determined that the audio frames contained in each audio signal in the audio signal sequence reach a front endpoint according to the first feature vector.

[0016] According to the embodiments of the present application, the running of the voice behavior detection module can be paused after the front endpoint is determined, so as to save resources.

[0017] In a second aspect, an embodiment of the present application provides a speech detection device, comprising: a first determining module configured to determine a second text symbol according to a first text symbol and a first audio signal in an audio signal sequence, wherein an initial value of the first text symbol is an empty character, and the second text symbol corresponds to content of the first audio signal; a second determining module configured to determine whether semantics of the second text symbol reaches a back end point according to the first text symbol, wherein the back end point represents an end of speech in the audio signal sequence; and a third determining module configured to, in a case where the semantics of the second text symbol does not reach the back end point, repeat the steps of determining the second text symbol according to the first text symbol and the first audio signal in the audio signal sequence, and the following steps, by taking the second text symbol as a new first text symbol, and taking an audio signal after the first audio signal in the audio signal sequence as a new first audio signal.

[0018] According to the second aspect, in a first possible implementation manner of the speech detection device, the device further comprises: a detecting module configured to detect whether an audio frame contained in each audio signal in the audio signal sequence reaches a front end point in sequence, wherein the front end point represents a start of speech in the audio signal sequence; and a fourth determining module configured to, when detecting a first audio frame reaching the front end point, determine the audio signal in which the first audio frame is located as the first audio signal, and stop the detection.

[0019] According to the first possible implementation manner of the second aspect, in a second possible implementation manner of the speech detection device, the device is configured to a target detection model, wherein the target detection model comprises a speech behavior detection module, an association network module, a semantic end point detection module, an encoder module, and a prediction network module; the encoder module is configured to obtain a first feature vector of an audio frame contained in an audio signal in the audio signal sequence; the speech behavior detection module is configured to determine whether the audio frame contained in each audio signal in the audio signal sequence reaches a front end point according to the first feature vector; the prediction network module is configured to obtain a second feature vector of the first text symbol; the association network module is configured to obtain the second text symbol according to the first feature vector and the second feature vector; and the semantic end point detection module is configured to determine whether semantics of the second text symbol reaches a back end point according to the second feature vector.

[0020] According to the second possible implementation manner of the second aspect, in a third possible implementation manner of the speech detection device, the device further comprises: a pre-training module configured to pre-train the prediction network module and the semantic end point detection module; and a training module configured to train a target detection model comprising the speech behavior detection module, the association network module, the encoder module, and the pre-trained prediction network module and the pre-trained semantic end point detection module, to obtain a trained target detection model.

[0021] According to a fourth possible implementation of the speech detection apparatus according to the second possible implementation of the second aspect, the pre-training module is configured to: input a previous text symbol in the text sample into the prediction network module, and the prediction network module predicts a current text symbol according to a feature vector of the previous text symbol; input the current text symbol into the semantic endpoint detection module to determine whether the semantic of the current text symbol reaches a rear endpoint; calculate a first loss value of the prediction network module according to a label of the current text symbol, and calculate a second loss value of the semantic endpoint detection module according to a label of whether the current text symbol reaches the rear endpoint; and adjust parameters of the prediction network module and the semantic endpoint detection module according to the first loss value and the second loss value.

[0022] According to a fifth possible implementation of the speech detection apparatus according to the second possible implementation of the second aspect, the speech behavior detection module stops running when it is determined according to the first feature vector that an audio frame contained in each audio signal in the audio signal sequence reaches a front endpoint.

[0023] In a third aspect, an embodiment of the present application provides a speech detection apparatus, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the speech detection method of the first aspect or one or more of the possible implementation manners of the first aspect when executing the instructions.

[0024] In a fourth aspect, an embodiment of the present application provides a non-volatile computer-readable storage medium having stored thereon computer program instructions, which, when executed by a processor, implement the speech detection method of the first aspect or one or more of the possible implementation manners of the first aspect.

[0025] In a fifth aspect, an embodiment of the present application provides a terminal device, which can execute the speech detection method of the first aspect or one or more of the possible implementation manners of the first aspect.

[0026] In a sixth aspect, an embodiment of the present application provides a computer program product, comprising computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, when the computer-readable code is executed in an electronic device, a processor in the electronic device executes the speech detection method of the first aspect or one or more of the possible implementation manners of the first aspect.

[0027] These and other aspects of the application will become more fully understood from the following (a few) embodiment descriptions. BRIEF DESCRIPTION OF DRAWINGS

[0028] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples of the present application, and together with the description, serve to explain the principles of the present application.

[0029] Figure 1 A schematic diagram showing an implementation environment of a method of voice detection according to an embodiment of the present application.

[0030] Figure 2 A flowchart showing a method of training a model of a pre-training stage according to an embodiment of the present application.

[0031] Figure 3 A schematic diagram showing input and output sequences of a pre-training stage according to an embodiment of the present application.

[0032] Figure 4 A flowchart showing a method of training a model of a whole training stage according to an embodiment of the present application.

[0033] Figure 5 A schematic diagram showing input and output sequences of a whole training stage according to an embodiment of the present application.

[0034] Figure 6 A flowchart showing a model prediction stage according to an embodiment of the present application.

[0035] Figure 7 A flowchart showing a method of voice detection according to an embodiment of the present application.

[0036] Figure 8 A flowchart showing a method of voice detection according to an embodiment of the present application.

[0037] Figure 9 A flowchart showing a method of voice detection according to an embodiment of the present application.

[0038] Figure 10 A flowchart showing a method of voice detection according to an embodiment of the present application.

[0039] Figure 11 A structural diagram of a voice detection apparatus according to an embodiment of the present application.

[0040] Figure 12 A structural diagram of a terminal device according to an embodiment of the present application.

[0041] Figure 13 A structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] Various exemplary embodiments, features, and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numbers in different drawings represent the same or similar elements. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0043] The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0044] In addition, for the purpose of convenience and brevity, detailed descriptions of well-known methods, apparatuses, circuits, and circuit elements are omitted so as not to obscure the concepts of the present application. It should be noted that the use of particular

[0045] In the prior art, the voice activity detection (VAD) model and the automatic speech recognition (ASR) model are usually two independent models, resulting in that the identification of the audio interval between the models needs to rely on the result of text recognition, and the rules for identifying the audio interval need to be manually configured, which is poor in flexibility, slow in recognition, low in accuracy of judgment, and consumes a large amount of resources; in the prior art, there is also an ASR model that fuses the VAD model and the back-end point recognition, but it depends on the annotation of the training data of the audio and cannot identify the front-end point of the audio, and the effect is poor when the audio back-end point is identified based on the text semantics.

[0046] To solve the above technical problems, the present application provides a voice detection method, which can determine whether the user has the intention to continue speaking based on the recognized text semantics, without the need for manual rule setting, is more flexible, can more accurately determine the back-end point of the input audio, prevent misjudgment from ending the recording too early, and can also identify the front-end point of the input audio, more accurately determine the interval of the audio, and simplify the number of models and the deployment process, saving resources.

[0047] Figure 1 A schematic diagram showing an implementation environment of a voice detection method according to an embodiment of the present application is shown. As shown in Figure 1 The implementation environment can include a terminal device and a voice detection platform.

[0048] Referring to Figure 1 , the terminal device can be a vehicle-mounted terminal 101, a smart phone 102, a smart speaker 103, or a robot 104. Of course, Figure 1The several terminal devices shown are merely examples, and the terminal device can also be other electronic devices supporting voice detection functions, such as a netbook, a tablet computer, a notebook computer, a wearable electronic device (such as a smart bracelet, a smart watch, etc.), a TV, a virtual reality device, a sound box, electronic ink, and the like. The type of the terminal device is not limited by the embodiments of the present application. The terminal device can run an application program supporting voice detection. The application program can be a navigation application, a voice assistant, a smart question and answer application, and the like. For example, the terminal device is a terminal device used by a user, and the application program running on the terminal device logs in a user account, which can be pre-registered in the voice detection platform. The terminal device can be connected to the voice detection platform in a wireless connection manner or a wired connection manner. The wireless connection manner refers to a connection manner through a wireless connection such as wifi or Bluetooth.

[0049] The voice detection platform is configured to provide background services for the application program supporting voice detection. For example, the voice detection platform can execute the following method embodiments to obtain a voice detection model (for example, the target detection model of the embodiments of the present application below), and send the model to the terminal device (for example, the model can be exported from the platform through a model format conversion tool to be deployed on the terminal device), so that the terminal device utilizes the model to perform voice detection. The terminal device can accurately identify the front and rear endpoints of the voice in the received audio by utilizing the model, and simultaneously perform voice recognition to realize subsequent applications such as voice control.

[0050] The voice detection platform can include a server 201 and a database 202. The server 201 can be a single server or a cluster composed of multiple servers. The database 202 can be configured to store training data, such as training data containing a large amount of text, audio training data, and the like. The server 201 can access the database 202 to obtain the training data stored in the database 202, and train the model through the training data.

[0051] Those skilled in the art can know that the number of the terminal device, the server, or the database described above can be more or less. For example, the terminal device, the server, or the database described above can be only one, or several tens or hundreds, or more, at this time, although not shown in the figure, other terminal devices, other servers, or other databases can also be included.

[0052] The above exemplary introduces the system architecture, and the following exemplary introduces a method flow of voice detection based on the system architecture provided above.

[0053] The method flow of voice detection can include a model training phase and a model prediction phase. In the following, the model training phase is described first. Figure 2 、 4The example describes the method and process of the model training phase. Figure 6 The example describes the method and process for the model prediction stage.

[0054] The process of training a model can include two stages: pre-training and overall training.

[0055] Figure 2 A flowchart illustrating a training method for a model during the pre-training phase according to an embodiment of this application is shown. Figure 2 As shown, this method can be applied to electronic devices, which can be... Figure 1 The terminal devices in the system architecture shown can also be Figure 1 The speech detection platform in the system architecture shown is, for example, server 201. The training data used in the pre-training phase can be text data, which can include text and annotations indicating whether the semantics of each character (which can be a text symbol) in the text are complete. The model in the pre-training stage includes a prediction network module and a semantic endpoint detection module. The prediction network module can be a twelve-layer transformer structure, and the semantic endpoint detection module can be a two-layer long short-term memory (LSTM) network structure or other structures capable of detecting subsequent endpoints. The prediction network module and the semantic endpoint detection module can also be other structures, which are not limited in this application. y(u-1) can represent the previous text symbol in the input text sequence, y(u) can represent the current text symbol, e(u) can represent whether the current text symbol has reached the subsequent endpoint, and u can represent the sequence number of the text symbol. The prediction network module is used to obtain the high-dimensional feature vector corresponding to the previous text symbol y(u-1) in the text sequence, and to obtain the current text symbol y(u) based on the high-dimensional feature vector. The semantic endpoint detection module is used to determine whether the current text symbol y(u) has reached the end of the sentence (i.e., whether it has reached the subsequent endpoint).

[0056] In the pre-training phase, for a text sequence in the training data, the previous text symbol y(u-1) in the text sequence can be used as the input of the prediction network module, the current text symbol y(u) in the text sequence can be used as the output of the prediction network module, and whether the current text symbol y(u) has reached the endpoint e(u) (for example, e(u) is 1 when it reaches the endpoint, and 0 otherwise) can be used as the output of the semantic endpoint detection module. This allows the trained model to determine whether the semantics of the current text symbol has reached the end of the sentence based on the previous text symbol, that is, whether the current text symbol y(u) is the last text symbol at the end of the sentence. Cross-entropy loss is used to optimize the output of the prediction network module, and binary cross-entropy loss is used to optimize the output of the semantic endpoint detection module, until the loss optimization value of the model in the pre-training phase converges to a predetermined value, thus obtaining the trained model in the pre-training phase.

[0057] The prediction network module and the semantic endpoint detection module can be integrated into the overall model. By training only these two modules during the pre-training phase, training resources can be saved, and targeted training can lead to better performance of the final object detection model. The pre-training phase can utilize only text-based training data, which can be obtained through purchase or web scraping; this application does not impose any restrictions on this. Since text-based data is readily available, pre-training with text-based data can significantly improve the accuracy of the target model's semantic-based speech detection while also conserving resources during the pre-training phase.

[0058] Figure 3 This diagram illustrates the input-output sequence of a pre-training phase according to an embodiment of this application. Figure 3 As shown, the prediction network module predicts each text symbol (e.g., ...) in the input sequence. Figure 3 Each text symbol within a box in the input sequence of the prediction network module corresponds to a corresponding y(u-1), and each text symbol in the output sequence of the prediction network module (e.g., ...) Figure 3 Each text symbol within a box in the output sequence of the prediction network module corresponds to a corresponding y(u), and each value in the output sequence of the semantic endpoint detection module (such as...) Figure 3 The value within each dashed box in the output sequence of the semantic endpoint detection module can correspond to the corresponding e(u).

[0059] by Figure 3For example, the text data in the training data can be "today how is the weather", the input sequence of the corresponding prediction network module can include "today", "day", "day", "weather", "how", and "how", the output sequence of the corresponding prediction network module can output "day", "day", "weather", "how", "how", and "how", and the output sequence of the corresponding semantic endpoint detection module can output 000001. For example, when the input text symbol of the prediction network module is "today", the output text symbol of the corresponding prediction network module can be "day", at this time, the value output by the semantic endpoint detection module can be 0, indicating that the end of the sentence has not been reached (e(u) is 1, indicating that the end of the sentence has been reached); when the input text symbol of the prediction network module is "how", the output text symbol of the prediction network module can be "how", at this time, the value output by the semantic endpoint detection module can be 1, indicating that the end of the sentence has been reached, i.e. the sentence is terminated, and the same applies when the input text symbol of the prediction network module is "day", "weather", or "how".

[0060] In the case where the trained model in the pre-training stage is obtained, the trained model can be spliced with other modules to obtain the model in the overall training stage.

[0061] Figure 4 A flowchart of a training method of the model in the overall training stage according to an embodiment of the present application is shown. As shown in FIG. 6, the training method of the model in the overall training stage according to an embodiment of the present application includes the following steps. Figure 4As shown, the training data used in the overall training stage can include a speech corpus, which can include audio data and text data corresponding to the audio data, the audio data can include audio and annotations indicating whether each frame of the audio contains human voice, and the text data can include text and annotations indicating whether the semantics corresponding to each character (i.e., each text symbol) in the text is complete (e.g., whether it is the end of a complete sentence), wherein the content of the text data corresponds to the content of the human voice. Based on the pre-trained prediction network module and the semantic endpoint detection module, the model of the overall training stage further includes an encoder module, a speech behavior detection module, and a correlation network module, wherein the encoder module can be a six-layer transformer structure or other structure capable of extracting high-dimensional feature vectors of audio, the speech behavior detection module can be a one-layer LSTM structure or other structure capable of detecting front endpoints, and the correlation network module can be a one-layer fully connected network structure or other structure capable of predicting the next text symbol. The encoder module, the speech behavior detection module, and the correlation network module can also be other structures, which are not limited in the present application. x(t) can represent the audio of the current frame, v(t) can represent whether the audio of the current frame contains human voice, and t can represent the sequence number of the audio frame. The encoder module is used to obtain a high-dimensional feature vector of the frame of audio from the input audio x(t) of the current frame, the speech behavior detection module is used to detect whether the input x(t) contains human voice according to the high-dimensional feature vector, and the correlation network module is used to predict the current text symbol y(u) according to the high-dimensional feature vector of the audio output by the encoder module and the high-dimensional feature vector of the previous text symbol output by the prediction network module. In the pre-training process described above, the prediction network module outputs the predicted current text symbol, while in the overall training stage, the prediction network module outputs the intermediate result, i.e., the high-dimensional feature vector of the previous text symbol.

[0062] In the overall training stage, for a certain piece of audio in the training data, the audio x(t) of each frame can be taken as the input of the encoder module, and the last text symbol y(u-1) in the text sequence can be taken as the input of the prediction network module. Corresponding to the two inputs, the audio whether containing human voice v(t) (when containing human voice, v(t) takes the value of 1, otherwise 0) can be taken as the output of the speech behavior detection module, the current text symbol y(u) of the text sequence can be taken as the output of the association network module, and the current text symbol whether reaching the end point e(u) can be taken as the output of the semantic endpoint detection module. The binary cross-entropy loss is used to optimize the loss of the outputs v(t) of the speech behavior detection module and the outputs e(u) of the semantic endpoint detection module, and the transducer loss is used to optimize the loss of the output y(u) of the association network module to solve the problem that the sequence number t of the audio frame does not correspond to the sequence number u of the text symbol. Through loss optimization until the loss optimization value of the model corresponding to the overall training stage converges to a predetermined value, a trained target detection model can be obtained.

[0063] Figure 5 A schematic diagram of the input and output sequences of the overall training stage according to an embodiment of the present application is shown. As shown in Figure 5 The input of the encoder module can correspond to x(t) for each frame of audio in the audio sequence, and the input sequence of the prediction network module can refer to Figure 3 The output sequence of the association network module can correspond to y(u) for each text symbol (such as Figure 5 The text symbol in each box in the output sequence of the association network module) can correspond to y(u), and the output sequence of the semantic endpoint detection module can correspond to e(u) for each value (such as Figure 5 The value in each dashed box in the output sequence of the semantic endpoint detection module) can correspond to e(u), and the output sequence of the speech behavior detection module can correspond to v(t) for each value (such as Figure 5 The value in each dashed box in the output sequence of the speech behavior detection module) can correspond to v(t), wherein each v(t) can correspond to an audio frame, each y(u) and e(u) can correspond to a text symbol, and one text symbol can correspond to one or more audio frames, which is not limited by the present application. In the case where one text symbol corresponds to multiple audio frames, multiple x(t) and one y(u-1) can be used to obtain one y(u), and the multiple x(t) corresponding to one text symbol can be referred to as "audio signal" hereinafter.

[0064] In the overall training stage, for a certain piece of audio in the training data, the audio x(t) of each frame can be taken as the input of the encoder module, and the last text symbol y(u-1) in the text sequence can be taken as the input of the prediction network module. Corresponding to the two inputs, the audio whether containing human voice v(t) (when containing human voice, v(t) takes the value of 1, otherwise 0) can be taken as the output of the speech behavior detection module, the current text symbol y(u) of the text sequence can be taken as the output of the association network module, and the current text symbol whether reaching the end point e(u) can be taken as the output of the semantic endpoint detection module. The binary cross-entropy loss is used to optimize the loss of the outputs v(t) of the speech behavior detection module and the outputs e(u) of the semantic endpoint detection module, and the transducer loss is used to optimize the loss of the output y(u) of the association network module to solve the problem that the sequence number t of the audio frame does not correspond to the sequence number u of the text symbol. Through loss optimization until the loss optimization value of the model corresponding to the overall training stage converges to a predetermined value, a trained target detection model can be obtained. Figure 5For example, for a certain input audio and text sequence (such as "today weather how") in the training data, the output sequence of the corresponding association network module can be "today", "day", "day", "weather", "how", "how", and "how", the output sequence of the corresponding semantic endpoint detection module can be 0000001, and the output sequence of the corresponding speech behavior detection module can be 00111110011110. For a certain audio (such as Figure 5 For a certain frame of audio x(t) input into the encoder module, one output y(u) (that is, a predicted current text symbol, such as "today") can be obtained, or multiple frames of audio x(t) can correspond to one output y(u). Similarly, for a certain frame of audio x(t) input into the encoder module, one output e(u) can be obtained, or multiple frames of audio x(t) can correspond to one output e(u). For a certain frame of audio x(t) input into the encoder module, one output v(t) can be obtained, where v(t) is 1 when the frame x(t) contains human voice, and 0 otherwise. For the first input y(u-1) of the prediction network module in a certain text sequence, a default value (such as a null character) can be used, and the first output y(u) of the association network module can be determined according to the default value and at least one corresponding audio frame x(t) (such as Figure 5 For example, the first text character "today" in the output sequence of the association network module).

[0065] The trained target detection model fuses the VAD model and the ASR model in structure, so that multiple models can be fused in one model. The association network module, the encoder module, and the prediction network module can be in the transducer structure of the ASR model to identify the text in the audio, and the speech behavior detection module and the semantic endpoint detection module can be used as the VAD model to detect the front endpoint and the rear endpoint of the audio, respectively. By judging the front and rear endpoints of the audio based on semantics, the complex and artificially set multi-model decision process can be avoided, and the number of models and the deployment process can be simplified, thereby saving a large amount of resources.

[0066] Figure 6 A flowchart of a model prediction stage according to an embodiment of the present application is shown. Through the pre-training stage and the training stage of the model, a trained target detection model can be obtained, which can be deployed on a terminal device, such as Figure 6 As shown, the flow of the model prediction stage can include:

[0067] Step S601, front endpoint detection stage.

[0068] In the front endpoint detection stage, the target detection model deployed on the terminal device is started by default, such as Figure 6(a) is shown, at this time, the model can only run the encoder module and the speech behavior detection module to save the amount of calculation of model runtime, the running encoder module and the speech behavior detection module can listen to the speech in the background of the terminal device, when the v(t) output by the speech behavior detection module takes the value of 1, it indicates that the front end point is reached (that is, it is recognized that the current audio frame contains human voice), and the next stage can be entered, that is, the back end point detection stage in step S602.

[0069] Wherein, the user can open the application supporting voice detection in the terminal device to start the model and enter the front end point detection stage, and can also say the wake-up word (for example, "Xiaoyi Xiaoyi") to start the model and enter the front end point detection stage, which is not limited in the present application.

[0070] For example, after entering the front end point detection stage, when the user speaks, the speech behavior detection module of the model can judge that the user is speaking (v(t) takes the value of 1), and starts to receive sound, at this time, the speech behavior detection module can be stopped to save resources, and enter the back end point detection stage. When the v(t) output by the speech behavior detection module takes the value of 1, the speech behavior detection module can stop working to stop detection. The speech behavior detection module can be restarted to detect the front end point again after detecting the user's wake-up instruction.

[0071] Step S602, back end point detection stage.

[0072] In the back end point detection stage, as shown in Figure 6(b) as shown, at this time, the model can only run the encoder module, the prediction network module, the association network module and the semantic endpoint detection module, and stop running the speech behavior detection module, at this time, the encoder module can obtain a high-dimensional feature vector of audio x(t) according to each frame of audio x(t) in the input audio when the user speaks, the prediction network module can obtain a high-dimensional feature vector of the text symbol y(u-1) according to the last text symbol y(u-1) identified by the association network module (wherein, for the first input text symbol y(u-1) when the prediction network module is running, a default value can be taken, for example, an empty character), the association network module can identify the current text symbol y(u) corresponding to the user speaking according to the high-dimensional feature vector of the audio x(t) and the high-dimensional feature vector of the text symbol y(u-1), and the identified y(u) can be used as the next input of the prediction network module, that is, the next y(u-1), and finally the text corresponding to the audio content of the user speaking can be obtained according to the output y(u), which is used for subsequent operations (such as language understanding, dialogue control, etc.), and the semantic endpoint detection module can judge whether the current text symbol y(u) reaches the semantic endpoint according to the high-dimensional feature vector of the text symbol y(u-1) (that is, whether the sentence semantics is complete at this time, or in other words, whether y(u) is the last word of a sentence expressing complete semantics), when the e(u) output by the semantic endpoint detection module takes the value 1, it indicates that the semantic endpoint is reached (that is, it is judged that the sentence semantics is complete at this time), and then the sound collection can be stopped and the next stage (such as language understanding, etc.) can be entered.

[0073] For example, after entering the endpoint detection stage, when the user speaking is stalled, the semantic endpoint detection module can be used to judge whether the semantics is complete at this time, in the case where the semantics is not complete, the sound collection can be continued until the semantics is complete or the user's stall time exceeds the predetermined time, and the sound collection can be closed to end the speech detection, thereby preventing the situation of stopping the sound collection too early due to misjudgment, improving the prediction accuracy of the target detection model, and improving the user experience.

[0074] Figure 7 A flowchart of a speech detection method according to an embodiment of the present application is shown. The method can be used in a terminal device, such as a mobile phone, a smart speaker, etc. Figure 7 As shown, the method comprises:

[0075] In step S701, a second text symbol is determined according to a first text symbol and a first audio signal of an audio signal sequence, the initial value of the first text symbol is an empty character, and the second text symbol corresponds to the content of the first audio signal;

[0076] In step S702, it is determined whether the semantics of the second text symbol reaches an endpoint according to the first text symbol, and the endpoint represents the end of speech in the audio signal sequence.

[0077] Step S703, in the case where the semantic of the second text symbol does not reach the end point, taking the second text symbol as a new first text symbol, taking an audio signal after the first audio signal in the audio signal sequence as a new first audio signal, repeating the steps of determining a second text symbol according to a first text symbol and a first audio signal in the audio signal sequence, and the following steps.

[0078] According to the embodiments of the present application, by determining a second text symbol according to a first text symbol and a first audio signal in the audio signal sequence, and determining whether the semantic of the second text symbol reaches the end point according to the first text symbol, the semantic can be used to determine whether the text symbol reaches the end point, so that the audio recording can be stopped when the user is speaking with a pause, and the false judgment can be prevented, so that the detection of the end point is more accurate, and the user experience is improved. By repeating the steps of determining a second text symbol according to a first text symbol and a first audio signal in the audio signal sequence, and the following steps in the case where the semantic of the second text symbol does not reach the end point, taking the second text symbol as a new first text symbol, and taking an audio signal after the first audio signal in the audio signal sequence as a new first audio signal, the end point detection of the audio can be performed without relying on a complex and artificially set multi-model decision process, and the voice detection method is more flexible, and the accuracy of the voice detection method is improved.

[0079] The first audio signal can include one audio frame, and can also include multiple audio frames. The content of the first audio signal can be the content of the user's speech. The end of speech can be the user's intention to stop speaking. One audio frame in the first audio signal can refer to Figure 6 x(t) in (b), the first text symbol can refer to Figure 6 y(u-1) in (b). When predicting the first second text symbol, there is no previous text symbol, so the first text symbol at this time can be a null character (i.e., the initial value is a null character). The second text symbol can refer to Figure 6 y(u) in (b). Whether the semantic of the second text symbol reaches the end point can be determined according to, for example Figure 6 The value of e(u) in (b). The semantic of the second text symbol not reaching the end point can include the case where the user is speaking with a pause. In the case where the semantic of the second text symbol reaches the end point, the audio recording can be stopped, and subsequent operations (such as language understanding, dialogue control, etc.) can be performed, which is not limited in the present application.

[0080] Examples of steps S701-S703 can refer toStep S602 in (a). Figure 6

[0081] Figure 8A flowchart of a voice detection method according to an embodiment of the present application is shown. As shown in Figure 8 The method further includes:

[0082] In step S801, it is detected in sequence whether an audio frame contained in each audio signal in the audio signal sequence reaches a front end point, the front end point indicating a voice start in the audio signal sequence.

[0083] In step S802, when a first audio frame reaching the front end point is detected, an audio signal in which the audio frame is located is determined as the first audio signal, and the detection is stopped.

[0084] According to the embodiments of the present application, by detecting in sequence whether an audio frame contained in each audio signal in the audio signal sequence reaches a front end point, when a first audio frame reaching the front end point is detected, a first audio signal in which the audio frame is located is determined as the first audio signal, and the detection is stopped, the front end point of the audio can be detected on the basis of the detection of the rear end point, so that the human voice interval can be more accurately determined. When the audio signal reaching the front end point is detected, the detection is stopped, and the calculation amount related can be reduced, and resources can be saved.

[0085] The audio frame can or can not include human voice, and whether the audio frame reaches the front end point can be determined according to, for example Figure 6 The value of v(t) in (a) can be determined, and the voice start can mean that the user starts to speak.

[0086] Examples of steps S801-S802 can refer to step S601 in Figure 6

[0087] In a possible implementation, the method can be used for a target detection model, the target detection model including a voice behavior detection module, an association network module, a semantic end point detection module, an encoder module, and a prediction network module. The encoder module is configured to obtain a first feature vector of an audio frame contained in an audio signal in an audio signal sequence. The voice behavior detection module is configured to determine whether the audio frame contained in each audio signal in the audio signal sequence reaches a front end point according to the first feature vector. The prediction network module is configured to obtain a second feature vector of a first text symbol. The association network module is configured to obtain the second text symbol according to the first feature vector and the second feature vector. The semantic end point detection module is configured to determine whether a semantic of the second text symbol reaches a rear end point according to the second feature vector.

[0088] ​According to the embodiment of the present application, by means of the target detection model including the speech behavior detection module, the association network module, the semantic endpoint detection module, the encoder module and the prediction network module, the VAD model and the ASR model can be integrated in one model, the front endpoint can be detected based on the semantic detection backend point, the model quantity and the deployment process are greatly simplified, a large amount of resources is saved, and the accuracy of the speech detection is improved based on the semantic detection backend point.

[0089] The encoder module can be connected with the speech behavior detection module and the association network module, the prediction network module can be connected with the association network module and the semantic endpoint detection module, the speech behavior detection module can be a one-layer LSTM structure, the association network module can be a one-layer fully connected network structure, the semantic endpoint detection module can be a two-layer LSTM structure, the encoder module can be a six-layer transformer structure, and the prediction network module can be a twelve-layer transformer structure. The speech behavior detection module, the association network module, the semantic endpoint detection module, the encoder module and the prediction network module can also be other structures, which are not limited in the present application. The first feature vector, for example, includes the high-dimensional feature vector of the audio mentioned above, and the second feature vector, for example, includes the high-dimensional feature vector of the text symbol mentioned above.

[0090] Examples of the speech behavior detection module, the association network module, the semantic endpoint detection module, the encoder module and the prediction network module can be referred to as shown in Figure 4

[0091] In a possible implementation, the speech behavior detection module stops running when it is determined that the audio frames contained in each audio signal in the audio signal sequence reach the front endpoint according to the first feature vector.

[0092] According to the embodiment of the present application, the running of the speech behavior detection module can be paused after the front endpoint is determined, so as to save resources.

[0093] Figure 9 A flowchart of a speech detection method according to an embodiment of the present application is shown. The method can be used in the speech detection platform described above, as shown in Figure 9 The method further includes,

[0094] In step S901, the prediction network module and the semantic endpoint detection module are pre-trained.

[0095] In step S902, the target detection model including the speech behavior detection module, the association network module, the encoder module, the pre-trained prediction network module and the pre-trained semantic endpoint detection module are trained to obtain the trained target detection model.

[0096] ​According to the embodiment of the present application, the target detection module including the speech behavior detection module, the association network module, the encoder module, the pre-trained prediction network module and the pre-trained semantic endpoint detection module is trained to obtain the trained target detection module, which can make the training process more targeted and the semantic judgment-based endpoint more accurate. By pre-training first, the subsequent training model can obtain better training effect, and the prediction accuracy of the trained target detection model is higher.

[0097] In the present application, the pre-training and training methods are not limited, and the target detection model obtained from the pre-trained prediction network module and the pre-trained semantic endpoint detection module is not limited. For example, the pre-trained prediction network module and the pre-trained semantic endpoint detection module can be spliced with the speech behavior detection module, the association network module and the encoder module.

[0098] The example of step S901 can refer to the model training method flowchart of the pre-training phase in the present application. Figure 2 The example of step S902 can refer to the model training method flowchart of the whole training phase in the present application.

[0099] Figure 10 The flowchart of the speech detection method according to an embodiment of the present application is shown. As shown in Figure 10 The pre-training of the prediction network module and the semantic endpoint detection module includes:

[0100] Step S1001, input the previous text symbol in the text sample into the prediction network module, and the prediction network module predicts the current text symbol according to the feature vector of the previous text symbol;

[0101] Step S1002, input the current text symbol into the semantic endpoint detection module to determine whether the semantic of the current text symbol reaches the end point;

[0102] Step S1003, calculate the first loss value of the prediction network module according to the label of the current text symbol, and calculate the second loss value of the semantic endpoint detection module according to the label of whether the current text symbol reaches the end point, and adjust the parameters of the prediction network module and the semantic endpoint detection module according to the first loss value and the second loss value.

[0103] According to the embodiment of the present application, by inputting the previous text symbol in the text sample into the prediction network module, the prediction network module predicts the current text symbol according to the feature vector of the previous text symbol, inputting the current text symbol into the semantic endpoint detection module, determining whether the semantic of the current text symbol reaches the rear endpoint, calculating the first loss value of the prediction network module according to the label of the current text symbol, and calculating the second loss value of the semantic endpoint detection module according to the label of whether the current text symbol reaches the rear endpoint, and adjusting the parameters of the prediction network module and the semantic endpoint detection module according to the first loss value and the second loss value, the function of the model based on semantic detection of the rear endpoint can be more targeted in the pre-training stage, thereby improving the accuracy of the model in speech detection. In the pre-training stage, the model is trained by using the easily obtained text sample, without using the audio sample, which can save related resources, and further improve the accuracy of the model in speech detection based on semantics.

[0104] The text sample can be obtained by crawling or purchasing, and the present application does not limit the way of obtaining the text sample. The present application does not limit the method of calculating the first loss value and the second loss value, for example, the first loss value can be calculated using cross-entropy loss, and the second loss value can be calculated using binary cross-entropy loss. The previous text symbol in the text sample can be, for example Figure 2 The current text symbol can be, for example Figure 2 Whether the semantic of the current text symbol reaches the rear endpoint can be determined according to, for example Figure 2 The value of e(u).

[0105] The examples of steps S1001-S1003 can refer to the flow of the model training method in the pre-training stage shown in Figure 2 .

[0106] Figure 11 The structure diagram of the speech detection device according to an embodiment of the present application is shown. As shown in Figure 11 , the device includes:

[0107] The first determination module 1101 is configured to determine the second text symbol according to the first text symbol and the first audio signal in the audio signal sequence, the initial value of the first text symbol being an empty character, and the second text symbol corresponding to the content of the first audio signal.

[0108] The second determination module 1102 is configured to determine whether the semantic of the second text symbol reaches the rear endpoint according to the first text symbol, the rear endpoint indicating the end of speech in the audio signal sequence.

[0109] The third determination module 1103 is configured to, in a case where the semantic of the second text symbol does not reach the rear endpoint, take the second text symbol as a new first text symbol, take an audio signal after the first audio signal in the audio signal sequence as a new first audio signal, and repeatedly perform the steps of determining the second text symbol according to the first text symbol and the first audio signal in the audio signal sequence, and determining whether the semantic of the second text symbol reaches the rear endpoint according to the first text symbol.

[0110] According to the embodiments of the present application, by determining the second text symbol according to the first text symbol and the first audio signal in the audio signal sequence, and determining whether the semantic of the second text symbol reaches the rear endpoint according to the first text symbol, it can be realized that whether the text symbol reaches the rear endpoint is determined according to the semantic, the audio is ended in advance when the user speaks is avoided, the misjudgment is prevented, the detection of the rear endpoint is more accurate, and the experience of the user is improved. By repeatedly performing the steps of determining the second text symbol according to the first text symbol and the first audio signal in the audio signal sequence, and determining whether the semantic of the second text symbol reaches the rear endpoint according to the first text symbol in a case where the semantic of the second text symbol does not reach the rear endpoint, taking the second text symbol as a new first text symbol, and taking an audio signal after the first audio signal in the audio signal sequence as a new first audio signal, it can be realized that the rear endpoint detection of the audio does not need to rely on a complex and artificially set multi-model decision process, the voice detection method is more flexible, and the accuracy of the voice detection method is improved.

[0111] In a possible implementation, the apparatus further includes a detection module configured to sequentially detect whether an audio frame contained in each audio signal in the audio signal sequence reaches a front endpoint, the front endpoint indicating a start of speech in the audio signal sequence; and a fourth determination module configured to, when a first audio frame reaching the front endpoint is detected, determine an audio signal in which the audio frame is located as the first audio signal, and stop the detection.

[0112] According to the embodiments of the present application, by sequentially detecting whether an audio frame contained in each audio signal in the audio signal sequence reaches a front endpoint, and when a first audio frame reaching the front endpoint is detected, determining a first audio signal in which the audio frame is located as the first audio signal and stopping the detection, the front endpoint of the audio can be detected on the basis of the detection of the rear endpoint, so that the human voice interval can be more accurately determined, the detection is stopped when the audio signal reaching the front endpoint is detected, the calculation amount related can be reduced, and resources can be saved.

[0113] In a possible implementation, the apparatus is used for a target detection model, the target detection model comprising a speech behavior detection module, an association network module, a semantic endpoint detection module, an encoder module, and a prediction network module, the encoder module being configured to obtain a first feature vector of an audio frame contained in an audio signal in an audio signal sequence; the speech behavior detection module being configured to determine whether the audio frame contained in the audio signal in the audio signal sequence reaches a front endpoint according to the first feature vector; the prediction network module being configured to obtain a second feature vector of a first text symbol; the association network module being configured to obtain the second text symbol according to the first feature vector and the second feature vector; and the semantic endpoint detection module being configured to determine whether semantics of the second text symbol reaches a back endpoint according to the second feature vector.

[0114] According to the embodiment of the present application, by means of the target detection model comprising the speech behavior detection module, the association network module, the semantic endpoint detection module, the encoder module, and the prediction network module, the VAD model and the ASR model can be integrated in one model, the front endpoint can be detected while the back endpoint is detected based on semantics, the model quantity and the deployment process are greatly simplified, a large amount of resources is saved, and the accuracy of speech detection is improved based on semantic detection of the back endpoint.

[0115] In a possible implementation, the apparatus further comprises a pre-training module configured to pre-train the prediction network module and the semantic endpoint detection module; and a training module configured to train the target detection model comprising the speech behavior detection module, the association network module, the encoder module, and the pre-trained prediction network module and the pre-trained semantic endpoint detection module, to obtain a trained target detection model.

[0116] According to the embodiment of the present application, by means of pre-training the prediction network module and the semantic endpoint detection module, training the target detection model comprising the speech behavior detection module, the association network module, the encoder module, and the pre-trained prediction network module and the pre-trained semantic endpoint detection module, to obtain the trained target detection model, the training process can be more targeted, the back endpoint based on semantic judgment can be more accurate, and by means of pre-training, the subsequent training model can obtain better training effect, and the prediction accuracy of the trained target detection model is higher.

[0117] In a possible implementation, the pre-training module is configured to: input a previous text symbol in the text sample into the prediction network module, and the prediction network module predicts a current text symbol according to a feature vector of the previous text symbol; input the current text symbol into the semantic endpoint detection module, and determine whether the semantic of the current text symbol reaches the back endpoint; calculate a first loss value of the prediction network module according to a label of the current text symbol, and calculate a second loss value of the semantic endpoint detection module according to a label of whether the current text symbol reaches the back endpoint; and adjust parameters of the prediction network module and the semantic endpoint detection module according to the first loss value and the second loss value.

[0118] According to the embodiments of the present application, by inputting a previous text symbol in the text sample into the prediction network module, and the prediction network module predicting a current text symbol according to a feature vector of the previous text symbol, inputting the current text symbol into the semantic endpoint detection module, and determining whether the semantic of the current text symbol reaches the back endpoint, calculating a first loss value of the prediction network module according to a label of the current text symbol, and calculating a second loss value of the semantic endpoint detection module according to a label of whether the current text symbol reaches the back endpoint, and adjusting parameters of the prediction network module and the semantic endpoint detection module according to the first loss value and the second loss value, the function of the model detecting the back endpoint based on the semantic can be more targetedly trained in the pre-training stage, and the accuracy of the model in speech detection is improved. In the pre-training stage, the model is trained by using the text sample which is easy to obtain, and the audio sample is not needed, so that the related resources can be saved, and the accuracy of the model in speech detection based on the semantic is further improved.

[0119] In a possible implementation, the speech behavior detection module stops running when it is determined, according to the first feature vector, that the audio frame contained in each audio signal in the audio signal sequence reaches the front endpoint.

[0120] According to the embodiments of the present application, the running of the speech behavior detection module can be paused after the front endpoint is determined, so as to save resources.

[0121] Figure 12 A structural schematic diagram of a terminal device according to an embodiment of the present application is shown. Taking a mobile phone as an example, Figure 12 A structural schematic diagram of the mobile phone 200 is shown.

[0122] The mobile phone 200 can include a processor 210, an external memory interface 220, an internal memory 221, a USB interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 251, a wireless communication module 252, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headset jack 270D, a sensor module 280, a key 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a SIM card interface 295, etc. The sensor module 280 can include a gyroscope sensor 280A, an acceleration sensor 280B, a proximity light sensor 280G, a fingerprint sensor 280H, and a touch sensor 280K (of course, the mobile phone 200 can also include other sensors, such as a temperature sensor, a pressure sensor, a distance sensor, a magnetic sensor, an ambient light sensor, an air pressure sensor, a bone conduction sensor, etc., which are not shown in the figure).

[0123] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the mobile phone 200. In other embodiments of the present application, the mobile phone 200 can include more or fewer components than shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0124] The processor 210 can include one or more processing units, for example: the processor 210 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices, or can be integrated in one or more processors. The controller can be the nerve center and command center of the mobile phone 200. The controller can generate operation control signals according to instruction operation codes and timing signals to complete the control of fetching and executing instructions.

[0125] The processor 210 can also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory can hold instructions or data that the processor 210 has just used or is recycling. If the processor 210 needs to use the instructions or data again, it can be called directly from the memory. This avoids repeated access and reduces the latency of the processor 210, thus improving the efficiency of the system.

[0126] The processor 210 can run the voice detection method provided by the embodiments of the present application to accurately determine the human voice interval of the audio based on semantic detection of the back end point and detection of the front end point of the audio. The processor 210 can include different devices, such as an integrated CPU and GPU. The CPU and GPU can cooperate to execute the voice detection method provided by the embodiments of the present application, such as the CPU executing part of the algorithm in the voice detection method and the GPU executing another part of the algorithm, to obtain faster processing efficiency.

[0127] The internal memory 221 can be used to store computer executable program codes, which include instructions. The processor 210 executes various functional applications and data processing of the mobile phone 200 by running the instructions stored in the internal memory 221. The internal memory 221 can include a program storage area and a data storage area. The program storage area can store an operating system, codes of application programs (such as a camera application, a WeChat application, etc.), and the like. The data storage area can store data created during use of the mobile phone 200 (such as images and videos collected by the camera application), and the like.

[0128] The internal memory 221 can also store one or more computer programs 1310 corresponding to the voice detection method provided by the embodiments of the present application. The one or more computer programs 1304 are stored in the above-mentioned memory 221 and are configured to be executed by the one or more processors 210. The one or more computer programs 1310 include instructions, which can be used to perform the above-mentioned voice detection method, the above-mentioned voice detection method, and the like. Figure 2 、 Figure 4 、 Figure 6 、 Figures 7-10The computer program 1310 can include a first determination module 1101, a second determination module 1102, and a third determination module 1103 according to various steps in the embodiments. The first determination module 1101 is configured to determine a second text symbol according to a first text symbol and a first audio signal in an audio signal sequence, an initial value of the first text symbol being an empty character, the second text symbol corresponding to content of the first audio signal. The second determination module 1102 is configured to determine whether semantics of the second text symbol reaches a rear end point according to the first text symbol, the rear end point indicating an end of speech in the audio signal sequence. The third determination module 1103 is configured to, in a case where the semantics of the second text symbol does not reach the rear end point, repeat the steps of determining the second text symbol according to the first text symbol and the first audio signal in the audio signal sequence, and the following steps, with the second text symbol as a new first text symbol, and an audio signal after the first audio signal in the audio signal sequence as a new first audio signal. When the code of the transmission method of the data stored in the internal memory 221 is executed by the processor 210, the processor 210 can control the display screen to display the prediction result of the target model.

[0129] In addition, the internal memory 221 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like.

[0130] Of course, the code of the speech detection method provided by the embodiments of the present application can also be stored in the external memory. In this case, the processor 210 can execute the code of the speech detection method stored in the external memory through the external memory interface 220.

[0131] The wireless communication function of the mobile phone 200 can be realized through the antenna 1, the antenna 2, the mobile communication module 251, the wireless communication module 252, the modem processor, and the baseband processor, and the like.

[0132] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the mobile phone 200 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example, the antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0133] The mobile communication module 251 can provide a solution for wireless communication including 2G / 3G / 4G / 5G, etc. applied to the mobile phone 200. The mobile communication module 251 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 251 can receive electromagnetic waves by the antenna 1, and perform filtering, amplification, etc. on the received electromagnetic waves, and transmit the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 251 can also amplify the signals modulated by the modem processor, and radiate the signals as electromagnetic waves through the antenna 1. In some embodiments, at least part of the function modules of the mobile communication module 251 can be arranged in the processor 210. In some embodiments, at least part of the function modules of the mobile communication module 251 can be arranged in the same device as at least part of the modules of the processor 210. In the embodiments of the present application, the mobile communication module 251 can also be used to interact with other electronic devices to obtain the trained target detection model.

[0134] The wireless communication module 252 can provide a solution for wireless communication including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. applied to the mobile phone 200. The wireless communication module 252 can be one or more devices integrated with at least one communication processing module. The wireless communication module 252 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 210. The wireless communication module 252 can also receive signals to be sent from the processor 210, perform frequency modulation and amplification on the signals, and radiate the signals as electromagnetic waves through the antenna 2. In the embodiments of the present application, the wireless communication module 252 is used to transmit data between other electronic devices under the control of the processor 210, for example, when the processor 210 runs the voice detection method provided in the embodiments of the present application, the processor can control the wireless communication module 252 to receive the trained target detection model sent by the electronic device.

[0135] In addition, the mobile phone 200 can realize an audio function through an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headset interface 270D, an application processor, and the like. For example, receiving user input audio, performing voice broadcast, and the like. It should be understood that in actual applications, the mobile phone 200 can include more or fewer components than those shown in the figure, and the embodiments of the present application are not limited. The mobile phone 200 shown in the figure is only an example, and the mobile phone 200 can have more or fewer components than those shown in the figure, can combine two or more components, or can have a different component configuration. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits. Figure 12

[0136] Figure 13 A structural diagram of an electronic device according to an embodiment of the present application is shown. As shown in the figure, the electronic device 40 includes at least one processor 1801, at least one memory 1802, and at least one communication interface 1803. In addition, the electronic device can also include general-purpose components such as an antenna, which are not described in detail here. Figure 13

[0137] Through the electronic device shown in the embodiments of the present application, the model can be pre-trained as shown in Figure 2 and overall training as shown in Figure 4 to obtain a trained target detection model. After training, the target detection model can also be exported through a model format conversion tool and deployed on a terminal device as shown in Figure 12 .

[0138] The processor 1801 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of programs of the above solutions. The processor 1801 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices, or can be integrated in one or more processors.​​

[0139] The communication interface 1803 is configured to communicate with other devices or communication networks, such as an Ethernet, a Radio Access Network (RAN), a core network, a Wireless Local Area Networks (WLAN), and the like.

[0140] The memory 1802 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, a magnetic disk storage or other magnetic storage devices, or any other medium capable of storing desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to this. The memory can exist independently and be connected to the processor through a bus. The memory can also be integrated with the processor.

[0141] The memory 1802 is configured to store application program codes for implementing the above solutions, and the processor 1801 is configured to control the execution of the application program codes.

[0142] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0143] Embodiments of the present application provide a voice detection apparatus, including a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions.

[0144] Embodiments of the present application provide a non-volatile computer readable storage medium, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the above method.

[0145] Embodiments of the present application provide a computer program product including computer readable code, or a nonvolatile computer readable storage medium carrying computer readable code, which, when run in a processor of an electronic device, causes the processor in the electronic device to perform the above method.

[0146] Embodiments of the present application provide a terminal device which can perform the above method.

[0147] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital video disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or punched tape, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves.

[0148] Computer readable program instructions or code described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0149] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0150] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0151] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or nonvolatile memory, or a suitable combination of the different types of computer readable storage media. The computer readable storage medium can also be a computer readable storage medium encoded in a machine-readable or used language that includes any type of markers that the computer readable storage medium can use to program a computing device to perform a particular function or enable a particular function or action.

[0152] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0153] The flow diagrams and the block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses, systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical functions (s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts.

[0154] It is also important to note that each of the boxes in the block diagrams and / or flowcharts, and combinations of boxes in the block diagrams and / or flowcharts, can be implemented by hardware, for example, circuitry or ASICs (Application Specific Integrated Circuits), or can be implemented by a combination of hardware and software, such as firmware, etc.

[0155] Although the present application is described in relation to various embodiments, it will be appreciated that other variations of the disclosed embodiments can be understood and effected by those skilled in the art in the field of the application, in light of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit can fulfil the functions of several items recited in the claims. A plurality of distinct items recited as means for performing a function can be implemented by a single item providing that function, without deviating from the scope of the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0156] Having described various embodiments of the application, it is to be understood that the above description is meant not to limit and not to encompass all of the possible embodiments. Many modifications and variations of this application can be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. It is intended that the scope of the application be defined by the scope of the patent and by the claims as allowed by the patent office, which can include adaptations based on the description, equivalents, and / or substitutions of elements individually or collectively to the entire disclosure.

Claims

1. A voice detection method characterized by, The method comprises: sequentially detecting whether an audio frame contained in each audio signal in the audio signal sequence reaches a front end point, the front end point indicating the beginning of speech in the audio signal sequence; wherein whether the audio frame contained in each audio signal in the audio signal sequence reaches the front end point is determined according to a first feature vector of the audio frame; when a first audio frame reaching the front end point is detected, determining an audio signal in which the audio frame is located as a first audio signal, and stopping the detection; determining a second text symbol according to a second feature vector of the first text symbol and the first feature vector of the audio frame contained in the first audio signal in the audio signal sequence, the initial value of the first text symbol being an empty character, the second text symbol corresponding to the content of the first audio signal; determining whether the semantics of the second text symbol reaches a back end point according to the second feature vector, the back end point indicating the end of speech in the audio signal sequence; in the case where the semantics of the second text symbol does not reach the back end point, taking the second text symbol as a new first text symbol, taking an audio signal after the first audio signal in the audio signal sequence as a new first audio signal, and repeatedly performing the steps of determining a second text symbol according to a second feature vector of the first text symbol and a first feature vector of an audio frame contained in the first audio signal in the audio signal sequence, and the subsequent steps.

2. The method of claim 1, wherein, The method is used for a target detection model, the target detection model comprising a speech behavior detection module, an association network module, a semantic end point detection module, an encoder module and a prediction network module, the encoder module is configured to obtain a first feature vector of an audio frame contained in an audio signal in an audio signal sequence; the speech behavior detection module is configured to determine whether the audio frame contained in each audio signal in the audio signal sequence reaches a front end point according to the first feature vector; the prediction network module is configured to obtain a second feature vector of a first text symbol; the association network module is configured to obtain the second text symbol according to the first feature vector and the second feature vector; the semantic end point detection module is configured to determine whether the semantics of the second text symbol reaches a back end point according to the second feature vector.

3. The method of claim 2, wherein, The method further comprises: pre-training the prediction network module and the semantic end point detection module; training the target detection model comprising the speech behavior detection module, the association network module, the encoder module and the pre-trained prediction network module and the pre-trained semantic end point detection module to obtain a trained target detection model.

4. The method of claim 3, wherein, The pre-training of the prediction network module and the semantic end point detection module comprises: inputting a previous text symbol in a text sample into the prediction network module, the prediction network module predicting a current text symbol according to a feature vector of the previous text symbol; inputting the current text symbol into the semantic end point detection module to determine whether the semantics of the current text symbol reaches a back end point; The first loss value of the prediction network module is calculated according to the annotation of the current text symbol, and the second loss value of the semantic endpoint detection module is calculated according to the annotation of whether the current text symbol reaches the rear endpoint, and the prediction network module and the semantic endpoint detection module are adjusted in parameters according to the first loss value and the second loss value.

5. The method of claim 3, wherein, The voice behavior detection module stops running when it is determined according to the first feature vector that an audio frame contained in each audio signal in the audio signal sequence reaches the front endpoint.

6. A voice detection apparatus characterized by comprising: The device comprises: A detection module is configured to sequentially detect whether an audio frame contained in each audio signal in the audio signal sequence reaches a front endpoint, the front endpoint indicating the start of speech in the audio signal sequence; wherein whether the audio frame contained in each audio signal in the audio signal sequence reaches the front endpoint is determined according to a first feature vector of the audio frame. A fourth determination module is configured to determine the audio signal in which the audio frame reaching the front endpoint is located as the first audio signal when the first audio frame reaching the front endpoint is detected, and stop the detection. A first determination module is configured to determine a second text symbol according to a second feature vector of the first text symbol and the first feature vector of the audio frame contained in the first audio signal in the audio signal sequence, the initial value of the first text symbol being an empty character, and the second text symbol corresponding to the content of the first audio signal; and a second determination module is configured to determine whether the semantic of the second text symbol reaches a rear endpoint according to the second feature vector, the rear endpoint indicating the end of speech in the audio signal sequence. A third determination module is configured to, in the case that the semantic of the second text symbol does not reach the rear endpoint, take the second text symbol as a new first text symbol, take an audio signal after the first audio signal in the audio signal sequence as a new first audio signal, and repeatedly execute the steps of determining the second text symbol according to the second feature vector of the first text symbol and the first feature vector of the audio frame contained in the first audio signal in the audio signal sequence.

7. A voice detection apparatus characterized by comprising: Comprise: A processor; A memory for storing processor-executable instructions; The processor is configured to implement the method of any one of claims 1-5 when the processor executes the instructions.

8. A non-transitory computer readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions are executed by the processor to implement the method of any one of claims 1-5.

9. A computer program product comprising computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, which when run in an electronic device, a processor in the electronic device executes the method of any one of claims 1-5.

Images