Speech keyword detection method and apparatus, device, and storage medium

Abstract

The embodiment of the application provides a voice keyword detection method and device, electronic equipment and a storage medium, belongs to the technical field of voice processing, and is suitable for the field of financial technology. The method comprises the following steps: acquiring voice sample data; acquiring an initial feature extraction submodel and an initial adversarial classification submodel; performing keyword detection on the voice sample data based on the initial feature extraction submodel to obtain a predicted keyword; calculating feature extraction loss data based on the predicted keyword and a key frame label; performing type identification on the voice sample data based on the initial adversarial classification submodel and sample voice features to obtain a voice prediction category; calculating adversarial classification loss data based on the voice prediction category and a voice label; and performing gradient inversion on the initial feature extraction submodel and the initial adversarial classification submodel based on the feature extraction loss data and the adversarial classification loss data. The embodiment of the application can reduce the cost consumption of voice keyword detection.

Description

Technical Field

[0001] This application relates to the field of speech processing technology, and is applicable to the financial technology field, particularly to a speech keyword detection method and device, electronic device and storage medium. Background Technology

[0002] Voice keyword detection refers to the detection and extraction of keywords in voice data. For example, in the financial field, voice keyword detection can be performed on the voice data of communication between financial customer service and users to help financial customer service quickly review the content of the communication with users.

[0003] Currently, common methods for speech keyword detection typically employ neural networks to detect keywords in speech data. However, training neural networks requires a large amount of speech data, resulting in significant costs. Therefore, reducing the cost of speech keyword detection has become an urgent technical problem to be solved. Summary of the Invention

[0004] The main objective of this application is to provide a method, apparatus, electronic device, and storage medium for voice keyword detection, aiming to reduce the cost of voice keyword detection.

[0005] To achieve the above objectives, a first aspect of this application proposes a speech keyword detection method, the method comprising:

[0006] Acquire speech sample data containing speech tags and keyframe markers, wherein the speech sample data includes real speech data and synthesized speech data, the speech tags are used to indicate whether the speech sample data is real speech data or synthesized speech data, and the keyframe markers are used to identify speech frames in the speech sample data;

[0007] Obtain an initial keyword detection model, wherein the initial keyword detection model includes an initial feature extraction sub-model and an initial adversarial classification sub-model;

[0008] Based on the initial feature extraction sub-model, feature extraction is performed on the speech sample data to obtain sample speech features;

[0009] Based on the initial feature extraction sub-model and the sample speech features, keyword detection is performed on the speech sample data to obtain predicted keywords;

[0010] Based on the predicted keywords and the keyframe markers, feature extraction loss data is calculated.

[0011] Based on the initial adversarial classification sub-model and the sample speech features, the speech sample data is type-identified to obtain a speech prediction category, wherein the speech prediction category is used to indicate whether the speech sample data is real speech data or synthetic speech data;

[0012] Calculate the adversarial classification loss data based on the predicted speech category and the speech label;

[0013] Based on the feature extraction loss data and the adversarial classification loss data, the gradient inversion of the initial keyword detection model is performed to obtain the target keyword detection model.

[0014] Based on the target keyword detection model, keyword detection is performed on the target speech data to obtain the target keywords.

[0015] In some embodiments, the step of performing gradient inversion on the initial keyword detection model based on the feature extraction loss data and the adversarial classification loss data to obtain the target keyword detection model includes:

[0016] Based on the feature extraction loss data and the preset model parameters, the feature extraction gradient data is calculated, wherein the model parameters refer to the parameters in the initial keyword detection model;

[0017] Based on the adversarial classification loss data and the model parameters, calculate the adversarial classification gradient data;

[0018] The adversarial classification gradient data is inverted to obtain inverted gradient data;

[0019] Based on the feature extraction gradient data, the adversarial classification gradient data, and the inverted gradient data, the initial keyword detection model is backpropagated to obtain the target keyword detection model.

[0020] In some embodiments, the backpropagation of the initial keyword detection model based on the feature extraction gradient data, the adversarial classification gradient data, and the inverted gradient data to obtain the target keyword detection model includes:

[0021] Based on the adversarial classification gradient data and the feature extraction gradient data, the parameters of the initial adversarial classification sub-model are adjusted to obtain the target adversarial classification sub-model.

[0022] Based on the feature extraction gradient data and the inverted gradient data, the parameters of the initial feature extraction sub-model are adjusted to obtain the target feature extraction sub-model;

[0023] The target feature extraction sub-model and the target adversarial classification sub-model are concatenated to obtain the target keyword detection model.

[0024] In some embodiments, acquiring speech sample data including speech tags and keyframe markers includes:

[0025] Acquire recorded audio data;

[0026] The recorded voice data is labeled to obtain the real voice data, wherein the real voice data includes real tags and real keyframe markers. The real tags are used to indicate that the manually recorded data is the real voice data, and the real keyframe markers are used to identify the voice frames in the real voice data.

[0027] The real speech data is subjected to speech simulation to obtain simulated speech data;

[0028] The simulated speech data is labeled to obtain the synthesized speech data, wherein the synthesized speech data includes a synthesis tag and a synthesis keyframe marker. The synthesis tag is used to indicate that the simulated speech data is the synthesized speech data, and the synthesis keyframe marker is used to identify the speech frames in the synthesized speech data.

[0029] The real speech data and the synthesized speech data are merged to obtain the speech sample data.

[0030] In some embodiments, the step of performing speech simulation on the real speech data to obtain simulated speech data includes:

[0031] The real speech data is transcribed to obtain speech-text data;

[0032] The speech-text data is augmented to obtain augmented text data;

[0033] Speech synthesis is performed based on the enhanced text data to obtain the simulated speech data.

[0034] In some embodiments, the step of detecting keywords in the target speech data based on the target keyword detection model to obtain target keywords includes:

[0035] Based on the target keyword detection model, feature extraction is performed on the target speech data to obtain target speech features;

[0036] Based on the target keyword detection model and the target speech features, the target speech data is subjected to contextual understanding to obtain contextual information;

[0037] Based on the context information, the target speech features are classified to obtain the target feature category;

[0038] Based on the target feature category, the target speech feature is linearly mapped to obtain the keyword probability, wherein the keyword probability is used to represent the probability that the target speech feature is a keyword;

[0039] Based on the keyword probabilities, the target speech features are filtered to obtain the target keywords.

[0040] In some embodiments, the step of performing type identification on the speech sample data based on the initial adversarial classification sub-model and the sample speech features to obtain the speech prediction category includes:

[0041] Based on the initial adversarial classification sub-model, feature recognition is performed on the sample speech features to obtain the sample feature categories;

[0042] Based on the sample feature category, data tracing is performed on the sample speech features to obtain the speech prediction category.

[0043] To achieve the above objectives, a second aspect of this application provides a speech keyword detection device, the device comprising:

[0044] The sample data acquisition module is used to acquire speech sample data containing speech tags and keyframe markers. The speech sample data includes real speech data and synthesized speech data. The speech tags are used to indicate whether the speech sample data is real speech data or synthesized speech data. The keyframe markers are used to identify the speech frames in the speech sample data.

[0045] An initial model acquisition module is used to acquire an initial keyword detection model, wherein the initial keyword detection model includes an initial feature extraction sub-model and an initial adversarial classification sub-model;

[0046] The speech feature extraction module is used to extract features from the speech sample data based on the initial feature extraction sub-model to obtain sample speech features;

[0047] The keyword prediction module is used to detect keywords in the speech sample data based on the initial feature extraction sub-model and the sample speech features to obtain predicted keywords;

[0048] The first loss calculation module is used to calculate features and extract loss data based on the predicted keywords and the keyframe markers.

[0049] The speech data classification module is used to perform type identification on the speech sample data based on the initial adversarial classification sub-model and the sample speech features to obtain a speech prediction category, wherein the speech prediction category is used to indicate whether the speech sample data is real speech data or synthetic speech data;

[0050] The second loss calculation module is used to calculate adversarial classification loss data based on the speech prediction category and the speech label.

[0051] The model parameter adjustment module is used to perform gradient inversion on the initial keyword detection model based on the feature extraction loss data and the adversarial classification loss data to obtain the target keyword detection model.

[0052] The keyword detection module is used to detect keywords in the target speech data based on the target keyword detection model to obtain the target keywords.

[0053] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0054] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0055] The speech keyword detection method, apparatus, electronic device, and storage medium proposed in this application extract features from speech sample data using an initial feature extraction sub-model to obtain sample speech features. Then, based on the initial feature extraction sub-model and the sample speech features, keyword detection is performed on the speech sample data to obtain predicted keywords. Feature extraction loss data is calculated based on the predicted keywords and keyframe markers. This enables keyword detection on both real and synthesized speech data. Furthermore, using an initial adversarial classification sub-model and sample speech features, type recognition is performed on the speech sample data to obtain predicted speech categories. Adversarial classification loss data is then calculated based on the predicted speech categories and speech labels. Finally, gradient inversion is performed on the initial keyword detection model based on the feature extraction loss data and the adversarial classification loss data. This makes the speech features extracted by the initial feature extraction sub-model more closely resemble the common features of both synthesized and real speech data, thereby reducing the need for real speech data to train the keyword detection model and thus reducing the cost of speech keyword detection. Attached Figure Description

[0056] Figure 1This is a flowchart of the speech keyword detection method provided in the embodiments of this application;

[0057] Figure 2 yes Figure 1 The flowchart of step S101 in the text;

[0058] Figure 3 yes Figure 2 The flowchart of step S203 in the process;

[0059] Figure 4 yes Figure 1 The flowchart of step S106 in the process;

[0060] Figure 5 yes Figure 1 The flowchart of step S108 in the process;

[0061] Figure 6 yes Figure 5 The flowchart of step S504 in the process;

[0062] Figure 7 yes Figure 1 The flowchart of step S109 in the document;

[0063] Figure 8 This is a schematic diagram of the structure of the speech keyword detection device provided in the embodiments of this application;

[0064] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0066] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0068] First, let's analyze some of the terms used in this application:

[0069] The Virtouso model aims to achieve efficient processing and generation of speech data by combining language models and audio feature extraction techniques. It supports various application scenarios, such as speech recognition, music composition, and voice synthesis, to improve the naturalness and fluency of voice interaction.

[0070] Audio Language Model: An audio language model is a model for audio generation and understanding, designed to generate coherent audio segments from existing audio data.

[0071] Voice keyword detection refers to the detection and extraction of keywords in voice data. For example, in the financial field, voice keyword detection can be performed on the voice data of communication between financial customer service and users to help financial customer service quickly review the content of the communication with users.

[0072] Currently, common methods for speech keyword detection typically employ neural networks to detect keywords in speech data. However, training neural networks requires a large amount of speech data, resulting in significant costs. Therefore, reducing the cost of speech keyword detection has become an urgent technical problem to be solved.

[0073] Based on this, embodiments of this application provide a method and apparatus for voice keyword detection, an electronic device and a storage medium, aiming to reduce the cost of voice keyword detection.

[0074] The speech keyword detection method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the speech keyword detection method in this application is described.

[0075] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0076] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0077] The speech keyword detection method provided in this application relates to the field of speech processing technology and is applicable to the fintech field. The speech keyword detection method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speech keyword detection method, but is not limited to the above forms.

[0078] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0079] Figure 1 This is an optional flowchart of the speech keyword detection method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S109.

[0080] Step S101: Obtain speech sample data containing speech tags and keyframe markers. The speech sample data includes real speech data and synthesized speech data. Speech tags are used to indicate whether the speech sample data is real speech data or synthesized speech data. Keyframe markers are used to identify speech frames in the speech sample data.

[0081] Step S102: Obtain the initial keyword detection model, which includes an initial feature extraction sub-model and an initial adversarial classification sub-model.

[0082] Step S103: Based on the initial feature extraction sub-model, feature extraction is performed on the speech sample data to obtain the sample speech features;

[0083] Step S104: Based on the initial feature extraction sub-model and sample speech features, perform keyword detection on the speech sample data to obtain predicted keywords;

[0084] Step S105: Calculate feature extraction loss data based on predicted keywords and keyframe markers;

[0085] Step S106: Based on the initial adversarial classification sub-model and sample speech features, type identification is performed on the speech sample data to obtain the speech prediction category, wherein the speech prediction category is used to indicate whether the speech sample data is real speech data or synthetic speech data.

[0086] Step S107: Calculate adversarial classification loss data based on speech prediction category and speech label;

[0087] Step S108: Based on the feature extraction loss data and adversarial classification loss data, perform gradient inversion on the initial keyword detection model to obtain the target keyword detection model;

[0088] Step S109: Based on the target keyword detection model, perform keyword detection on the target speech data to obtain the target keywords.

[0089] Steps S101 to S109 as shown in the embodiments of this application involve: using the acquired initial feature extraction sub-model to extract features from the acquired speech sample data to obtain sample speech features; then, based on the initial feature extraction sub-model and the sample speech features, performing keyword detection on the speech sample data to obtain predicted keywords; and calculating feature extraction loss data based on the predicted keywords and keyframe markers. Next, using the acquired initial adversarial classification sub-model and the aforementioned sample speech features, performing type identification on the speech sample data to obtain a speech prediction category indicating whether the speech sample data is real speech data or synthetic speech data; and calculating adversarial classification loss data based on the speech prediction category and speech tags. Finally, performing gradient inversion on the initial keyword detection model based on the feature extraction loss data and the adversarial classification loss data to obtain a target keyword detection model; and performing keyword detection on the target speech data based on the target keyword detection model to obtain target keywords. Therefore, this application uses an initial feature extraction sub-model to extract features from speech sample data to obtain sample speech features. Then, based on the initial feature extraction sub-model and the sample speech features, keyword detection is performed on the speech sample data to obtain predicted keywords. Based on the predicted keywords and keyframe markers, feature extraction loss data is calculated, enabling keyword detection on both real and synthesized speech data. Furthermore, using an initial adversarial classification sub-model and sample speech features, type recognition is performed on the speech sample data to obtain the predicted speech category. Based on the predicted speech category and speech labels, adversarial classification loss data is calculated. Based on the feature extraction loss data and adversarial classification loss data, gradient inversion is performed on the initial keyword detection model, making the speech features extracted by the initial feature extraction sub-model more similar to the common features of synthesized and real speech data. This reduces the requirement for real speech data to train the keyword detection model, thereby reducing the cost of speech keyword detection.

[0090] In step S101 of some embodiments, the real speech data can be speech data obtained by manual speech recording. The synthesized speech data can be speech data formed based on speech synthesis technology.

[0091] This application can obtain real speech data through manual speech recording, and then perform data labeling on the real speech data to obtain real labels and real keyframe markers. Based on the real speech data, speech simulation is performed to generate simulated speech data. Furthermore, the simulated speech data is further labeled with synthesized speech and keyframe markers to obtain synthesized speech data containing synthesized labels and synthesized keyframe markers. Finally, the above real speech data and synthesized speech data are merged to obtain speech sample data.

[0092] In detail, in some embodiments, step S101 may include, but is not limited to, steps S201 to S205:

[0093] Step S201: Obtain recorded voice data;

[0094] Step S202: Data identification is performed on the recorded voice data to obtain real voice data. The real voice data includes real tags and real keyframe markers. The real tags are used to indicate that the manually recorded data is real voice data, and the real keyframe markers are used to identify the voice frames in the real voice data.

[0095] Step S203: Perform speech simulation on the real speech data to obtain simulated speech data;

[0096] Step S204: Data identification is performed on the simulated speech data to obtain synthesized speech data. The synthesized speech data includes synthesis tags and synthesis keyframe markers. The synthesis tags are used to indicate that the simulated speech data is synthesized speech data, and the synthesis keyframe markers are used to identify the speech frames in the synthesized speech data.

[0097] Step S205: Merge the real speech data and the synthesized speech data to obtain speech sample data.

[0098] In steps S201 and S202 of some embodiments, by acquiring manually recorded voice data, recorded voice data can be obtained. Then, by tagging the recorded voice data, real tags can be obtained to indicate that the recorded voice data is real voice data, as well as real keyframe tags that identify key voice frames in the real voice data.

[0099] In step S203 of some embodiments, real speech data is converted into speech-text data, and then the speech-text data is modified to obtain various enhanced text data. The enhanced text data is then converted into speech data to obtain simulated speech data.

[0100] In detail, in some embodiments, step S203 may include, but is not limited to, steps S301 to S303:

[0101] Step S301: Transcribe the real speech data to obtain speech text data;

[0102] Step S302: Perform data augmentation on the speech-text data to obtain augmented text data;

[0103] Step S303: Speech synthesis is performed based on the enhanced text data to obtain simulated speech data.

[0104] In step S301 of some embodiments, the real speech data is preprocessed by noise removal, audio segmentation and other audio data preprocessing to obtain standard speech data. Then, feature extraction is performed on the standard speech data to convert the audio data into feature vectors. The feature vectors are then input into a trained long short-term memory network, convolutional neural network or deep learning model based on a converter architecture to achieve speech transcription of the real speech data, thereby obtaining speech text data.

[0105] In step S302 of some embodiments, data enhancement of speech text data can be achieved by replacing keywords in speech text data, thereby obtaining enhanced text data. For example, in the scenario of sales personnel promoting insurance business, replacing phrases such as "investment return" and "diversified investment" that appear in speech text data with synonyms or near-synonyms can achieve data enhancement of speech text data, thereby obtaining enhanced text data.

[0106] It should be noted that, in the embodiments of this application, real speech data can also be processed in a model that has speech transcription and data enhancement capabilities. For example, the Virtousso model can be used to perform speech transcription and data enhancement on real speech data.

[0107] In step S303 of some embodiments, a language model-based audio generation model can be used to generate simulated speech data based on enhanced text data. Specifically, the audio generation model can be AudioLM (Audio Language Model).

[0108] In steps S301 to S303 of this embodiment, speech text data is obtained by transcribing real speech data, data augmentation is performed on the speech text data to obtain augmented text data, and then speech synthesis is performed based on the augmented text data to obtain simulated speech data. This reduces the need for real speech data, thereby reducing the cost of model training.

[0109] In step S204 of some embodiments, by tagging the simulated speech data, it is possible to obtain a representation that the simulated speech data is synthetic speech data. By tagging the simulated speech data with speech frames, it is possible to obtain a synthetic keyframe marker that represents the key speech frames in the synthetic speech data. Through the above-mentioned synthetic tags and synthetic keyframe markers, the simulated speech data can be converted into synthetic speech data containing synthetic tags and synthetic keyframe markers.

[0110] In step S205 of some embodiments, the above-mentioned real speech data and synthesized speech data are combined into a dataset to obtain speech sample data.

[0111] In steps S201 to S205 of this embodiment, the acquired recorded speech data is labeled to obtain real speech data including real labels and real keyframe markers. Then, speech simulation is performed on the real speech data to obtain simulated speech data. Further, the simulated speech data is labeled to obtain synthesized speech data including synthesized labels and synthesized keyframe markers. Finally, the real speech data and the synthesized speech data are merged to obtain speech sample data. This reduces the need for real speech data in training the model, thereby reducing the cost of model training.

[0112] In step S102 of some embodiments, the initial keyword detection model is used to detect keywords in the input speech data. Specifically, the initial keyword detection model can be divided into an initial feature extraction sub-model and an initial adversarial classification sub-model. The initial feature extraction sub-model is used to extract speech features from the speech data and then extract keywords based on the speech features. The initial adversarial classification sub-model is used to determine whether the input speech data is real speech data or synthetic speech data based on the speech features extracted by the initial feature extraction sub-model.

[0113] This application enables the use of speech features extracted by the initial feature extraction sub-model as input data for the initial adversarial classification sub-model by connecting the input of the initial adversarial classification sub-model with the feature extraction output interface of the initial feature extraction sub-model.

[0114] In step S103 of some embodiments, this application can convert speech sample data into Mel spectrum, and then determine the sample speech features of the speech sample data based on the amplitude, frequency and other characteristics of the Mel spectrum, thereby realizing feature extraction of speech sample data using the initial feature extraction sub-model.

[0115] In step S104 of some embodiments, the initial feature extraction sub-model can be used to perform contextual understanding on the extracted sample speech features to obtain contextual information of the sample speech features. Then, combined with the contextual information, the sample speech features are classified as keywords to obtain speech feature categories. Then, a pre-set activation function is used to perform linear mapping on the speech feature categories to obtain the keyword sample probability that the sample speech feature is a keyword. Finally, based on the keyword sample probability, some features are selected from the sample speech features as predicted keywords.

[0116] In step S105 of some embodiments, speech transcription processing is performed on the keyframe-marked regions in the speech sample data to obtain actual speech keywords. Then, the predicted keywords are compared with the actual speech keywords to obtain feature extraction loss data representing the difference between the predicted keywords and the actual speech keywords.

[0117] In step S106 of some embodiments, the category of sample speech features is identified using an initial adversarial classification sub-model, which can obtain the sample feature category. Then, based on the identified sample feature category, the sample speech features are traced to obtain the type classification of each speech sample data, i.e., the speech prediction category.

[0118] For details, please refer to Figure 4 In some embodiments, step S106 may include, but is not limited to, steps S401 to S402:

[0119] Step S401: Based on the initial adversarial classification sub-model, perform feature recognition on the sample speech features to obtain the sample feature category;

[0120] Step S402: Based on the sample feature category, perform data tracing on the sample speech features to obtain the speech prediction category.

[0121] In step S401 of some embodiments, since there are similarities and differences between synthetic speech data and real speech data, in the initial stage of model training, the initial feature extraction sub-model often extracts the differences between the two. Therefore, the initial adversarial classification sub-model can perform feature classification on the input sample speech features based on the differences between the pre-learned synthetic speech data and real speech data, thereby obtaining the sample feature category.

[0122] In step S402 of some embodiments, after obtaining the sample feature category, the speech sample data can be classified by tracing the speech sample data represented by the sample speech features, thereby obtaining the speech prediction category.

[0123] In steps S401 to S402 of this embodiment, the initial adversarial classification sub-model can be used to identify the features of the sample speech features, obtain the sample feature category, and then trace the sample speech features according to the sample feature category. This can realize the type identification of speech sample data, thereby helping the initial keyword extraction model to distinguish between synthetic speech data and real speech data.

[0124] In step S107 of some embodiments, the actual speech category of the speech sample data can be determined by the speech labels in the speech sample data, and the adversarial classification loss data of the initial adversarial classification sub-model can be obtained by calculating the difference between the predicted speech category and the actual speech category.

[0125] In step S108 of some embodiments, the feature extraction loss data and adversarial classification loss data are calculated during backpropagation of the initial keyword detection model. The gradient is then inverted and passed to the initial feature extraction sub-model, so that the initial feature extraction sub-model and the initial adversarial classification sub-model form an adversarial relationship. This can increase the probability that the initial feature extraction sub-model extracts common features between synthetic speech data and real speech data, making it difficult for the initial adversarial classification sub-model to distinguish the speech category of the speech sample data. Thus, it is possible to use synthetic speech data instead of real speech data for training the initial keyword detection model.

[0126] It's important to know that the above operations can be accomplished by adding a gradient inversion layer to the initial keyword detection model.

[0127] For details, please refer to Figure 5 In some embodiments, step S108 may include, but is not limited to, steps S501 to S504:

[0128] Step S501: Based on the feature extraction loss data and the preset model parameters, calculate the feature extraction gradient data, where the model parameters refer to the parameters in the initial keyword detection model;

[0129] Step S502: Calculate the adversarial classification gradient data based on the adversarial classification loss data and model parameters;

[0130] Step S503: Invert the adversarial classification gradient data to obtain inverted gradient data;

[0131] Step S504: Based on the feature extraction gradient data, adversarial classification gradient data, and inverted gradient data, backpropagation is performed on the initial keyword detection model to obtain the target keyword detection model.

[0132] In steps S501 and S502 of some embodiments, based on the feature extraction loss data and the adversarial classification loss data, the rate of change of the feature extraction loss function and the adversarial classification loss function for each parameter in the initial keyword detection model under the current parameter value can be obtained, i.e., the feature extraction gradient data and the adversarial classification gradient data. The feature extraction loss function is a function that calculates the feature extraction loss data based on the predicted keywords and keyframe tags, and the adversarial classification loss function is a function that calculates the adversarial classification loss data based on the speech prediction category and speech label.

[0133] In step S503 of some embodiments, by reversing the adversarial classification gradient data, reversed adversarial classification gradient data, i.e., reversed gradient data, can be obtained so that the initial feature extraction submodule can make parameter adjustments to compete with the initial adversarial classification submodel based on the reversed gradient data.

[0134] In step S504 of some embodiments, the parameters of the initial feature extraction sub-model and the initial adversarial classification sub-model are adjusted by backpropagating the aforementioned feature extraction gradient data, adversarial classification gradient data, and inverted gradient data to the initial feature extraction sub-model and the initial adversarial classification sub-model, thereby obtaining the target keyword detection model.

[0135] For details, please refer to Figure 6 In some embodiments, step S504 may include, but is not limited to, steps S601 to S603:

[0136] Step S601: Based on the adversarial classification gradient data and feature extraction gradient data, adjust the parameters of the initial adversarial classification sub-model to obtain the target adversarial classification sub-model;

[0137] Step S602: Based on the feature extraction gradient data and the inverted gradient data, adjust the parameters of the initial feature extraction sub-model to obtain the target feature extraction sub-model;

[0138] Step S603: The target feature extraction sub-model and the target adversarial classification sub-model are concatenated to obtain the target keyword detection model.

[0139] In step S601 of some embodiments, the adversarial classification gradient data and feature extraction gradient data are combined to obtain forward combined gradient data. Then, based on the forward combined gradient data, the parameters of the initial adversarial classification sub-model are adjusted, which enables the initial adversarial classification sub-model to identify the category of speech sample data more accurately, thereby obtaining the target adversarial classification sub-model.

[0140] In step S602 of some embodiments, the inverted gradient data and the feature extraction gradient data are combined to obtain inversely combined gradient data. Then, based on the inversely combined gradient data, the parameters of the initial feature extraction sub-model are adjusted so that the initial feature extraction model tends to prevent the initial adversarial classification sub-model from enhancing its ability to distinguish the category of speech sample data. This helps the initial feature extraction model extract the common features between synthetic speech data and real speech data, forming the target feature extraction sub-model.

[0141] In step S603 of some embodiments, the target feature extraction sub-model and the target adversarial classification sub-model are spliced ​​together to obtain the trained target keyword detection model.

[0142] In steps S601 to S603 of this embodiment, the parameters of the initial adversarial classification sub-model are adjusted based on the adversarial classification gradient data and the feature extraction gradient data to obtain the target adversarial classification sub-model. Then, the parameters of the initial feature extraction sub-model are adjusted based on the feature extraction gradient data and the inverted gradient data to obtain the target feature extraction sub-model. Finally, the target feature extraction sub-model and the target adversarial classification sub-model are concatenated to obtain the target keyword detection model. This reduces the need for real speech data during the training of the target keyword detection model and reduces the cost of training the target keyword detection model.

[0143] In steps S501 to S504 of this embodiment, feature extraction gradient data is calculated based on feature extraction loss data and model parameters. Simultaneously, adversarial classification gradient data is calculated based on adversarial classification loss data and model parameters. Furthermore, the adversarial classification gradient data is inverted to obtain inverted gradient data. Then, based on the feature extraction gradient data, adversarial classification gradient data, and inverted gradient data, the initial keyword detection model is backpropagated to obtain the target keyword detection model. This allows the initial feature extraction sub-model and the initial adversarial classification sub-model to form an adversarial relationship, thereby increasing the probability that the initial feature extraction sub-model extracts common features between synthetic speech data and real speech data.

[0144] In step S109 of some embodiments, the target voice data may be voice data that has not yet been tagged and marked with keyframes. In insurance sales dialogues, the target voice data may be real-time recording data of the salesperson.

[0145] This application can utilize the aforementioned trained target keyword detection model to perform feature extraction, feature context understanding, and feature classification on target speech data, thereby obtaining the target keywords in the target speech data.

[0146] For details, please refer to Figure 7 In some embodiments, prior to step S106, the speech keyword detection method may include, but is not limited to, steps S701 to S705:

[0147] Step S701: Based on the target keyword detection model, extract features from the target speech data to obtain target speech features;

[0148] Step S702: Based on the target keyword detection model and target speech features, perform contextual understanding on the target speech data to obtain contextual information;

[0149] Step S703: Based on contextual information, classify the target speech features to obtain the target feature category;

[0150] Step S704: Based on the target feature category, perform a linear mapping on the target speech features to obtain the keyword probability, where the keyword probability is used to represent the probability that the target speech feature is a keyword;

[0151] Step S705: Based on keyword probabilities, filter the target speech features to obtain target keywords.

[0152] In step S701 of some embodiments, the method of extracting features from target speech data based on the target keyword detection model to obtain target speech features is similar to the above-mentioned method of "extracting features from speech sample data based on the initial feature extraction sub-model to obtain sample speech features", so it will not be described in detail.

[0153] In step S702 of some embodiments, by performing calculations such as vector multiplication between the target speech features and the target speech features in the context of the target speech features in the target speech data, the relationship between the target speech features and the context of the target speech features can be clarified, thereby obtaining the contextual information of the target speech features.

[0154] In step S703 of some embodiments, semantic understanding of the target speech features is performed based on the contextual information of the target speech features, which can clarify the feature category of the target speech features, thereby realizing feature classification of the target speech features and obtaining the target feature category.

[0155] In steps S704 and S705 of some embodiments, the target speech features are linearly mapped according to the target feature category by using a pre-set activation function in the target keyword detection model to obtain the keyword probability that the target speech feature is the target keyword. Then, based on the keyword probability, the target speech features that meet the preset conditions are selected from the target speech features and output, thereby obtaining the target keyword.

[0156] In steps S701 to S705 of this embodiment, feature extraction is performed on the target speech data according to the target keyword detection model to obtain target speech features. Then, context understanding is performed on the target speech data according to the target keyword detection model and the target speech features to obtain context information. Further, the target speech features are classified according to the context information to obtain target feature categories. Next, the target speech features are linearly mapped according to the target feature categories to obtain keyword probabilities. Here, keyword probability is used to represent the probability that the target speech feature is a keyword. Finally, the target speech features are filtered according to the keyword probabilities. This method can quickly and conveniently extract keywords from target speech data and improve the efficiency of speech keyword detection.

[0157] This application uses an initial feature extraction sub-model to extract features from speech sample data, obtaining sample speech features. Then, based on the initial feature extraction sub-model and the sample speech features, keyword detection is performed on the speech sample data to obtain predicted keywords. Feature extraction loss data is calculated based on the predicted keywords and keyframe markers, enabling keyword detection on both real and synthesized speech data. Furthermore, using an initial adversarial classification sub-model and sample speech features, type recognition is performed on the speech sample data to obtain the predicted speech category. Adversarial classification loss data is then calculated based on the predicted speech category and speech labels. Gradient inversion is performed on the initial keyword detection model based on the feature extraction loss data and the adversarial classification loss data, making the speech features extracted by the initial feature extraction sub-model more closely resemble the common features of synthesized and real speech data. This reduces the need for real speech data to train the keyword detection model, thereby reducing the cost of speech keyword detection.

[0158] Please see Figure 8 This application also provides a voice keyword detection device that can implement the above-described voice keyword detection method. The device includes:

[0159] The sample data acquisition module 801 is used to acquire speech sample data containing speech tags and keyframe markers. The speech sample data includes real speech data and synthesized speech data. Speech tags are used to indicate whether the speech sample data is real speech data or synthesized speech data, and keyframe markers are used to identify speech frames in the speech sample data.

[0160] The initial model acquisition module 802 is used to acquire the initial keyword detection model, which includes an initial feature extraction sub-model and an initial adversarial classification sub-model.

[0161] The speech feature extraction module 803 is used to extract features from speech sample data based on the initial feature extraction sub-model to obtain sample speech features;

[0162] The keyword prediction module 804 is used to detect keywords in speech sample data based on the initial feature extraction sub-model and sample speech features, and obtain predicted keywords.

[0163] The first loss calculation module 805 is used to calculate feature extraction loss data based on predicted keywords and keyframe markers.

[0164] The speech data classification module 806 is used to identify the type of speech sample data based on the initial adversarial classification sub-model and the sample speech features to obtain the speech prediction category, wherein the speech prediction category is used to indicate whether the speech sample data is real speech data or synthetic speech data.

[0165] The second loss calculation module 807 is used to calculate adversarial classification loss data based on speech prediction category and speech label;

[0166] The model parameter adjustment module 808 is used to perform gradient inversion on the initial keyword detection model based on feature extraction loss data and adversarial classification loss data to obtain the target keyword detection model.

[0167] The keyword detection module 809 is used to detect keywords in target speech data based on the target keyword detection model to obtain target keywords.

[0168] The specific implementation of this voice keyword detection device is basically the same as the specific implementation of the above-described voice keyword detection method, and will not be repeated here.

[0169] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described voice keyword detection method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0170] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0171] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0172] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the voice keyword detection method of the embodiments of this application.

[0173] The input / output interface 903 is used to implement information input and output;

[0174] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0175] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0176] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0177] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described speech keyword detection method.

[0178] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0179] The speech keyword detection method, speech keyword detection device, electronic device, and storage medium provided in this application embodiment extract features from the acquired speech sample data using an initial feature extraction sub-model to obtain sample speech features. Then, based on the initial feature extraction sub-model and the sample speech features, keyword detection is performed on the speech sample data to obtain predicted keywords. Feature extraction loss data is calculated based on the predicted keywords and keyframe markers. Next, using the acquired initial adversarial classification sub-model and the aforementioned sample speech features, type identification is performed on the speech sample data to obtain a speech prediction category indicating whether the speech sample data is real speech data or synthetic speech data. Adversarial classification loss data is calculated based on the speech prediction category and speech tags. Finally, gradient inversion is performed on the initial keyword detection model based on the feature extraction loss data and the adversarial classification loss data to obtain a target keyword detection model. Keyword detection is then performed on the target speech data based on the target keyword detection model to obtain target keywords.

[0180] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0181] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0182] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0183] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0184] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0185] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0186] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The coupling or direct coupling or communication connection between the shown or discussed units may be through some interfaces, or indirect coupling or communication connection between the apparatus or units, and may be electrical, mechanical, or other forms.

[0187] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0189] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0190] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for detecting speech keywords, characterized in that, The method includes: Acquire speech sample data containing speech tags and keyframe markers, wherein the speech sample data includes real speech data and synthesized speech data, the speech tags are used to indicate whether the speech sample data is real speech data or synthesized speech data, and the keyframe markers are used to identify speech frames in the speech sample data; Obtain an initial keyword detection model, wherein the initial keyword detection model includes an initial feature extraction sub-model and an initial adversarial classification sub-model; Based on the initial feature extraction sub-model, feature extraction is performed on the speech sample data to obtain sample speech features; Based on the initial feature extraction sub-model and the sample speech features, keyword detection is performed on the speech sample data to obtain predicted keywords; Based on the predicted keywords and the keyframe markers, feature extraction loss data is calculated. Based on the initial adversarial classification sub-model and the sample speech features, the speech sample data is type-identified to obtain a speech prediction category, wherein the speech prediction category is used to indicate whether the speech sample data is real speech data or synthetic speech data; Calculate the adversarial classification loss data based on the predicted speech category and the speech label; Based on the feature extraction loss data and the adversarial classification loss data, the gradient inversion of the initial keyword detection model is performed to obtain the target keyword detection model. Based on the target keyword detection model, keyword detection is performed on the target speech data to obtain the target keywords.

2. The method according to claim 1, characterized in that, The step of performing gradient inversion on the initial keyword detection model based on the feature extraction loss data and the adversarial classification loss data to obtain the target keyword detection model includes: Based on the feature extraction loss data and the preset model parameters, the feature extraction gradient data is calculated, wherein the model parameters refer to the parameters in the initial keyword detection model; Based on the adversarial classification loss data and the model parameters, calculate the adversarial classification gradient data; The adversarial classification gradient data is inverted to obtain inverted gradient data; Based on the feature extraction gradient data, the adversarial classification gradient data, and the inverted gradient data, the initial keyword detection model is backpropagated to obtain the target keyword detection model.

3. The method according to claim 2, characterized in that, The step of backpropagating the initial keyword detection model based on the feature extraction gradient data, the adversarial classification gradient data, and the inverted gradient data to obtain the target keyword detection model includes: Based on the adversarial classification gradient data and the feature extraction gradient data, the parameters of the initial adversarial classification sub-model are adjusted to obtain the target adversarial classification sub-model. Based on the feature extraction gradient data and the inverted gradient data, the parameters of the initial feature extraction sub-model are adjusted to obtain the target feature extraction sub-model; The target feature extraction sub-model and the target adversarial classification sub-model are concatenated to obtain the target keyword detection model.

4. The method according to claim 1, characterized in that, The acquisition of speech sample data containing speech tags and keyframe markers includes: Acquire recorded audio data; The recorded audio data is labeled to obtain the real audio data, wherein the real audio data includes real tags and real keyframe markers. The real tags are used to indicate that the recorded audio data is real audio data, and the real keyframe markers are used to identify audio frames in the real audio data. The real speech data is subjected to speech simulation to obtain simulated speech data; The simulated speech data is labeled to obtain the synthesized speech data, wherein the synthesized speech data includes a synthesis tag and a synthesis keyframe marker. The synthesis tag is used to indicate that the simulated speech data is the synthesized speech data, and the synthesis keyframe marker is used to identify the speech frames in the synthesized speech data. The real speech data and the synthesized speech data are merged to obtain the speech sample data.

5. The method according to claim 4, characterized in that, The process of performing speech simulation on the real speech data to obtain simulated speech data includes: The real speech data is transcribed to obtain speech-text data; The speech-text data is augmented to obtain augmented text data; Speech synthesis is performed based on the enhanced text data to obtain the simulated speech data.

6. The method according to any one of claims 1-5, characterized in that, The step of detecting keywords in the target speech data based on the target keyword detection model to obtain target keywords includes: Based on the target keyword detection model, feature extraction is performed on the target speech data to obtain target speech features; Based on the target keyword detection model and the target speech features, the target speech data is subjected to contextual understanding to obtain contextual information; Based on the context information, the target speech features are classified to obtain the target feature category; Based on the target feature category, the target speech feature is linearly mapped to obtain the keyword probability, wherein the keyword probability is used to represent the probability that the target speech feature is a keyword; Based on the keyword probabilities, the target speech features are filtered to obtain the target keywords.

7. The method according to any one of claims 1-5, characterized in that, The step of performing type identification on the speech sample data based on the initial adversarial classification sub-model and the sample speech features to obtain the speech prediction category includes: Based on the initial adversarial classification sub-model, feature recognition is performed on the sample speech features to obtain the sample feature categories; Based on the sample feature category, data tracing is performed on the sample speech features to obtain the speech prediction category.

8. A speech keyword detection device, characterized in that, The device includes: The sample data acquisition module is used to acquire speech sample data containing speech tags and keyframe markers. The speech sample data includes real speech data and synthesized speech data. The speech tags are used to indicate whether the speech sample data is real speech data or synthesized speech data. The keyframe markers are used to identify the speech frames in the speech sample data. An initial model acquisition module is used to acquire an initial keyword detection model, wherein the initial keyword detection model includes an initial feature extraction sub-model and an initial adversarial classification sub-model; The speech feature extraction module is used to extract features from the speech sample data based on the initial feature extraction sub-model to obtain sample speech features; The keyword prediction module is used to detect keywords in the speech sample data based on the initial feature extraction sub-model and the sample speech features to obtain predicted keywords; The first loss calculation module is used to calculate features and extract loss data based on the predicted keywords and the keyframe markers. The speech data classification module is used to perform type identification on the speech sample data based on the initial adversarial classification sub-model and the sample speech features to obtain a speech prediction category, wherein the speech prediction category is used to indicate whether the speech sample data is real speech data or synthetic speech data; The second loss calculation module is used to calculate adversarial classification loss data based on the speech prediction category and the speech label. The model parameter adjustment module is used to perform gradient inversion on the initial keyword detection model based on the feature extraction loss data and the adversarial classification loss data to obtain the target keyword detection model. The keyword detection module is used to detect keywords in the target speech data based on the target keyword detection model to obtain the target keywords.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the speech keyword detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech keyword detection method according to any one of claims 1 to 7.

Images