Audio content detection method, device, equipment, medium and product

Abstract

The invention discloses an audio content detection method, device and equipment, a medium and a product, and relates to the technical field of artificial intelligence, in particular to the technical field of audio feature processing and content security. The method comprises the following steps: performing embedding processing on a to-be-detected audio through a pronunciation embedding model to obtain a target embedding vector; wherein the pronunciation embedding model is obtained by finely adjusting an initial embedding model through an audio triple of a sample sensitive word; performing similarity calculation on the target embedding vector and a core embedding vector in the sensitive pronunciation embedding set to obtain a similarity value; determining a signal-to-noise ratio estimation value and a dialect type of the audio to be detected, and determining a sensitivity threshold value; and determining a sensitive word detection result of the to-be-detected audio according to the similarity value and the sensitivity threshold. According to the technical scheme, audio real-time auditing in dialect, noise and homophone interference scenes can be improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more particularly to the field of audio feature processing and content security technology, specifically to an audio content detection method, apparatus, device, medium, and product. Background Technology

[0002] With the development of artificial intelligence technology, existing audio sensitive word detection is mostly based on the logic of "text matching". That is, after converting audio into text through Automatic Speech Recognition (ASR), it is matched with a preset sensitive word library. This solution has significant defects in complex scenarios: 1) Homophone / near-homophone misjudgment: For example, the sensitive word "gambling" is similar in pronunciation to the normal word "museum", which is easily confused by ASR, leading to misjudgment or missed judgment; 2) Poor dialect adaptation: The pronunciation rules of dialects (such as Cantonese and Sichuanese) are very different from those of Mandarin, resulting in low ASR transcription accuracy and the inability to capture sensitive words through text matching; 3) Weak robustness in noisy environments: In noisy scenarios (such as outdoor broadcasts and background noise in conferences), ASR transcription is prone to character omissions, resulting in the inability to match sensitive word segments; 4) Reliance on text transcription: If ASR cannot recognize the audio (such as pronunciation without clear semantics), even if it contains the pronunciation of sensitive words, it cannot pass the text matching detection.

[0003] In summary, existing text-dependent sensitive word detection methods cannot cope with cross-scene interference such as homophones, dialects, and noise, and there is an urgent need for a sensitive word detection scheme based on acoustic features. Summary of the Invention

[0004] This invention provides an audio content detection method, apparatus, device, medium, and product to improve the accuracy of audio-sensitive content detection.

[0005] According to one aspect of the present invention, an audio content detection method is provided, the method comprising:

[0006] The target embedding vector is obtained by embedding the audio to be detected using a pronunciation embedding model; wherein, the pronunciation embedding model is obtained by fine-tuning the initial embedding model using audio triples of sample sensitive words;

[0007] The similarity between the target embedding vector and the core embedding vector in the sensitive pronunciation embedding set is calculated to obtain a similarity value;

[0008] Determine the signal-to-noise ratio estimate and dialect type of the audio to be detected, and determine the sensitivity threshold;

[0009] Based on the similarity value and the sensitivity threshold, the sensitive word detection result of the audio to be detected is determined.

[0010] According to another aspect of the present invention, an audio content detection apparatus is provided, the apparatus comprising:

[0011] The target embedding vector determination module is used to perform embedding processing on the audio to be detected through a pronunciation embedding model to obtain a target embedding vector; wherein, the pronunciation embedding model is obtained by fine-tuning the initial embedding model through audio triples of sample sensitive words;

[0012] The similarity determination module is used to calculate the similarity between the target embedding vector and the core embedding vector in the sensitive pronunciation embedding set, and obtain a similarity value;

[0013] The sensitivity threshold determination module is used to determine the signal-to-noise ratio estimate and dialect type of the audio to be detected, and to determine the sensitivity threshold.

[0014] The detection result determination module is used to determine the sensitive word detection result of the audio to be detected based on the similarity value and the sensitivity threshold.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and

[0017] A memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the audio content detection method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the audio content detection method according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the audio content detection method according to any embodiment of the present invention.

[0021] The technical solution of this invention involves embedding the audio to be detected using a pronunciation embedding model to obtain a target embedding vector. The pronunciation embedding model is fine-tuned by using audio triples of sensitive words from samples. Similarity is calculated between the target embedding vector and the core embedding vectors in the sensitive pronunciation embedding set to obtain a similarity value. The signal-to-noise ratio estimate and dialect type of the audio to be detected are determined, and a sensitivity threshold is established. Based on the similarity value and the sensitivity threshold, the sensitive word detection result of the audio to be detected is determined. This technical solution enables cross-scenario audio sensitivity detection in various scenarios, such as broadcasting, outdoor audio, and dialect programs, and improves the accuracy of sensitivity detection.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of an audio content detection method provided according to an embodiment of the present invention;

[0025] Figure 2 This is a flowchart of an audio content detection method provided according to an embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of an audio content detection device according to an embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the audio content detection method of this invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Furthermore, it should be noted that the collection, storage, use, processing, transmission, provision, and disclosure of related data such as the audio triplet of the audio to be detected and the sensitive words in the sample in the technical solution of the present invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0031] Figure 1 This is a flowchart of an audio content detection method according to an embodiment of the present invention. This embodiment is applicable to audio review scenarios involving dialects, noise, and homophone interference. The method can be executed by an audio content detection device, which can be implemented in hardware and / or software. This device can be configured in an electronic device carrying audio content detection functionality, such as a server. Figure 1 As shown, the method includes:

[0032] S110. The audio to be detected is embedded using a pronunciation embedding model to obtain the target embedding vector.

[0033] In this embodiment, the pronunciation embedding model is used to extract the audio embedding vector; optionally, the pronunciation embedding model is obtained by fine-tuning the initial embedding model using the audio triples of sample sensitive words.

[0034] The audio to be detected refers to the audio whose content needs to be detected for sensitive words. The target embedding vector refers to the embedding vector corresponding to the audio to be detected.

[0035] One alternative approach is to input the audio to be detected into a pronunciation embedding model, and then perform feature extraction, weighted mapping, and other processing on the model to obtain the target embedding vector of the audio to be detected.

[0036] Another alternative is to perform frame-by-frame processing on the audio to be detected, for example, with a frame length of 25ms and a frame shift of 10ms, to obtain multiple frames of audio. For each frame of audio, the audio is input into the pronunciation embedding model, and after the model performs feature extraction, weighted mapping and other processing, the target embedding vector corresponding to that frame of audio is obtained.

[0037] S120. Calculate the similarity between the target embedding vector and the core embedding vector in the sensitive pronunciation embedding set to obtain the similarity value.

[0038] In this embodiment, the sensitive pronunciation embedding set refers to a pre-built set of embedding vectors for sensitive words used in audio sensitive word detection; it includes an embedding library for multiple pronunciation scenarios (Mandarin, dialects, different speaking speeds). It should be noted that the sensitive pronunciation embedding set can also be updated during real-time sensitive word detection. The core embedding vector refers to the cluster center of each type of audio in the sensitive pronunciation embedding set.

[0039] Specifically, the cosine similarity between the target embedding vector and each core embedding vector in the sensitive word pronunciation embedding set is calculated to obtain the similarity value.

[0040] S130. Determine the signal-to-noise ratio estimate and dialect type of the audio to be detected, and determine the sensitivity threshold.

[0041] Specifically, the signal-to-noise ratio (SNR) estimate of the audio to be detected is calculated. Then, the dialect type of the audio is determined based on a dialect type recognition model. Based on the correspondence between the dialect type, the SNR estimate, and the threshold, a sensitivity threshold is determined. The speech type recognition model can be determined based on existing audio recognition models; this invention does not impose specific limitations.

[0042] S140. Determine the sensitive word detection results of the audio to be detected based on the similarity value and sensitivity threshold.

[0043] Specifically, if the similarity value between the core embedding vector and the target embedding vector in the sensitive pronunciation embedding set of a certain sensitive word is greater than the sensitivity threshold, then the audio to be detected is determined to contain that sensitive word. The sensitive word type, occurrence timestamp, and similarity value are used as the sensitive word detection result for the audio to be detected. The occurrence timestamp refers to the time when the sensitive word appears in the audio to be detected.

[0044] Furthermore, the results of sensitive word detection are statistically analyzed. If the misjudgment rate of a certain sensitive word is greater than the misjudgment threshold, such as 5%, then comparative pronunciation samples of the sensitive word and easily confused words are added, and the pronunciation embedding model is lightly fine-tuned and updated to reduce the subsequent misjudgment rate.

[0045] The technical solution of this invention involves embedding the audio to be detected using a pronunciation embedding model to obtain a target embedding vector. The pronunciation embedding model is fine-tuned by using audio triples of sensitive words from samples. Similarity is calculated between the target embedding vector and the core embedding vectors in the sensitive pronunciation embedding set to obtain a similarity value. The signal-to-noise ratio estimate and dialect type of the audio to be detected are determined, and a sensitivity threshold is established. Based on the similarity value and the sensitivity threshold, the sensitive word detection result of the audio to be detected is determined. This technical solution enables cross-scenario audio sensitivity detection in various scenarios, such as broadcasting, outdoor audio, and dialect programs, and improves the accuracy of sensitivity detection.

[0046] Based on the above embodiments, as an optional approach of the present invention, the sensitive pronunciation embedding set is determined in the following manner: collecting sensitive word pronunciation samples of the target sensitive word; using a pronunciation embedding model to extract embedding features from the sensitive word pronunciation samples to obtain sample embedding vectors; and clustering the sample embedding vectors to obtain the sensitive pronunciation embedding set of the target sensitive word.

[0047] Here, the sample embedding vector refers to the embedding vector corresponding to the sensitive word.

[0048] Specifically, the process involves collecting pronunciation samples of target sensitive words. For each target sensitive word (e.g., "violation"), over 1000 pronunciation samples are collected, including samples from different language types, speech rates, and noise environments. Language types include Mandarin and over 30 mainstream dialects (Cantonese, Sichuanese, Wu, etc.); speech rates include slow (150 words / minute), medium (200 words / minute), and fast (250 words / minute); and noise environments include no noise, low noise (20dB SNR), and medium noise (10dB SNR). Then, a pronunciation embedding model is used to extract embedding features from the sensitive word pronunciation samples, resulting in sample embedding vectors. All sample embedding vectors for each target sensitive word are clustered (using the K-Means algorithm, with 50 clusters) to generate a "core pronunciation embedding set" for that sensitive word, which is then stored in a database.

[0049] Figure 2 This is a flowchart of an audio content detection method according to an embodiment of the present invention; this embodiment further elaborates on the fine-tuning process of the pronunciation embedding model based on the above embodiment. Figure 2 As shown, the method includes:

[0050] S210. The audio to be detected is embedded using a pronunciation embedding model to obtain the target embedding vector.

[0051] The pronunciation embedding model is obtained by fine-tuning the initial embedding model using audio triples of sample sensitive words.

[0052] S220. Calculate the similarity between the target embedding vector and the core embedding vector in the sensitive pronunciation embedding set to obtain the similarity value.

[0053] S230. Determine the signal-to-noise ratio estimate and dialect type of the audio to be detected, and determine the sensitivity threshold.

[0054] S240. Determine the sensitive word detection results of the audio to be detected based on the similarity value and sensitivity threshold.

[0055] In this embodiment, the initial embedding model includes a backbone network, a weighted feature fusion layer, a projector head, and an attention pooling layer. The backbone network comprises a low-level Transformer encoder and a high-level Transformer encoder. The projector head includes two fully connected layers. A batch normalization layer is added after the first fully connected layer to accelerate convergence and prevent gradient explosion. L2 normalization is applied to the features output by the second fully connected layer to ensure the embedding vector lies on a unit hypersphere, facilitating subsequent cosine similarity calculation. During training, a dropout (p=0.1) is added after the first fully connected layer to prevent overfitting. The attention pooling layer includes a single-head self-attention mechanism. The backbone network extracts low-level and deep features from the audio. The weighted feature fusion layer fuses the features output from each layer in the backbone network. The projector head performs dimensionality reduction mapping on the features output by the weighted feature fusion layer. The attention pooling layer performs weighted aggregation on the mapped features output by the projector head at the frame level.

[0056] In one alternative approach, the pronunciation embedding model is trained as follows: Audio triples of sample sensitive words are constructed; each audio triple includes a sample pronunciation segment, a forward pronunciation segment, and a reverse pronunciation segment; the audio frames of each audio triple are input into the backbone network to obtain the output features of each layer of the backbone network; the output features of each layer of the backbone network are fused through a weighted feature fusion layer to obtain fused features; the fused features are dimensionality-reduced and mapped using a projection head to obtain the frame-level mapping features of the audio triples; the frame-level mapping features of the audio triples are weighted and aggregated through an attention pooling layer to obtain segment-level embedding vectors; the training loss is determined based on the scaling factor, relaxation boundary, positive sample weights, negative sample weights, and segment-level embedding vectors; and the initial embedding model is fine-tuned based on the training loss to obtain the pronunciation embedding model.

[0057] Specifically, first, construct the audio triplets of sample sensitive words, where the audio triplets include sample pronunciation segments, forward pronunciation segments, and backward pronunciation segments. The sample pronunciation segment refers to the pronunciation segment of the sample sensitive word actually collected, such as the Cantonese version of "usury", with a duration of 0.8 - 1.5 seconds. The forward pronunciation segment refers to the pronunciation segment of the sample sensitive word in another dialect, which can be generated through data augmentation methods such as pitch perturbation and time stretching, such as the noisy Mandarin version of "usury". The backward pronunciation segment refers to the pronunciation segment of the confusing word or irrelevant word of the sample sensitive word, and difficult samples with a pronunciation similarity > 0.6 can be screened through the Hard Negative Mining strategy, such as "usularge" or "convenience store".

[0058] Then, input each frame of the audio of the audio triplets into the backbone network to obtain the output features corresponding to each frame of the audio of each layer of the backbone network. The output features of each layer of the backbone network are fused through a weighted feature fusion layer to obtain the fused features. A learnable weight vector is designed in the weighted feature fusion layer , and ensure through Softmax normalization = 1; Through the fusion formula: , where is the output feature (768 - dimensional) of the i-th layer; represents the fused feature; Initialization strategy: adopt uniform initialization ( = 1 / 7), and adaptively adjust the contribution degree of each layer during the training process; Through experimental observation, after training convergence, the weights of the 8th - 10th layers are usually the largest (0.18 - 0.22), verifying the importance of middle and high - level features for pronunciation recognition.

[0059] Next, perform dimensionality reduction mapping on the fused features (768 - dimensional) through two fully - connected layers of the projection head to obtain the frame - level mapping features (256 - dimensional) of the audio triplets. The feature dimension is from 768 - dimensional - 512 - dimensional (ReLU activation) - 256 - dimensional (without activation function).

[0060] Next, perform weighted aggregation on the frame - level mapping features of the audio triplets through the attention pooling layer to obtain the segment - level embedding vectors. The segment - level embedding vectors include positive sample embedding vectors and negative sample embedding vectors. Among them, the attention pooling layer adopts the single - head Self - Attention mechanism, Query is the learnable vector q∈R 256 , Key and Value are the embedding vectors V of each frame within the window, where the attention weight , where d = 256 is the scaling factor scalingfactor; Finally, perform weighted summation to obtain the segment - level embedding vector (256 - dimensional), output = w·V.

[0061] Next, the training loss can be determined using the Circle Loss function formula, based on the scaling factor, relaxation boundary, positive sample weights, negative sample weights, and fragment-level embedding vectors. :

[0062] ;

[0063] in, For scaling factors, for example, it can be set to 80; relax the boundary. This serves as a lower bound for positive sample similarity, for example, 0.9, indicating a relaxed boundary. Lower bound for negative sample similarity, for example, 0.4; adaptive weights. For positive sample weights, Adaptive weights For negative sample weights, ,in, Indicates ReLU operation, Represents the positive sample embedding vector. This represents the negative sample embedding vector.

[0064] Understandably, compared to Triplet Loss which only focuses on a single hard sample, Circle Loss optimizes all sample pairs simultaneously, improving convergence speed by about 30%.

[0065] Finally, the initial embedding model is fine-tuned based on the training loss to obtain the pronunciation embedding model. For example, fine-tuning the initial embedding model based on the training loss to obtain the pronunciation embedding model includes: keeping the network parameters of the low-level Transformer encoder of the backbone network unchanged; querying the projection matrix and value projection matrix in the attention module of the high-level Transformer encoder of the backbone network and inserting the LoRA decomposition matrix as a bypass; and fine-tuning the initial embedding model based on the LoRA technique and the training loss to obtain the pronunciation embedding model.

[0066] Specifically, during the fine-tuning process, a freezing strategy is adopted for the backbone network. This means keeping the network parameters of the low-level Transformer encoders of the backbone network unchanged, preserving the low-level acoustic feature extraction capabilities, while selectively fine-tuning the high-level Transformer encoders in the backbone network. Specifically, only the query projection matrix and value projection matrix of the multi-head attention module are fine-tuned, while the key matrix and the fully connected feedforward network are frozen. This freezes approximately 47M parameters, leaving approximately 47M trainable parameters (before using LoRA).

[0067] Furthermore, based on the Low-Rank Adaptation (LoRA) technique, the initial embedding model is fine-tuned according to the training loss by querying the projection matrix and value projection matrix in the attention module of the high-level Transformer encoder in the backbone network, and then inserting the LoRA decomposition matrix as a bypass.

[0068] Original projection: Query the projection matrix W_Q, the value projection matrix W_V ∈ R768×768 (parameters 2×768²=1.18M per layer); perform LoRA decomposition, i.e.:

[0069] W' = W + α·(B·A), where A ∈ R 768 ˣʳ,B∈Rʳˣ 768 , rank r=8.

[0070] Comparison of parameter quantities before and after LoRA decomposition: The original required 6 layers × 2 matrices × 1.18M = 14.16M parameters for fine-tuning, while using LoRA only requires 6 × 2 × (768 × 8 × 2) ≈ 0.15M parameters, resulting in a compression ratio of 94.7%.

[0071] It should be noted that the LoRA hyperparameter settings are as follows:

[0072] Rank r=8: Balances expressive power and parameter efficiency. Experiments show that the performance of r=8 is close to that of r=16, but the number of parameters is halved.

[0073] Scaling factor α=16: controls the LoRA update magnitude, α / r=2 ensures that fine-tuning does not destroy the pre-trained weights;

[0074] Initialization: Matrix A is initialized with Gaussian initialization N(0,σ²), σ=1 / √r; Matrix B is initialized as a zero matrix to ensure that LoRA does not affect the model output in the early stage of training;

[0075] Dropout: Adds Dropout (p=0.05) after the output of matrix A to enhance generalization ability.

[0076] When fine-tuning using LoRA technology, there are advantages in modular deployment: 1) Dialect adaptation: Independent LoRA modules are trained separately for dialects such as Cantonese and Sichuanese (each module has only 0.15M parameters), and the corresponding LoRA weights are dynamically loaded during inference; 2) Storage efficiency: Only one copy of the base model (backbone network WavLM + projection head) needs to be stored (about 400MB), and the 10 dialect LoRA modules only increase the storage overhead by 1.5MB; 3) Fast iteration: When adding a new dialect scene, there is no need to retrain the entire model, only to train a new LoRA module on the new dialect dataset (completed in 1-2 hours).

[0077] For example, after constructing the audio triplet of the sample sensitive words, the method further includes: enhancing the sample pronunciation segment and the positive pronunciation segment using at least one of the enhancement methods of temporal enhancement, frequency domain enhancement, noise injection and dialect mixing.

[0078] Specifically, temporal enhancement can involve velocity perturbation (0.9x-1.1x), volume scaling (±3dB), and time masking (SpecAugment Time Masking, maximum mask length 40ms). Frequency domain enhancement can involve Mel-spectrum masking (up to 8 mel bins) and dynamic range compression. Noise injection involves superimposing real-world environmental noise (such as noise libraries from street, office, and restaurant scenes, with an SNR range of 5-20dB) and synthetic white / pink noise. Dialect mixing involves splicing together different dialect samples of the same sensitive word to construct cross-dialect generalized samples.

[0079] Understandably, enhancing sample pronunciation segments and positive pronunciation segments can improve model robustness.

[0080] Furthermore, an online hard negative sample mining strategy is adopted. During model training, a negative sample update is performed every 5 epochs: the embedding distance of the current model to all candidate negative samples in the training set is calculated; semi-hard negative samples with a distance of [0.3, 0.8] from the sample pronunciation segment are selected to replace the original simple negative samples; 10% of random negative samples are retained to avoid overfitting to hard samples.

[0081] It should be noted that some configurations for training fine-tuning are as follows:

[0082] First, the training optimizer configuration is as follows: 1) The optimizer uses AdamW (weight decay of 0.01), separating the weight decay operation to avoid over-regularization of LoRA weights; 2) Learning rate scheduling: Cosine Annealing with Warmup strategy, the learning rate increases linearly from 0 to 1e-5 in the Warmup phase (first 500 steps), and then decays to 1e-7 according to the cosine curve; 3) Differentiated learning rate: LoRA module learning rate is 1e-5, projection head learning rate is 5e-5 (new modules use a higher learning rate to accelerate convergence), frozen layer learning rate is 0; 4) Gradient clipping: the maximum gradient norm is set to 1.0 to prevent training instability.

[0083] Then, the training hyperparameters and hardware configuration are as follows: 1) Batch Size: 32 (per H100 GPU), using Gradient Accumulation to accumulate gradients over 4 steps, equivalent to Batch Size=128; 2) Training epochs: 10 epochs, requiring approximately 80,000 steps (based on 1.2 million samples); 3) Mixed precision training: enabling BF16 (Brain Floating Point 16-bit) mixed precision, improving training speed by 2.1 times and reducing GPU memory usage to 15GB; 4) Distributed training: parallel training on a single node with 4 H100 GPUs, using a Distributed Data Parallel (DDP) strategy, with a total training time of approximately 16 hours; 5) Internal event checkpoint saving: saving model checkpoints every 2000 steps, retaining models with Top-3 performance on the validation set.

[0084] Finally, the validation and early stopping mechanism: 1) Validation set: 10% of the data (120,000 triplet pairs) is reserved as the validation set, which includes unseen dialect and noise combinations; 2) Evaluation metrics: Triplet Accuracy (the ratio of anchor point distance to positive sample distance to anchor point distance to negative sample distance) and Embedding Cluster Quality (variance of embedding vectors of the same type of sample); 3) Early stopping strategy: If the validation set Triplet Accuracy does not improve for 5 consecutive epochs (improvement threshold 0.5%), the training is terminated early and the system is rolled back to the optimal checkpoint.

[0085] It should also be noted that,

[0086] Figure 3 This is a schematic diagram of an audio content detection device according to an embodiment of the present invention. This embodiment is applicable to audio review scenarios involving dialects, noise, and homophone interference. The audio content detection device can be implemented in hardware and / or software, and can be configured in an electronic device that carries audio content detection functionality, such as a server. Figure 3 As shown, the device includes:

[0087] The target embedding vector determination module 310 is used to perform embedding processing on the audio to be detected through the pronunciation embedding model to obtain the target embedding vector; wherein, the pronunciation embedding model is obtained by fine-tuning the initial embedding model through the audio triples of the sample sensitive words;

[0088] The similarity determination module 320 is used to calculate the similarity between the target embedding vector and the core embedding vector in the sensitive pronunciation embedding set to obtain a similarity value;

[0089] The sensitivity threshold determination module 330 is used to determine the signal-to-noise ratio estimate and dialect type of the audio to be detected, and to determine the sensitivity threshold.

[0090] The detection result determination module 340 is used to determine the sensitive word detection result of the audio to be detected based on the similarity value and the sensitivity threshold.

[0091] The technical solution of this invention involves embedding the audio to be detected using a pronunciation embedding model to obtain a target embedding vector. The pronunciation embedding model is fine-tuned by using audio triples of sensitive words from samples. Similarity is calculated between the target embedding vector and the core embedding vectors in the sensitive pronunciation embedding set to obtain a similarity value. The signal-to-noise ratio estimate and dialect type of the audio to be detected are determined, and a sensitivity threshold is established. Based on the similarity value and the sensitivity threshold, the sensitive word detection result of the audio to be detected is determined. This technical solution enables cross-scenario audio sensitivity detection in various scenarios, such as broadcasting, outdoor audio, and dialect programs, and improves the accuracy of sensitivity detection.

[0092] Optionally, the initial embedding model includes a backbone network, a weighted feature fusion layer, a projection head, and an attention pooling layer; wherein the backbone network includes a low-level Transformer encoder and a high-level Transformer encoder; the projection head includes two fully connected layers; and the attention pooling layer includes a single-head self-attention mechanism.

[0093] Optionally, the device also includes a model training module for:

[0094] Construct audio triples for sample sensitive words; each audio triple includes a sample pronunciation segment, a forward pronunciation segment, and a reverse pronunciation segment;

[0095] Each frame of audio from the audio triplet is input into the backbone network to obtain the output features of each layer of the backbone network.

[0096] The output features of each layer of the backbone network are fused by a weighted feature fusion layer to obtain fused features;

[0097] The fusion features are reduced in dimension by using a projection head to obtain the frame-level mapping features of the audio triples;

[0098] The frame-level mapping features of audio triples are weighted and aggregated by an attention pooling layer to obtain a segment-level embedding vector.

[0099] The training loss is determined based on the scaling factor, relaxation boundary, positive sample weights, negative sample weights, and fragment-level embedding vectors.

[0100] The initial embedding model is fine-tuned based on the training loss to obtain the pronunciation embedding model.

[0101] Optionally, the device also includes a model training module for:

[0102] Keep the network parameters of the lower-level Transformer encoders in the backbone network unchanged;

[0103] In the attention module of the high-level Transformer encoder in the backbone network, query the projection matrix and value projection matrix and bypass insert the LoRA decomposition matrix.

[0104] Based on the LoRA technique, the initial embedding model is fine-tuned according to the training loss to obtain the pronunciation embedding model.

[0105] Optionally, the device also includes a model training module, which is further used for:

[0106] After constructing the audio triplet of the sensitive words in the samples, at least one of the enhancement methods, namely temporal enhancement, frequency enhancement, noise injection, and dialect mixing, is used to enhance the sample pronunciation segments and positive pronunciation segments.

[0107] Optionally, the device also includes a sensitive pronunciation embedding set determination module for:

[0108] Collect pronunciation samples of the target sensitive words;

[0109] An embedding model is used to extract embedding features from the pronunciation samples of sensitive words to obtain sample embedding vectors;

[0110] Clustering the sample embedding vectors yields a set of sensitive pronunciation embeddings for the target sensitive words.

[0111] The audio content detection device provided in this embodiment of the invention can execute the audio content detection method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method.

[0112] According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0113] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the audio content detection method of this invention. Figure 4A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0114] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0115] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0116] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as audio content detection methods.

[0117] In some embodiments, the audio content detection method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the audio content detection method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the audio content detection method by any other suitable means (e.g., by means of firmware).

[0118] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0120] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0121] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0122] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0123] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0124] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0125] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method of audio content detection, characterized by, The method comprises the following steps: The target embedding vector is obtained by embedding the to-be-detected audio through a pronunciation embedding model; wherein the pronunciation embedding model is obtained by fine-tuning an initial embedding model through an audio triple of a sample sensitive word; Similarity calculation is performed on the target embedding vector and a core embedding vector in a sensitive pronunciation embedding set to obtain a similarity value; The signal-to-noise ratio estimation value and the dialect type of the to-be-detected audio are determined, and a sensitivity threshold is determined; According to the similarity value and the sensitivity threshold, a sensitive word detection result of the to-be-detected audio is determined.

2. The method of claim 1, wherein, The initial embedding model comprises a backbone network, a weighted feature fusion layer, a projection head and an attention pooling layer; wherein the backbone network comprises a low-level Transformer encoder and a high-level Transformer encoder; the projection head comprises two fully connected layers; and the attention pooling layer comprises a single-head self-attention mechanism.

3. The method of claim 2, wherein, The pronunciation embedding model is obtained by training in the following manner: An audio triple of a sample sensitive word is constructed; the audio triple comprises a sample pronunciation segment, a positive pronunciation segment and a reverse pronunciation segment; Each frame of audio of the audio triple is input into the backbone network to obtain output features of each layer of the backbone network; The output features of each layer of the backbone network are fused through the weighted feature fusion layer to obtain fused features; The fused features are dimensionally mapped through the projection head to obtain frame-level mapping features of the audio triple; The frame-level mapping features of the audio triple are weighted and aggregated through the attention pooling layer to obtain a segment-level embedding vector; A training loss is determined according to a scaling factor, a relaxation boundary, a positive sample weight, a negative sample weight and the segment-level embedding vector; The initial embedding model is fine-tuned based on the training loss to obtain the pronunciation embedding model.

4. The method of claim 3, wherein, Fine-tuning the initial embedding model based on the training loss to obtain the pronunciation embedding model comprises: The network parameters of the low-level Transformer encoder of the backbone network are kept unchanged; A projection matrix and a value projection matrix are queried in the attention module of the high-level Transformer encoder of the backbone network, and a LoRA decomposition matrix is bypassed inserted; Based on the LoRA technology, the initial embedding model is fine-tuned according to the training loss to obtain the pronunciation embedding model.

5. The method of claim 1, wherein, After constructing the audio triple of the sample sensitive word, the following steps are further included: At least one enhancement method of time domain enhancement, frequency domain enhancement, noise injection and dialect mixing is used to enhance the sample pronunciation segment and the positive pronunciation segment.

6. The method of claim 1, wherein, The sensitive pronunciation embedding set is determined in the following manner: Sensitive word pronunciation samples of a target sensitive word are collected; The sensitive word pronunciation samples are embedded and feature-extracted through the pronunciation embedding model to obtain sample embedding vectors; The sample embedding vectors are clustered to obtain a sensitive pronunciation embedding set of the target sensitive word.

7. An audio content detection apparatus characterized by comprising: The method comprises the following steps: A target embedding vector determination module is configured to embed a to-be-detected audio through a pronunciation embedding model to obtain a target embedding vector; wherein the pronunciation embedding model is obtained by fine-tuning an initial embedding model through an audio triple of a sample sensitive word; The similarity determination module is configured to perform similarity calculation on the target embedding vector and a core embedding vector in the sensitive pronunciation embedding set, to obtain a similarity value. The sensitivity threshold determination module is configured to determine a signal-to-noise ratio estimation value and a dialect type of the audio to be detected, and determine a sensitivity threshold. The detection result determination module is configured to determine a sensitive word detection result of the audio to be detected according to the similarity value and the sensitivity threshold.

8. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected to the at least one processor in communication; wherein The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the audio content detection method of any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for enabling the processor to implement the audio content detection method of any one of claims 1-6 when executed.

10. A computer program product, characterised in that, The computer program product comprises a computer program, which, when executed by a processor, implements the audio content detection method according to any one of claims 1-6.

Images