Multi-language audio content retrieval method, device, equipment, medium and product

Abstract

The invention provides a multi-language audio content retrieval method which can be applied to the technical field of artificial intelligence. The method comprises the following steps: performing audio feature extraction on an example segment through a pre-trained audio encoder to obtain an example general voice sequence; discretizing the example general voice sequence to generate an example discrete sequence for representing audio vocabularies of the example general voice sequence; performing semantic extraction on the example discrete sequence through a pre-trained audio semantic embedding model to obtain an example semantic vector; calculating centroids of the example semantic vectors corresponding to all the example fragments through a preset average pooling algorithm to obtain query vectors; and obtaining cosine similarity between the storage semantic vectors in a preset retrieval library and the query vector, and based on cosine similarity sorting, taking the storage audio clips corresponding to the preset number of storage semantic vectors as target output audio. The invention further provides a multi-language audio content retrieval device and equipment, a medium and a product.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to the field of artificial intelligence-based speech signal processing technology, and more specifically to a method, apparatus, device, medium, and product for multilingual audio content retrieval. Background Technology

[0002] Currently, the mainstream technical approach for retrieving specific audio from massive amounts of audio relies heavily on ASR (Automatic Speech Recognition) technology, which, simply put, is the technology of converting speech into text. In other words, the basic process for retrieving specific audio is: first, the audio data is converted into text using an ASR system, and then text retrieval techniques, such as keyword matching and full-text search, are used to locate the target content.

[0003] However, this method has the following drawbacks when facing real-world applications involving multiple languages ​​and large datasets: High-precision ASR systems require extensive "audio-text" pairing data for supervised training. While this is manageable for high-resource languages ​​like English and Chinese, building an ASR system for thousands of low-resource languages, dialects, and minority languages ​​worldwide is extremely costly or even infeasible, resulting in significant blind spots in retrieval coverage and creating barriers for multilingual and low-resource languages. Furthermore, the ASR system only outputs text, completely discarding the rich paralinguistic information contained in the audio, such as emotion, tone, voiceprint identity, and non-speech acoustic events. This information is crucial for understanding and interpreting the nuances of audio data. While crucial for areas like content security and sentiment analysis, this approach cannot be retrieved via text keywords, resulting in the loss of paralinguistic information. Training both speech-to-text models within ASR systems and dedicated keyword retrieval models requires large-scale, high-quality labeled data. When new sensitive words or concepts emerge frequently, model updates lag, lacking agility and leading to poor robustness and stability. Furthermore, the need for full audio-to-text transcription of all audio segments in a massive audio library incurs enormous computational and storage costs, resulting in long retrieval paths, high latency, slow response times, and low performance in audio retrieval. Summary of the Invention

[0004] In view of the above problems, this application provides a method, apparatus, device, medium and product for retrieving multilingual audio content.

[0005] According to a first aspect of this application, a multilingual audio content retrieval method is provided, comprising: acquiring example segments to represent retrieval needs; extracting audio features from the example segments using a pre-trained audio encoder to obtain example general speech sequences to represent the speech features of the example segments; discretizing the example general speech sequences to generate example discrete sequences of audio words to represent the example general speech sequences; extracting semantics from the example discrete sequences using a pre-trained audio semantic embedding model to obtain example semantic vectors to represent the example discrete sequences; calculating the centroids of the example semantic vectors corresponding to all example segments using a preset average pooling algorithm to obtain a query vector; acquiring the cosine similarity between the stored semantic vectors in a preset retrieval database and the query vector; and, based on the ranking of the cosine similarity, selecting a preset number of stored audio segments corresponding to the stored semantic vectors as target output audio.

[0006] According to embodiments of this application, constructing the retrieval library includes: segmenting audio from a massive audio library into audio segments as stored audio segments; converting the stored audio segments into multilingual audio segments in different languages; extracting audio features from the multilingual audio segments using an audio encoder to obtain a stored general speech sequence representing the speech features of the multilingual audio segments; clustering the stored general speech sequence based on a preset clustering algorithm to obtain an audio vocabulary for the stored general speech sequence; assigning a token identifier to each vector in the stored general speech sequence based on the audio vocabulary to obtain a stored discrete sequence, where the token identifier represents the index of the nearest cluster center to each vector in the stored general speech sequence; extracting semantics from the stored discrete sequence using an audio semantic embedding model to obtain stored semantic vectors representing the stored discrete sequence; storing the stored semantic vectors in a preset vector database; constructing an approximate nearest neighbor index for each stored semantic vector in the vector database based on the token identifier; and establishing a correspondence between the approximate nearest neighbor index, each stored semantic vector in the vector database, and the stored audio segments in the massive audio library to obtain the retrieval library.

[0007] According to an embodiment of this application, pre-training an audio encoder includes: randomly masking a portion of the input audio as training sample audio input to a preset speech self-supervised learning architecture, repeatedly training the speech self-supervised learning architecture so that the speech self-supervised learning architecture predicts the discrete acoustic features of the masked portion of the audio based on the contextual information of the training sample audio; calculating the self-supervised audio loss based on the discrete acoustic features and the acoustic features of the masked portion of the audio, until the self-supervised audio loss reaches a preset self-supervised training stopping condition, and using the optimal speech self-supervised learning architecture as the audio encoder.

[0008] According to an embodiment of this application, a sample general speech sequence is discretized to generate a sample discrete sequence of audio words representing the sample general speech sequence, including: clustering the sample general speech sequence based on a clustering algorithm to determine the audio words related to the sample general speech sequence; assigning a sample token identifier to each vector in the sample general speech sequence to represent the audio words, thereby obtaining the sample discrete sequence.

[0009] According to embodiments of this application, pre-training an audio semantic embedding model includes: randomly masking sequence tokens of pre-acquired discrete sequences to obtain a randomly masked sequence; inputting the randomly masked sequence into a preset mask language architecture so that the mask language architecture understands the acoustic semantic relationships and acoustic grammatical structures between all sequence tokens in the discrete sequence based on the context of the randomly masked sequence, thereby obtaining an acoustic understanding result; calculating a loss function based on the acoustic understanding result and pre-acquired sequence sample identifiers until the loss function is lower than a preset loss threshold, at which point training stops, and the optimal mask language architecture is used as the audio semantic embedding model.

[0010] According to embodiments of this application, semantic extraction of example discrete sequences is performed using a pre-trained audio semantic embedding model to obtain example semantic vectors representing the example discrete sequences. This includes: adding learnable tokens to the beginning of the example discrete sequences to obtain sequences to be understood; extracting acoustic semantic relationships and acoustic grammatical structures between example tokens corresponding to each example token identifier in the sequence to be understood to obtain acoustic features, and updating the learnable tokens with the acoustic features; obtaining the final hidden state corresponding to the updated learnable tokens, and using the final hidden state as the example semantic vector.

[0011] According to an embodiment of this application, obtaining the cosine similarity between a stored semantic vector and a query vector in a preset retrieval library, and using the stored audio segments corresponding to a preset number of stored semantic vectors as target output audio based on the ranking of the cosine similarity, includes: inputting a query vector into a retrieval library to enable the retrieval library to obtain an approximate nearest neighbor index corresponding to the query vector as a target index, and determining the stored semantic vectors related to the target index; calculating the cosine similarity between the stored semantic vector and the query vector to obtain a similarity result; sorting the similarity results from high to low, and using the stored semantic vectors corresponding to the top preset number of similarity results in the ranking as target semantic vectors; and using the stored audio segments corresponding to the target semantic vectors as target output audio.

[0012] According to an embodiment of this application, a general speech sequence is clustered based on a preset clustering algorithm to obtain an audio vocabulary for the general speech sequence. This includes: using the general speech sequence corresponding to all multilingual audio segments as a vector pool; clustering the vector pool using a clustering algorithm to obtain a preset number of clusters; and using the cluster center of each cluster to represent an audio sub-word unit of the audio vocabulary. The higher the frequency of the audio sub-word unit, the larger the cluster to which the cluster center belongs; and the lower the frequency of the audio sub-word unit, the smaller the cluster to which the cluster center belongs.

[0013] The second aspect of this application provides a multilingual audio content retrieval device, comprising: an audio encoding module for acquiring example segments representing retrieval needs, extracting audio features from the example segments using a pre-trained audio encoder to obtain example general speech sequences representing the speech features of the example segments; a discretization module for discretizing the example general speech sequences to generate example discrete sequences of audio vocabulary representing the example general speech sequences; a semantic extraction module for extracting semantics from the example discrete sequences using a pre-trained audio semantic embedding model to obtain example semantic vectors representing the example discrete sequences; a query determination module for calculating the centroids of the example semantic vectors corresponding to all example segments using a preset average pooling algorithm to obtain a query vector; and a target output module for acquiring the cosine similarity between the stored semantic vectors in a preset retrieval database and the query vector, and, based on the ranking of the cosine similarity, using the stored audio segments corresponding to a preset number of stored semantic vectors as target output audio.

[0014] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0016] The fifth aspect of this application also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0017] The above one or more embodiments have the following beneficial effects: After obtaining example segments to represent retrieval needs, audio features are extracted from the example segments using a pre-trained audio encoder to obtain example general speech sequences that represent the speech features of the example segments. This aims to directly learn example discrete sequences representing audio vocabulary that are unrelated to text from the audio signal. These sequences are unrelated to natural language and text, thus allowing the audio retrieval process to abandon the process of first converting audio to text and then performing audio matching based on the text, thereby improving the robustness of audio retrieval. Then, the centroids of the example semantic vectors corresponding to all example segments are calculated to obtain the query vector. A single query vector can represent the summary vectors common to all example segments, facilitating retrieval matching and improving the directness and effectiveness of retrieval matching. The most similar stored audio segments are determined by calculating the cosine similarity between the example semantic vectors of the example segments and the stored semantic vectors in the retrieval database, thereby improving retrieval accuracy and robustness. Attached Figure Description

[0018] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0019] Figure 1 The illustrations depict application scenarios of multilingual audio content retrieval methods, apparatuses, devices, media, and program products according to embodiments of this application.

[0020] Figure 2 A flowchart illustrating a multilingual audio content retrieval method according to an embodiment of this application is shown schematically.

[0021] Figure 3 This diagram illustrates the data flow of a multilingual audio content retrieval method according to an embodiment of this application.

[0022] Figure 4 This illustration shows a schematic diagram of the data flow involved in constructing the retrieval library in the multilingual audio content retrieval method according to an embodiment of this application;

[0023] Figure 5 This schematic diagram illustrates the structure of a multilingual audio content retrieval device according to an embodiment of the present application;

[0024] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a multilingual audio content retrieval method according to an embodiment of this application. Detailed Implementation

[0025] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0027] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0028] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0029] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0030] The current basic process for retrieving specific audio is as follows: first, the audio data is converted into text using an ASR system, and then text retrieval techniques, such as keyword matching and full-text search, are used to locate the target content. This method has the following drawbacks when facing multilingual and large-scale data applications: High-precision ASR systems require a large amount of "audio-text" pairing data for supervised training. While this is manageable for high-resource languages ​​like English and Chinese, building an ASR system for thousands of low-resource languages, dialects, and minority languages ​​worldwide is extremely costly or even infeasible, resulting in a huge blind spot in retrieval coverage and causing multilingual and low-resource language barriers. Furthermore, ASR systems only output text, completely discarding the rich paralinguistic information contained in the audio, such as emotions (anger, joy, etc.). The information contained in audio retrieval data, such as tone (sarcasm, questioning, etc.), voiceprint identity, and non-vocal acoustic events (gunshots, explosions, broken glass, etc.), is crucial for content security and sentiment analysis, yet it cannot be retrieved through text keywords. This results in the loss of paralinguistic information. Furthermore, training both the speech-to-text model in the ASR system and a dedicated keyword retrieval model requires large-scale, high-quality labeled data. When new sensitive words or concepts emerge frequently, model updates lag, lacking agility and leading to poor robustness and stability. Additionally, the need for full audio-to-text transcription of all audio segments in a massive audio library incurs huge computational and storage costs, resulting in long retrieval paths, high latency, slow response times, and low performance in audio retrieval.

[0031] This application provides a multilingual audio content retrieval method, apparatus, device, medium, and product. The multilingual audio content retrieval method includes: acquiring example segments representing retrieval needs; extracting audio features from the example segments using a pre-trained audio encoder to obtain example general speech sequences representing the speech features of the example segments; discretizing the example general speech sequences to generate example discrete sequences of audio vocabulary representing the example general speech sequences; extracting semantics from the example discrete sequences using a pre-trained audio semantic embedding model to obtain example semantic vectors representing the example discrete sequences; calculating the centroids of the example semantic vectors corresponding to all example segments using a preset average pooling algorithm to obtain a query vector; obtaining the cosine similarity between the stored semantic vectors in a preset retrieval database and the query vector; and, based on the ranking of the cosine similarity, selecting a preset number of stored audio segments corresponding to the stored semantic vectors as target output audio.

[0032] Figure 1 The illustration shows an application scenario diagram of the multilingual audio content retrieval method according to an embodiment of this application.

[0033] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105.

[0034] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0035] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0036] Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables. Network 104 serves as the medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0037] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0038] It should be noted that the multilingual audio content retrieval method provided in this application embodiment can generally be executed by server 105. Correspondingly, the multilingual audio content retrieval device provided in this application embodiment can generally be located in server 105. The multilingual audio content retrieval method provided in this application embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the multilingual audio content retrieval device provided in this application embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0039] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0040] The following will be based on Figure 1 The described scene, through Figures 2-4 A multilingual audio content retrieval method according to embodiments of this application will be described in detail.

[0041] Figure 2 A flowchart illustrating a multilingual audio content retrieval method according to an embodiment of this application is shown.

[0042] like Figure 2 As shown, the multilingual audio content retrieval method of this embodiment includes steps S210 to S250, as detailed below:

[0043] Operation S210 obtains example segments to represent the retrieval requirements, and extracts audio features from the example segments using a pre-trained audio encoder to obtain example general speech sequences to represent the speech features of the example segments; Operation S220 discretizes the example general speech sequences to generate example discrete sequences of audio words to represent the example general speech sequences; Operation S230 extracts semantics from the example discrete sequences using a pre-trained audio semantic embedding model to obtain example semantic vectors to represent the example discrete sequences; Operation S240 calculates the centroids of the example semantic vectors corresponding to all example segments using a preset average pooling algorithm to obtain the query vector; Operation S250 obtains the cosine similarity between the stored semantic vectors in the preset retrieval database and the query vector, and based on the ranking of the cosine similarity, selects the stored audio segments corresponding to a preset number of stored semantic vectors as the target output audio.

[0044] Figure 3 The diagram illustrates the data flow of a multilingual audio content retrieval method according to an embodiment of this application.

[0045] exist Figure 3In the example shown, the above steps are performed based on a pre-built retrieval system. This retrieval system includes a trained audio encoder, an audio semantic embedding model, and a retrieval library. When performing a retrieval, this retrieval system can be directly applied. The example fragments used to represent the retrieval requirements can be example fragments uploaded by the user or example fragments pre-corresponding to the user's language or text input. For example, the user directly inputs example audio fragments about "explosion sound" or "joyful laughter." These example audio fragments are positive example audio fragments. Usually, 5 to 20 examples are sufficient to output a large number of target output audios. These target output audios are diverse audio fragments about "explosion sound" and "joyful laughter."

[0046] Then, the audio waveform X of the example segment of arbitrary length is converted into a series of hidden layer vector representations Z = (z1, z2, ..., zT) rich in context information by a pre-trained audio encoder. The hidden layer vector representation Z = (z1, z2, ..., zT) is the example general speech sequence, and T represents a positive integer. This audio encoder is the foundation of the entire audio content retrieval operation process, and its output quality directly determines the performance of all subsequent modules.

[0047] Then, the example general speech sequence is discretized to generate an example discrete sequence of audio vocabulary to represent the example general speech sequence. That is, the continuous example speech sequence Z is discretized by a preset audio segmenter and quantizer to generate a finite "audio vocabulary" V, and each vector in Z is converted into a token ID in V, i.e., a token identifier, so as to obtain the discrete sequence Q = (q1, q2, ..., qT).

[0048] Semantic extraction is performed on discrete example sequences using a pre-trained audio semantic embedding model to obtain example semantic vectors representing the discrete example sequences. Then, a pre-defined average pooling algorithm is used to calculate the centroids of the example semantic vectors corresponding to all example segments, resulting in a query vector. Specifically, these example segments are processed through the above process to obtain N example semantic embedding vectors {E1, E2, ..., En}. Average pooling is then used to calculate the centroids of these example semantic embedding vectors, yielding the query vector V_q for the target feature. V_q defines a "semantic center point" in the vector space; the semantic vectors corresponding to all audio segments semantically similar to the example segments will be close to this point. More specifically, a retrieval process is first performed: the query vector V_q is input into the retrieval system; the retrieval system uses an ANN search to quickly find the cosine similarity of V_q in the vector database. The top-K vectors with the highest cosine similarity and their corresponding audio segments can be used as the search results. For example, the top 50 semantic vectors with the highest cosine similarity, the corresponding audio segments of the semantic vectors, and multilingual audio segments can be stored as the search results. The audio in these search results is the target output audio.

[0049] Based on this, after obtaining example segments to represent retrieval needs, audio features are extracted from the example segments using a pre-trained audio encoder to obtain example general speech sequences representing the speech features of the example segments. This aims to directly learn discrete example sequences representing audio vocabulary unrelated to text from audio signals. These sequences are independent of natural language and text, thus eliminating the need to first convert audio to text and then perform audio matching based on the text, thereby improving the robustness of audio retrieval. Then, the centroids of the example semantic vectors corresponding to all example segments are calculated to obtain the query vector. A single query vector can represent the summary vectors common to all example segments, facilitating retrieval matching and improving the directness and effectiveness of retrieval matching. Finally, the most similar stored audio segments are determined by calculating the cosine similarity between the example semantic vectors of the example segments and the stored semantic vectors in the retrieval database, improving retrieval accuracy and robustness. It boasts robustness, and because it does not perform speech-to-text conversion, it is not constrained by the requirement of labeling for speech-to-text model training. Therefore, audio processing is unrelated to text, allowing for easy training using multiple languages / languages. Based on this, it can handle any language, dialect, and even non-linguistic sound events, breaking through the barriers of multilingualism and low-resource languages. The core audio encoder's audio segmentation and semantic embedding model training are completely unsupervised, requiring no labeling, thus reducing costs. Furthermore, defining a new retrieval target only requires a few audio examples, improving agility and making it suitable for quickly responding to newly emerging sensitive words in the content security field. It can not only retrieve keywords, but also, thanks to the preservation of paralinguistic information by audio encoding, it can simultaneously retrieve keywords, speakers, emotions, acoustic events, etc., improving functional integration. Semantic similarity-based matching is more tolerant of pronunciation differences, speech rate variations, background noise, and other interference than phoneme- or template-based methods, improving accuracy and robustness.

[0050] Figure 4 The illustration shows a schematic diagram of the data flow involved in constructing the retrieval library in the multilingual audio content retrieval method according to an embodiment of this application.

[0051] exist Figure 4In the illustrated embodiment, constructing the retrieval library includes: segmenting audio from a massive audio library into audio segments for storage; converting the stored audio segments into multilingual audio segments in different languages; extracting audio features from the multilingual audio segments using an audio encoder to obtain a stored general speech sequence representing the speech features of the multilingual audio segments; clustering the stored general speech sequence based on a preset clustering algorithm to obtain an audio vocabulary for the stored general speech sequence; assigning a token identifier to each vector in the stored general speech sequence based on the audio vocabulary to obtain a stored discrete sequence, where the token identifier represents the index of the cluster center closest to each vector in the stored general speech sequence; extracting semantics from the stored discrete sequence using an audio semantic embedding model to obtain a stored semantic vector representing the stored discrete sequence; storing the stored semantic vectors in a preset vector database; constructing an approximate nearest neighbor index for each stored semantic vector in the vector database based on the token identifier; and establishing a correspondence between the approximate nearest neighbor index, each stored semantic vector in the vector database, and the stored audio segments in the massive audio library to obtain the retrieval library.

[0052] As an example, the retrieval library is pre-built, meaning it was created when the retrieval system was established. The steps for building the retrieval library are actually quite similar to the process of processing the example clips described above, except that the example clips are replaced with a massive amount of audio from a massive audio library. The process of building the database also uses pre-trained audio encoders, pre-trained audio semantic embedding models, and other functional modules.

[0053] Specifically, when constructing the retrieval library, the massive audio data is first segmented into audio segments for storage. These stored audio segments are then converted into multilingual audio segments in different languages, overcoming the barrier of related technologies that cannot be applied to multiple languages / languages. Next, an audio encoder extracts audio features from the multilingual audio segments to obtain a stored universal speech sequence that represents the speech features of the multilingual audio segments. This follows the same procedure as the example segment, converting the waveform X of a multilingual audio segment of arbitrary length into a series of hidden layer vector representations Z = (z1, z2, ..., zT) rich in contextual information. This hidden layer vector representation Z = (z1, z2, ..., zT) is then... The operation of zT, which stores general speech sequences, is the foundation of the entire retrieval process. Its output quality directly determines the performance of all subsequent modules. Then, the stored general speech sequences are clustered using a pre-trained audio segmentation and quantization unit based on a preset clustering algorithm to obtain an audio vocabulary for the stored general speech sequences. Each vector in the stored general speech sequences is assigned a token identifier in the audio vocabulary to obtain a stored discrete sequence. The token identifier represents the index of the nearest cluster center to each vector in the stored general speech sequences. Specifically, all stored general speech sequences form a huge vector pool {z}. Then, cluster analysis is performed to obtain a learning vocabulary / audio vocabulary. Specifically, the k-means clustering algorithm is first applied to all vectors in the collected vector pool. The number of clusters K is a hyperparameter, usually between 5000 and 50000, which determines the size of the "audio vocabulary". Each cluster center represents an "audio sub-word unit", or "audio vocabulary", or AudioPiece. The k-means clustering process is similar to an iterative merging process. High-frequency acoustic patterns, such as common syllables and phoneme combinations, form dense, large clusters, similar to common audio words; while rare acoustic patterns form small clusters, representing rare audio words. After the audio vocabulary is determined, quantized word segmentation is performed. In other words, after the vocabulary V, i.e., K cluster centers, is determined, for any new audio / multilingual audio segment / feature sequence of stored audio segments / stored general speech sequence Z, a token ID q_t is assigned to each vector z_t, where q_t is the index of the nearest cluster center to z_t, and the sequence Q is the "word segmentation" result of the audio.

[0054] Then, the stored semantic vector is obtained. Specifically, a special learnable token can be added at the beginning of the sequence. The final hidden state corresponding to this token is used as the semantic summary vector of the entire input audio sequence, that is, the required embedding vector E. The embedding vector E is the stored semantic vector that represents the stored discrete sequence.

[0055] In a more specific example, the massive audio library to be retrieved (such as millions of hours of audio) is divided into segments of fixed length, for example, 4 seconds each. Each audio segment is then passed through a general audio encoder -> audio segmentation and quantization -> audio semantic embedding model to obtain its corresponding semantic embedding vector E. All semantic vectors E are stored in an efficient vector database, and an ANN (Approximate Nearest Neighbor) index is built on them to facilitate fast retrieval when applying a retrieval system.

[0056] Based on this, a general speech sequence is stored that is speech-related but language-independent by directly extracting features from a massive amount of multi-speech audio segments. Then, without going through the intermediate text features, the speech sequence is directly extracted as a speech sequence and clustered into an inverted audio vocabulary. After that, the corresponding stored semantic vector is obtained, and an approximate nearest neighbor index is established. Based on this process, when searching later, the most detailed stored audio segment with the example audio can be directly determined based on the differences between the semantic vectors, thereby improving the accuracy and precision of audio retrieval.

[0057] In this embodiment, pre-training the audio encoder includes: randomly masking a portion of the input audio as training sample audio input to a preset speech self-supervised learning architecture, repeatedly training the speech self-supervised learning architecture so that the speech self-supervised learning architecture predicts the discrete acoustic features of the masked portion of the audio based on the contextual information of the training sample audio; calculating the self-supervised audio loss based on the discrete acoustic features and the acoustic features of the masked portion of the audio, until the self-supervised audio loss reaches a preset self-supervised training stopping condition, and using the optimal speech self-supervised learning architecture as the audio encoder.

[0058] As an example, this speech self-supervised learning architecture can adopt a large-scale multilingual self-supervised learning architecture based on the Transformer architecture. These models are pre-trained on tens of thousands of hours of audio in dozens or even hundreds of languages, and can extract irrelevant languages ​​and focus on the general speech features of speech. Specifically, this speech self-supervised learning architecture is trained with self-supervised objectives such as overmasking prediction. During training, some frames of the input audio are randomly masked, and the model must predict the acoustic units / audio words of the masked parts according to the context. This process forces the model to learn discrete and meaningful acoustic patterns in speech, rather than just low-level acoustic features.

[0059] For an input audio segment, the output of the intermediate Transformer layer of the trained audio encoder, such as the output of the 12th layer, yields a general speech sequence Z, where each vector z_t in the general speech sequence Z represents a deep contextual feature of a short time segment (e.g., 20ms) in the audio.

[0060] The audio encoder trained based on this process can focus on the deep contextual features of discrete, meaningful acoustic patterns in speech, thereby improving robustness and generalization ability, and consequently improving the accuracy and effectiveness of extracting audio features from audio.

[0061] In this embodiment, the example general speech sequence is discretized to generate example discrete sequences of audio words that represent the example general speech sequence. This includes: performing clustering processing on the example general speech sequence based on a clustering algorithm to determine the audio words related to the example general speech sequence; and assigning example token identifiers to each vector in the example general speech sequence to represent the audio words, thereby obtaining the example discrete sequences.

[0062] As an example, this process uses the same clustering algorithm as the one used when building the retrieval library. However, the amount of data for this example general speech sequence is smaller, and since it has already been clustered during the retrieval library construction, the clustering algorithm can retain traces of the clustering process during retrieval library construction. For example, each example general speech sequence is used to form a vector pool, and then cluster analysis is performed to obtain an audio vocabulary for the example general speech sequence. Since the k-means clustering algorithm has already been applied to all vectors in the collected vector pool during the retrieval library construction stage, the number of clusters K is a hyperparameter, usually between 5000 and 50000. That is, the size of the "audio vocabulary" has been determined by clustering a large number of audio segments during the retrieval library construction stage. Each cluster center represents an "audio vocabulary", and there are only a few example segments. Therefore, the "audio vocabulary" generated when clustering the example general speech sequence is likely to belong to one of the audio vocabulary generated during the retrieval library construction. Similarly, for the example general speech sequence Z, a token ID q_t needs to be assigned to each vector z_t to obtain the example discrete sequence, where q_t is the index of the nearest cluster center to z_t, and the sequence Q is the "segmentation" result of the audio.

[0063] Based on this, exemplary audio words and example token identifiers representing those audio words can be obtained from example general speech sequences, and then example discrete sequences can be obtained. This allows the example discrete sequences to accurately represent the audio features of the example segments, thereby providing a feature basis for improving the accuracy and relevance of audio retrieval.

[0064] In this embodiment, the pre-training of the audio semantic embedding model includes: randomly masking the sequence tokens of a pre-acquired discrete sequence to obtain a randomly masked sequence; inputting the randomly masked sequence into a preset mask language architecture so that the mask language architecture can understand the acoustic semantic relationships and acoustic grammatical structures between all sequence tokens in the discrete sequence based on the context of the randomly masked sequence, thereby obtaining an acoustic understanding result; calculating a loss function based on the acoustic understanding result and the pre-acquired sequence sample identifiers until the loss function is lower than a preset loss threshold, at which point training stops, and the optimal mask language architecture is used as the audio semantic embedding model.

[0065] As an example, the goal of the audio semantic embedding model trained by this masked language architecture is to learn the semantic information of example discrete sequences / stored discrete sequences, thereby converting audio segments of arbitrary length into a fixed-dimensional vector E representing its overall semantics, i.e., embedding. In a specific example, this masked language architecture can adopt a standard Transformer encoder structure, that is, inputting a discrete sequence Q with audio token IDs into the masked language architecture for repeated training. The training task is to randomly mask the tokens in the discrete sequence Q, such as masking 15% of the tokens, so that the Transformer encoder structure can predict the masked tokens based on the context. This task forces the model to deeply understand the semantic relationships and grammatical structures between audio tokens, which can be understood as acoustic grammar or semantic order in audio. Based on this, when obtaining the output of the trained audio semantic embedding model, a special learnable token can be added to the beginning of the discrete sequence Q. The final hidden state corresponding to this token is used as the SSV (Semantic Summary Vector) of the entire input audio sequence, that is, the embedding vector E required to represent the semantics of the audio segment, which is also the semantic vector. Due to the global attention mechanism of Transformer, this learnable vector can gather the semantic information of the entire sequence, thereby increasing the attention to the audio semantic information.

[0066] This process is based on unsupervised training using random masking, thereby learning acoustic semantic relationships and acoustic grammatical structures in an unsupervised manner, which in turn improves the robustness and semantic feature recognition accuracy of the trained audio semantic embedding model.

[0067] In this embodiment, semantic extraction is performed on the example discrete sequence using a pre-trained audio semantic embedding model to obtain an example semantic vector representing the example discrete sequence. This includes: adding a learnable token to the beginning of the example discrete sequence to obtain a sequence to be understood; extracting the acoustic semantic relationship and acoustic grammatical structure between the example tokens corresponding to each example token identifier in the sequence to be understood to obtain acoustic features, and updating the learnable token with the acoustic features; obtaining the final hidden state corresponding to the updated learnable token, and using the final hidden state as the example semantic vector.

[0068] As an example, similar to the training process, a learnable token is added at the beginning of the example discrete sequence. The final hidden state corresponding to the learnable token is used as the SSV (Semantic Summary Vector) of the entire input audio sequence, which is the embedding vector E required to represent the semantics of the example audio segment, i.e., the example semantic vector. Due to the global attention mechanism of the Transformer, the learnable vector can aggregate the semantic information of the entire example discrete sequence, thereby increasing the attention to the audio semantic information.

[0069] Thus, by adding a learnable token at the beginning of the sequence, a sequence to be understood with a learnable token is obtained. During the learning process, the learnable token is updated with acoustic features. The final hidden layer state corresponding to the updated learnable token is the example semantic vector. Based on this, the acoustic semantic relationships and acoustic grammatical structure of the sequence to be understood can be captured to the maximum extent. This facilitates the calculation of cosine similarity between the generated example semantic vector and the stored semantic vector, thereby improving the relevance and effectiveness of audio retrieval.

[0070] In this embodiment, obtaining the cosine similarity between the stored semantic vector and the query vector in a preset retrieval library, and using the stored audio segments corresponding to a preset number of the stored semantic vectors as target output audio based on the ranking of the cosine similarity, includes: inputting the query vector into the retrieval library so that the retrieval library obtains the approximate nearest neighbor index corresponding to the query vector as the target index, and determining the stored semantic vector related to the target index; calculating the cosine similarity between the stored semantic vector and the query vector to obtain a similarity result; sorting the similarity results from high to low, and using the stored semantic vectors corresponding to the top preset number of similarity results in the ranking as target semantic vectors; and using the stored audio segments corresponding to the target semantic vectors as target output audio.

[0071] As an example, after obtaining N example semantic vectors {E1, E2, ..., En}, the centroids of these example semantic embedding vectors are calculated using average pooling to obtain the query vector V_q for the target feature. V_q defines a "semantic center point" in the vector space. The semantic vectors corresponding to all audio segments that are semantically similar to the example segments will be close to this point. More specifically, a retrieval is performed first: the query vector V_q is input into the retrieval system; the retrieval system uses an ANN search to quickly find the Top-K vectors with the highest cosine similarity to V_q and their corresponding audio segments in the vector database. For example, the top 50 semantic vectors with the highest cosine similarity / the audio segments corresponding to the stored semantic vectors / multilingual audio segments / stored audio segments can be output as the retrieval results. The audio in this retrieval result is the target output audio.

[0072] During the database construction phase, an ANN (Approximate Nearest Neighbor) index was built. Searching based on this index can greatly improve search speed, reduce response time, and improve the overall efficiency of audio retrieval. Furthermore, based on cosine similarity calculation, a preset number of audio segments most similar to the example audio are selected for storage, thereby improving the accuracy and effectiveness of retrieval.

[0073] In this embodiment, a general speech sequence is clustered based on a preset clustering algorithm to obtain an audio vocabulary for the general speech sequence. This includes: using the general speech sequence corresponding to all multilingual audio segments as a vector pool; clustering the vector pool using a clustering algorithm to obtain a preset number of clusters; and using the cluster center of each cluster to represent the audio sub-word unit of the audio vocabulary. The higher the frequency of the audio sub-word unit, the larger the cluster to which the cluster center belongs; and the lower the frequency of the audio sub-word unit, the smaller the cluster to which the cluster center belongs.

[0074] As an example, a pre-trained audio segmentation and quantization unit performs clustering processing on stored general speech sequences based on a preset clustering algorithm to obtain an audio vocabulary for the stored general speech sequences. Each vector in the stored general speech sequences is assigned a token identifier from the audio vocabulary to obtain a stored discrete sequence. The token identifier represents the index of the nearest cluster center to each vector in the stored general speech sequences. Specifically, all stored general speech sequences form a huge vector pool {z}, and then cluster analysis is performed to obtain a learned vocabulary / audio vocabulary. Specifically, the k-means clustering algorithm is first applied to all vectors in the collected vector pool. The number of clusters K is a hyperparameter, typically between 5000 and 50000, which determines the size of the "audio vocabulary". Each cluster center represents an "audio sub-word unit", or "audio vocabulary", or AudioPiece. The k-means clustering process is similar to an iterative merging process. High-frequency acoustic patterns, such as common syllables and phoneme combinations, form dense, large clusters, similar to common audio words; while rare acoustic patterns form small clusters, representing rare audio words. After the audio vocabulary is determined, quantized word segmentation is performed. In other words, after the vocabulary V, i.e., K cluster centers, is determined, for any new audio / multilingual audio segment / feature sequence of stored audio segments / stored general speech sequence Z, a token ID q_t is assigned to each vector z_t, where q_t is the index of the nearest cluster center to z_t, and the sequence Q is the "word segmentation" result of the audio.

[0075] This process enables the generation of an audio vocabulary that highlights audio features through clustering, thus eliminating the need for text recognition in the entire audio retrieval process. The retrieval can be completed by directly extracting audio features, thereby enabling the processing of sound events in any language, dialect, or even non-linguistic languages, breaking through the barriers of multilingualism and low-resource languages.

[0076] As described above, the multilingual audio content retrieval method provided in this embodiment employs a completely unsupervised training process for its core audio segmentation and semantic embedding models, requiring no annotations and reducing costs. Defining a new retrieval target requires only a few audio examples, improving agility and making it suitable for rapid response to newly emerging sensitive words in the content security field. In addition to retrieving keywords, thanks to the preservation of paralinguistic information by audio encoding, it can simultaneously retrieve keywords, speakers, emotions, acoustic events, etc., improving functional integration. Semantic similarity-based matching is more tolerant of interference from pronunciation differences, speech rate variations, and background noise than phoneme- or template-based methods, improving accuracy and robustness.

[0077] Based on the above-described multilingual audio content retrieval method, this application also provides a multilingual audio content retrieval device. The following will combine... Figure 5The device is described in detail.

[0078] Figure 5 A schematic block diagram of a multilingual audio content retrieval device according to an embodiment of this application is shown.

[0079] like Figure 5 As shown, the multilingual audio content retrieval device 500 of this embodiment includes an audio encoding module 510, a discrete processing module 520, a semantic extraction module 530, a query determination module 540, and a target output module 550.

[0080] The audio encoding module 510 can perform operation S210 to obtain example segments to represent retrieval needs, and extract audio features from the example segments using a pre-trained audio encoder to obtain example general speech sequences to represent the speech features of the example segments.

[0081] The discrete processing module 520 can perform operation S220 to discretize the example general speech sequence and generate an example discrete sequence of audio words to represent the example general speech sequence.

[0082] The semantic extraction module 530 can perform operation S230 to extract semantics from the example discrete sequence using a pre-trained audio semantic embedding model to obtain an example semantic vector that represents the example discrete sequence.

[0083] The query determination module 540 can perform operation S240, which is used to calculate the centroid of the example semantic vectors corresponding to all example fragments through a preset average pooling algorithm to obtain the query vector;

[0084] The target output module 550 can perform operation S250 to obtain the cosine similarity between the stored semantic vector in the preset retrieval library and the query vector, and based on the ranking of the cosine similarity, use the stored audio segments corresponding to a preset number of the stored semantic vectors as the target output audio.

[0085] In this embodiment, the retrieval library involved in the target output module 550 is pre-set. The construction of the retrieval library includes: segmenting audio from a massive audio library into audio segments for storage; converting the stored audio segments into multilingual audio segments in different languages; extracting audio features from the multilingual audio segments using an audio encoder to obtain a stored general speech sequence representing the speech features of the multilingual audio segments; clustering the stored general speech sequence based on a pre-set clustering algorithm to obtain an audio vocabulary for the stored general speech sequence; assigning a token identifier from the audio vocabulary to each vector in the stored general speech sequence to obtain a stored discrete sequence, where the token identifier represents the index of the nearest cluster center to each vector in the stored general speech sequence; and extracting semantics from the stored discrete sequence using an audio semantic embedding model to obtain a sequence representing the speech features of the stored discrete sequence. The document describes a process for creating a retrieval database. It involves storing semantic vectors of a discrete sequence, constructing an approximate nearest neighbor index for each stored semantic vector in the database based on token identifiers, establishing a correspondence between the approximate nearest neighbor index, the stored semantic vectors in the database, and the stored audio segments in a massive audio library, to obtain a retrieval database. Specifically, it describes a clustering process for storing general speech sequences using a pre-defined clustering algorithm to obtain an audio vocabulary for these sequences. This includes using the stored general speech sequences corresponding to all multilingual audio segments as a vector pool, and clustering the vector pool to obtain a pre-defined number of clusters. The cluster centers of these clusters represent the audio sub-word units of the audio vocabulary; the higher the frequency of an audio sub-word unit, the larger the cluster to which the cluster center belongs, and vice versa.

[0086] The audio encoder used by the pre-trained audio encoding module 510 includes: randomly masking a portion of the input audio as training sample audio input to a preset speech self-supervised learning architecture, repeatedly training the speech self-supervised learning architecture, so that the speech self-supervised learning architecture predicts the discrete acoustic features of the masked portion of the audio based on the context information of the training sample audio; calculating the self-supervised audio loss based on the discrete acoustic features and the acoustic features of the masked portion of the audio, until the self-supervised audio loss meets the preset self-supervised training stopping condition, and using the optimal speech self-supervised learning architecture as the audio encoder.

[0087] The discrete processing module 520 discretizes the example general speech sequence to generate example discrete sequences of audio words that represent the example general speech sequence. This includes: clustering the example general speech sequence based on a clustering algorithm to determine the audio words related to the example general speech sequence; and assigning example token identifiers to each vector in the example general speech sequence to represent the audio words, thereby obtaining the example discrete sequences.

[0088] The audio semantic embedding model used by the pre-trained semantic extraction module 530 includes: randomly masking the sequence tokens of a pre-acquired discrete sequence to obtain a randomly masked sequence; inputting the randomly masked sequence into a preset mask language architecture so that the mask language architecture understands the acoustic semantic relationships and acoustic grammatical structures between all sequence tokens in the discrete sequence based on the context of the randomly masked sequence, and obtains an acoustic understanding result; calculating a loss function based on the acoustic understanding result and the pre-acquired sequence sample identifiers until the loss function is lower than a preset loss threshold, then stopping training and using the optimal mask language architecture as the audio semantic embedding model; wherein, the semantic extraction module 530 performs semantic extraction on example discrete sequences through the pre-trained audio semantic embedding model to obtain example semantic vectors used to represent example discrete sequences, including: adding learnable tokens at the beginning of the example discrete sequence to obtain a sequence to be understood; extracting the acoustic semantic relationships and acoustic grammatical structures between example tokens corresponding to each example token identifier in the sequence to be understood, obtaining acoustic features, and updating the learnable tokens with acoustic features; obtaining the final layer hidden state corresponding to the updated learnable tokens, and using the final layer hidden state as the example semantic vector.

[0089] Furthermore, according to embodiments of this application, any multiple modules among the audio encoding module 510, discrete processing module 520, semantic extraction module 530, query determination module 540, and target output module 550 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the audio encoding module 510, discrete processing module 520, semantic extraction module 530, query determination module 540, and target output module 550 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the audio encoding module 510, discrete processing module 520, semantic extraction module 530, query determination module 540, and target output module 550 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0090] It should be noted that the implementation methods, technical problems solved, functions achieved, and technical effects of each module in the device embodiment are the same as or similar to the implementation methods, technical problems solved, functions achieved, and technical effects of each corresponding step in the method embodiment, and will not be repeated here.

[0091] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a multilingual audio content retrieval method according to an embodiment of this application.

[0092] like Figure 6 As shown, an electronic device 600 according to an embodiment of this application includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0093] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0094] According to embodiments of this application, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0095] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0096] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0097] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code enables the computer system to implement the multilingual audio content retrieval method provided in the embodiments of this application.

[0098] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0099] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0100] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0101] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0103] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A method for retrieving multilingual audio content, characterized in that, The method includes: Obtain example segments to represent retrieval needs, and extract audio features from the example segments using a pre-trained audio encoder to obtain example general speech sequences to represent the speech features of the example segments; The example general speech sequence is discretized to generate an example discrete sequence of audio words that characterize the example general speech sequence; The example discrete sequence is semantically extracted by a pre-trained audio semantic embedding model to obtain an example semantic vector that represents the example discrete sequence; The query vector is obtained by calculating the centroid of the example semantic vectors corresponding to all example fragments using a preset average pooling algorithm. Obtain the cosine similarity between the stored semantic vectors in the preset retrieval library and the query vector. Based on the ranking of the cosine similarity, select a preset number of stored audio segments corresponding to the stored semantic vectors as the target output audio.

2. The multilingual audio content retrieval method according to claim 1, characterized in that, Constructing the search library includes: The audio in the massive audio library is segmented into audio segments as stored audio segments. The stored audio segments are converted into multilingual audio segments in different languages. The audio encoder extracts audio features from the multilingual audio segments to obtain a stored general speech sequence that represents the speech features of the multilingual audio segments. The general speech sequence is clustered based on a preset clustering algorithm to obtain an audio vocabulary for the general speech sequence. Each vector in the general speech sequence is assigned a token identifier in the audio vocabulary to obtain a discrete speech sequence. The token identifier represents the index of the nearest cluster center to each vector in the general speech sequence. The stored discrete sequence is semantically extracted using the audio semantic embedding model to obtain a stored semantic vector that represents the stored discrete sequence. The stored semantic vectors are stored in a preset vector database, and an approximate nearest neighbor index for each stored semantic vector in the vector database is constructed based on the token identifier; The retrieval library is obtained by establishing the correspondence between the approximate nearest neighbor index, the stored semantic vectors in the vector database, and the stored audio segments in the massive audio library.

3. The multilingual audio content retrieval method according to claim 2, characterized in that, Pre-training the audio encoder includes: Randomly masking a portion of the input audio as training sample audio input to a preset speech self-supervised learning architecture, thereby repeatedly training the speech self-supervised learning architecture, so that the speech self-supervised learning architecture can predict the discrete acoustic features of the masked portion of the audio based on the context information of the training sample audio. The self-supervised audio loss is calculated based on the discrete acoustic features and the acoustic features of the masked partial frames until the self-supervised audio loss reaches a preset self-supervised training stopping condition, and the optimal speech self-supervised learning architecture is used as the audio encoder.

4. The multilingual audio content retrieval method according to claim 2, characterized in that, Discretizing the example general speech sequence to generate example discrete sequences of audio vocabulary representing the example general speech sequence includes: The example general speech sequence is clustered based on the clustering algorithm to determine the audio vocabulary related to the example general speech sequence; Each vector in the example general speech sequence is assigned an example token identifier to represent the audio vocabulary, resulting in an example discrete sequence.

5. The multilingual audio content retrieval method according to claim 2, characterized in that, Pre-trained audio semantic embedding models include: The sequence tokens of the pre-collected discrete sequences are randomly masked to obtain a randomly masked sequence; The random masking sequence is input into a preset masking language architecture so that the masking language architecture can understand the acoustic semantic relationships and acoustic grammatical structures between all sequence tokens in the discrete sequence based on the context of the random masking sequence, and obtain the acoustic understanding result. The loss function is calculated based on the acoustic understanding results and the pre-acquired sequence sample identifiers. Training stops when the loss function is lower than a preset loss threshold, and the optimal masked language architecture is used as the audio semantic embedding model.

6. The multilingual audio content retrieval method according to claim 5, characterized in that, Semantic extraction is performed on the example discrete sequence using a pre-trained audio semantic embedding model to obtain an example semantic vector representing the example discrete sequence, including: A learnable token is added to the beginning of the example discrete sequence to obtain the sequence to be understood; Extract the acoustic semantic relationships and acoustic grammatical structures between the example tokens corresponding to each example token identifier of the sequence to be understood to obtain acoustic features, and update the learnable tokens with the acoustic features; Obtain the final hidden state corresponding to the updated learnable token, and use the final hidden state as an example semantic vector.

7. The multilingual audio content retrieval method according to claim 2, characterized in that, Obtain the cosine similarity between the stored semantic vectors in the preset retrieval database and the query vector. Based on the ranking of the cosine similarity, select a preset number of stored audio segments corresponding to the stored semantic vectors as the target output audio, including: The query vector is input into the retrieval database so that the retrieval database obtains the approximate nearest neighbor index corresponding to the query vector as the target index, and determines the storage semantic vector related to the target index; Calculate the cosine similarity between the stored semantic vector and the query vector to obtain the similarity result; The similarity results are sorted from high to low, and the stored semantic vectors corresponding to the first preset number of similarity results in the sort are used as the target semantic vectors. The stored audio segment corresponding to the target semantic vector is used as the target output audio.

8. The multilingual audio content retrieval method according to claim 2, characterized in that, The stored general speech sequences are clustered based on a preset clustering algorithm to obtain an audio vocabulary for the stored general speech sequences, including: All multilingual audio segments are stored as a general speech sequence as a vector pool; The vector pool is clustered using the clustering algorithm to obtain a preset number of clusters. The cluster center of each cluster represents an audio sub-word unit of the audio vocabulary. The higher the frequency of the audio sub-word unit, the larger the cluster to which the cluster center belongs; the lower the frequency of the audio sub-word unit, the smaller the cluster to which the cluster center belongs.

9. A multilingual audio content retrieval device, characterized in that, The device includes: An audio encoding module is used to acquire example segments to represent retrieval needs, and to extract audio features from the example segments using a pre-trained audio encoder to obtain example general speech sequences to represent the speech features of the example segments. A discrete processing module is used to discretize the example general speech sequence to generate an example discrete sequence of audio words that characterize the example general speech sequence. The semantic extraction module is used to extract semantics from the example discrete sequence using a pre-trained audio semantic embedding model to obtain an example semantic vector that represents the example discrete sequence. The query determination module is used to calculate the centroid of the example semantic vectors corresponding to all example fragments using a preset average pooling algorithm, and obtain the query vector. The target output module is used to obtain the cosine similarity between the stored semantic vectors in the preset retrieval library and the query vector, and based on the ranking of the cosine similarity, to take the stored audio segments corresponding to a preset number of the stored semantic vectors as the target output audio.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images