A method and system for audio semantic retrieval of law enforcement recorders based on retrieval enhancement

Abstract

The present invention relates to the technical fields of speech recognition and natural language processing, and specifically discloses a method and system for semantic retrieval of law enforcement recorder audio based on retrieval enhancement. The system comprises: a data acquisition module for acquiring audio data and text queries; a speech adapter module for projecting the audio data into a text embedding space to obtain a speech representation; a cross-modal retriever for performing cross-modal retrieval of the speech representation and text query to obtain speech tokens; a speech language model for obtaining text hypotheses; a query generation module for extracting query fragments that may contain entity names; an entity retrieval module for searching an entity database based on the query fragments to obtain relevant entity names; a context construction module for constructing contextual information; and a large language model for obtaining semantic retrieval results. The present invention improves the recognition and retrieval accuracy of entity names and key information in law enforcement recorder audio.

Description

Technical Field

[0001] The present invention relates to the technical field of speech recognition and natural language processing, and in particular to a method and system for audio semantic retrieval of law enforcement recorders based on retrieval enhancement. Background Art

[0002] Body cameras are widely used in law enforcement, recording audio and video footage from the scene. However, due to environmental noise, accent differences, and the limitations of speech recognition technology, traditional automatic speech recognition (ASR) systems suffer from high error rates when processing body camera audio, particularly when recognizing uncommon entity names such as names of people, places, and license plates. Traditional ASR systems rely on large amounts of training data, but in law enforcement scenarios, entity names in body camera audio are often low-frequency and diverse, making it difficult for traditional ASR systems to accurately identify these entities, resulting in numerous errors in semantic retrieval results.

[0003] In recent years, retrieval-augmented generation (RAG) technology has demonstrated remarkable performance in natural language processing tasks, particularly when supported by external knowledge bases, significantly improving language model performance. However, existing RAG systems primarily rely on ASR transcripts, which causes ASR errors to propagate to subsequent retrieval and generation steps. Due to the lack of external knowledge base support, RAG systems perform poorly when processing low-frequency entities and complex semantics, making it difficult to correct errors in the ASR system.

[0004] In summary, the existing law enforcement recorder audio semantic retrieval technology has significant deficiencies in entity name recognition, semantic understanding, external knowledge base support, adaptability and scalability, and is unable to meet the law enforcement work's requirements for efficient, accurate and real-time processing. Summary of the Invention

[0005] The present invention aims to address the aforementioned issues. To this end, it provides a search-enhanced semantic retrieval method and system for body camera audio. By directly processing speech data and combining it with an external knowledge base and a large language model, the system avoids the error propagation problem inherent in traditional automatic speech recognition systems, significantly improving retrieval and generation performance, and enhancing the accuracy of identifying and retrieving entity names and key information in body camera audio.

[0006] The present invention provides an audio semantic retrieval system for law enforcement recorders based on retrieval enhancement, and the technical solution adopted is as follows:

[0007] Data acquisition module, used to obtain audio data and text query recorded by law enforcement recorders;

[0008] The speech adapter module is used to project audio data into the text embedding space to obtain speech representation;

[0009] A cross-modal retriever, which is used to perform cross-modal retrieval on speech representations and text queries, and obtain speech tokens with text embedding and speech embedding in the same embedding space;

[0010] The speech language model is used to obtain text hypotheses corresponding to speech tokens based on speech tokens;

[0011] A query generation module, which extracts query fragments that may contain entity names from text hypotheses;

[0012] An entity retrieval module is used to search the entity database and obtain relevant entity names based on query fragments that may contain entity names;

[0013] The context building module is used to build context information based on relevant entity names and textual assumptions;

[0014] Large language models are used to obtain semantic retrieval results based on contextual information.

[0015] Furthermore, the voice adapter module includes a voice encoder and a downsampler,

[0016] A speech encoder, used to convert audio data into a high-dimensional speech representation. The speech encoder is pre-trained on cross-dialect speech data through self-supervised learning;

[0017] The downsampler is used to project the high-dimensional speech representation into the text embedding space to obtain the speech representation.

[0018] Furthermore, the working process of the cross-modal retriever is as follows:

[0019] The prompt template, speech representation and text query are input into a key encoder to obtain a retrieval key. The key encoder adopts a multi-level Transformer architecture based on causal multi-head attention;

[0020] Use the search key to search in the search database to obtain the search value;

[0021] The retrieval value is input into a value encoder to obtain a value vector, wherein the value encoder adopts a multi-level Transformer architecture;

[0022] The prompt template, speech representation, text query and value vector are input into the retrieval enhancement decoder to obtain speech tokens. The retrieval enhancement decoder adopts a multi-level Transformer architecture based on cross attention and causal multi-head attention, and the multi-level Transformer architecture is connected to the language model head.

[0023] Furthermore, the processing of the query generation module is as follows:

[0024] Clean the text hypothesis; use the word segmentation tool to segment the text hypothesis and tag each word with part of speech;

[0025] Use the named entity recognition model to tag each word in the text hypothesis, identify the entity name and its category, and extract the fragments that may contain the entity name;

[0026] Combine fragments that may contain entity names with relevant context words to generate query fragments; sort the query fragments according to their importance or relevance, ensuring that the most important or most relevant fragments are output first, and output query fragments that may contain entity names.

[0027] Furthermore, the entity retrieval module combines acoustic neighbor embedding, semantic embedding and traditional text retrieval methods for retrieval.

[0028] Furthermore, the results of acoustic neighbor embedding, semantic embedding and traditional text retrieval methods are fused to generate relevant entity names.

[0029] Furthermore, the processing of the context building module is as follows:

[0030] Based on the Euclidean distance threshold and the count threshold, the N entities most relevant to the text hypothesis are selected from the relevant entity names to obtain candidate entities;

[0031] Construct hint information for each candidate entity;

[0032] Combine prompt information with textual hypotheses to form contextual information.

[0033] Furthermore, the processing process of the large language model is as follows:

[0034] The large language model uses the self-attention mechanism and multi-layer Transformer structure to deeply understand the relationship between the prompt information in the context information and the text hypothesis; infer the incorrect parts of the text hypothesis and generate corrected text; use greedy decoding to generate the most likely output sequence to obtain semantic retrieval results.

[0035] The present invention also provides a method for audio semantic retrieval of law enforcement recorders based on retrieval enhancement, which adopts the following technical solution: comprising the following steps:

[0036] Obtain audio data and text queries recorded by body cameras;

[0037] Project the audio data into the text embedding space to obtain speech representation;

[0038] Perform cross-modal retrieval on speech representations and text queries to obtain speech tokens with text embedding and speech embedding in the same embedding space;

[0039] According to the speech tokens, obtain the text hypothesis corresponding to the speech tokens;

[0040] Extract query fragments that may contain entity names from text hypotheses;

[0041] According to the query fragment that may contain the entity name, the entity database is searched to obtain the relevant entity names;

[0042] Construct contextual information based on relevant entity names and textual assumptions;

[0043] According to the context information, semantic retrieval results are obtained.

[0044] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0045] 1. This invention completely bypasses the ASR system by directly processing the speech data, avoiding the problem of ASR error propagation. Especially in high-noise or low-quality audio scenarios, SLM can generate more accurate text representations, significantly improving the robustness and reliability of the system.

[0046] 2. This invention achieves the alignment of text and speech in the same embedding space through a cross-modal retriever, enabling the system to retrieve relevant information directly from speech data, reducing dependence on ASR transcription, which not only improves the efficiency of the system but also enhances its applicability in complex scenarios.

[0047] 3. By introducing an external knowledge base and Retrieval-Augmented Generation (RAG) technology, this paper effectively corrects ASR system errors in recognizing uncommon entity names when processing body camera audio. Experimental results show that this approach significantly reduces the word error rate (WER) in entity name recognition tasks, particularly when processing low-frequency entities, improving recognition accuracy by 33.3%.

[0048] 4. By combining a large language model (LLM) with context-building technology, this invention can better understand the semantic information in body camera audio, thereby improving the accuracy of semantic retrieval. This invention not only recognizes entity names but also infers additional key information from context, such as event descriptions, time, and location, providing more comprehensive support for law enforcement analysis.

[0049] 5. This paper employs a two-step context generation process, first filtering the retrieved relevant entity names and then constructing the prompt information. This effectively reduces the size of the context input, thereby reducing computational and memory overhead. Furthermore, using the LoRa adapter to adapt the large language model reduces the number of task-specific parameters while maintaining model performance, further reducing computational resource consumption.

[0050] 6. The entity retrieval method of the present invention supports multiple embedding technologies, including acoustic neighbor embedding, semantic embedding, and Okapi BM25, and can flexibly respond to different data characteristics and retrieval requirements.

[0051] 7. By automatically processing key information in the audio of law enforcement recorders, the present invention can significantly reduce the workload of relevant personnel in manually transcribing and analyzing audio data, thereby improving law enforcement efficiency.

[0052] 8. The design of the present invention is highly scalable and can flexibly integrate new speech encoders, text retrievers, and language models to adapt to ever-changing technical requirements and application scenarios, and can better support multi-language and multi-domain speech data processing tasks.

[0053] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 It is a structural diagram of the system provided by the present invention.

[0056] Figure 2 It is a schematic diagram of the pre-training of the speech adapter module provided by the present invention.

[0057] Figure 3 This is an architectural diagram of the cross-modal retriever provided by the present invention.

[0058] Figure 4 It is a schematic diagram of the creation process of the retrieval database provided by the present invention.

[0059] Figure 5 It is a flow chart of the method provided by the present invention.

[0060] Reference numerals:

[0061] 1. Data acquisition module; 2. Speech adapter module; 3. Cross-modal retriever; 4. Speech language model; 5. Query generation module; 6. Entity retrieval module; 7. Context construction module; 8. Large language model. DETAILED DESCRIPTION

[0062] To make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention. Obviously, the embodiments described are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0063] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0064] The following combination Figures 1 to 5 The present invention is further described in detail, and a method and system for audio semantic retrieval of law enforcement recorders based on retrieval enhancement is described as follows:

[0065] In this embodiment, Figure 1 As shown, a law enforcement recorder audio semantic retrieval system based on retrieval enhancement is provided, including: a data acquisition module 1, a voice adapter module 2, a cross-modal retriever 3, a voice language model 4, a query generation module 5, an entity retrieval module 6, a context construction module 7 and a large language model 8.

[0066] The data acquisition module 1 is used to obtain the audio data and text query recorded by the law enforcement recorder, and input the audio data into the voice adapter module and input the text query into the cross-modal retriever.

[0067] The audio data here is the original audio data recorded by the law enforcement recorder and has not been processed in any way.

[0068] A text query is the content that the user wants to search for. For example, if you want to search for the entity name of a place in audio data, the text query can be described as: "Where did Li go?"

[0069] The speech adapter module 2 is used to project the audio data into the text embedding space to obtain the speech representation.

[0070] The audio data is input into the speech adapter module to calculate the speech representation projected into the text embedding space, which is used as an input to the cross-modal retriever.

[0071] The core function of the speech adapter module is to adapt the speech representation of the input audio data to the text embedding space of the downstream text model, thereby achieving cross-modal alignment between speech and text. The speech adapter module consists of two parts: a speech encoder and a downsampler, and is trained end-to-end using the cosine similarity loss propagated from the text model to ensure the consistency of the speech representation and the text embedding space, such as Figure 2 shown.

[0072] (1) The speech encoder is the front-end component of the speech adapter module, responsible for converting the input audio data into a high-dimensional speech representation. In the specific implementation, it is pre-trained on cross-dialect speech data through self-supervised learning. The key embeddings extracted by the speech encoder not only contain common dialect features such as phoneme distribution, but also retain dialect specificity such as tone patterns, supporting robust alignment of multiple dialects. The last layer of hidden states of the speech encoder is extracted as the input of the downsampler. These hidden states not only contain rich speech information such as phonemes, intonation, rhythm, etc., but also capture the contextual semantic information of the speech through self-supervised learning, providing high-quality input for subsequent cross-modal adaptation.

[0073] (2) The main function of the downsampler is to project the high-dimensional speech representation generated by the speech encoder into the text embedding space, thereby achieving alignment between speech and text. Since the sequence length of speech representation is usually much longer than that of text representation, for example, the speech frame rate is usually 100 frames / second, while the token length of text is much shorter, sequence length adaptation is required. The downsampler achieves this goal through the following two steps.

[0074] Temporal Average Pooling Layer: The downsampler downsamples the speech representation using a temporal average pooling layer. This layer compresses the sequence length of the speech representation to a length similar to the text representation while preserving key speech features. Average pooling effectively reduces redundant information while avoiding information loss caused by direct truncation or random sampling.

[0075] Projection layer: After downsampling, the downsampler uses a projection layer to map the speech representation to the same embedding dimensions as the text. This projection layer is typically implemented as a fully connected neural network, whose weights are optimized through end-to-end training to ensure alignment between the speech representation and the text embedding space. The output of the projection layer is the adapted speech representation, which can be directly used for downstream tasks such as speech-to-text retrieval and speech translation.

[0076] Throughout training, the downsampler is optimized using the cosine similarity loss. This loss function calculates the cosine similarity between speech embeddings and their corresponding text embeddings, guiding the speech representation to converge with the text embedding space, thereby achieving cross-modal alignment. Through end-to-end training, the downsampler learns the complex mapping relationship between speech and text, providing high-quality speech representations for multimodal tasks.

[0077] The advantages of the voice adapter module are as follows:

[0078] Advantages of self-supervised learning: The speech encoder uses self-supervised pre-training to learn robust speech feature representations without relying on large amounts of labeled data. This representation not only captures low-level acoustic features but also high-level semantic information, providing a solid foundation for cross-modal adaptation.

[0079] Flexibility of temporal pooling: The average pooling layer in the temporal dimension can adjust the size and step size of the pooling window according to task requirements, thereby flexibly controlling the degree of downsampling. For example, in tasks that require preserving more details, a smaller pooling window can be used; while in tasks that require compressing sequence length, a larger pooling window can be used.

[0080] Nonlinear capabilities of the projection layer: The projection layer usually contains nonlinear activation functions, such as ReLU, which can enhance the expressiveness of the model and better capture the complex relationship between speech and text.

[0081] The cross-modal retriever 3 is used to perform cross-modal retrieval on speech representations and text queries, and obtain speech tokens of text embedding and speech embedding in the same embedding space.

[0082] The speech representation and text query are input into the cross-modal retriever, which realizes cross-modal retrieval by mapping the text query and speech representation into the same embedding space, and obtains speech tokens with text embedding and speech embedding in the same embedding space.

[0083] The core goal of the cross-modal retriever is to achieve efficient retrieval and alignment between text and speech modalities. The architecture is as follows: Figure 3 As shown in the figure, the key encoder, value encoder and retrieval enhancement decoder work together to significantly improve the cross-modal retrieval performance through multimodal alignment.

[0084] (1) The retrieval-enhanced decoder, as the core reasoning component, receives a prompt template, a speech representation, and a text query, and retrieves relevant context based on a value vector. The prompt template is a preset content, and the content of the prompt template is the same each time it is input. The retrieval-enhanced decoder integrates a cross-attention mechanism, allowing it to dynamically focus on the retrieved multimodal context when generating each text token.

[0085] The retrieval-enhanced decoder is built based on cross-attention and causal multi-head attention. The prompt template, speech representation, text query and value vector are processed by cross-attention and causal multi-head attention, and then passed through the language model head to obtain speech tokens.

[0086] Cross-Attention integrates the retrieved value vector into the generation process. The cross-attention layer uses standard multi-head attention (without causal masking) because the retrieved context is complete and there's no need to mask future information. The query comes from the hidden state of the retrieval-enhanced decoder, and the key / value comes from the output of the value encoder.

[0087] Causal Multi-Head Attention ensures that when generating text through autoregression, the output at each position depends only on the input before the current position. In the causal multi-head attention layer, a lower triangular mask matrix is ​​used to mask the attention weights for future positions.

[0088] The language model head is connected after the last hidden state layer of the retrieval-enhanced decoder. Its structure is a linear layer followed by a softmax layer, mapping the hidden state to vocabulary probability distribution information. Based on this information, the corresponding speech segment is found, and this information and the corresponding speech segment are combined to form speech tokens.

[0089] (2) The key encoder is built on causal multi-head attention and shares the same multimodal language model architecture as the retrieval-enhanced decoder. It is responsible for encoding the input sequence into a retrieval key. The key corresponds to the position of the text token in the retrieval database. Its encoding process integrates speech and text modal information to ensure that the retrieval key contains cross-modal semantic features.

[0090] Causal Multi-Head Attention (CMA) processes sequences (the sequence of prompt templates, speech representations, and text queries obtained through the embedding layer) and maintains strict temporal dependencies when generating key embeddings. When the input is a sequence of speech frames or text, a causal mask is used to ensure that the key embedding at each position depends only on the current and previous inputs. For example, when processing audio, the key embedding for the tth frame is calculated only based on the previous t frames to avoid information leakage. This has the advantage of generating time-aligned key embeddings, improving retrieval accuracy, especially in speech recognition, where phoneme order is crucial.

[0091] The search key is used to search in the search database, and the search value (the context window of the key token surrounding the search key in the search database) is obtained, and then input into the value encoder.

[0092] (3) The value encoder uses a lightweight multi-level Transformer architecture to encode the search value. Each search value contains a key tag, and after encoding, it generates a fixed-dimensional value vector through pooling.

[0093] Conventional multi-head attention processes retrieved context windows, such as text paragraphs or speech snippets, ensuring a complete context without blocking future information. By taking a fixed-length context window as input, such as a token sequence of [t−7, t+8], multi-head attention captures global dependencies. The output is a value embedding of the same dimension as the key embedding, which is used to retrieve the cross-attention of the enhanced decoder. This allows for the full modeling of complex relationships within the context, such as long-range dependencies and semantic cohesion.

[0094] (4) The process of creating a search database is as follows: Figure 4 Key generation: The hidden state corresponding to each text token in the retrieval database is used as the key. The encoding incorporates the associated speech context, such as the complete audio segment, to ensure that the key contains joint acoustic-semantic features. Value extraction: Centered on the key token, the multimodal token sequence (including speech and text tokens) within the preceding and following windows is extracted. These are processed by the value encoder and stored in the retrieval database. Figure 4 In

[15] , p0 represents the prompt template, x1 and x2 represent the voice representation, y0, y1, y2, and y3 represent the text query, and k0, k1, k2, and k3 represent the search keys.

[0095] (5) The working process of the cross-modal retriever is:

[0096] The prompt template, speech representation, and text query are input into a key encoder to obtain a search key. The key encoder adopts a multi-level Transformer architecture based on causal multi-head attention. The prompt template is a preset content, and the content of the prompt template is the same each time it is input.

[0097] Use the search key to search in the search database to obtain the search value;

[0098] The retrieval value is input into a value encoder to obtain a value vector. The value encoder adopts a lightweight multi-level Transformer architecture.

[0099] The prompt template, voice representation, text query and value vector are input into the retrieval enhancement decoder to obtain voice tokens. The retrieval enhancement decoder adopts a multi-level Transformer architecture based on cross attention and causal multi-head attention, and the language model head is connected after the multi-level Transformer architecture.

[0100] The Transformer architectures of the key encoder, value encoder, and retrieval-enhanced decoder all use a 15-layer architecture in this embodiment. Specifically, in the key encoder, each layer of the architecture includes a causal multi-head attention layer, an addition layer, a layer normalization, a feedforward layer, an addition layer, and a layer normalization connected in sequence, and the 15-layer architecture is preceded by an embedding layer. In the value encoder, each layer of the architecture includes a multi-head attention layer, an addition layer, a layer normalization, a feedforward layer, an addition layer, and a layer normalization connected in sequence, and the 15-layer architecture is preceded by an embedding layer and followed by a pooling layer. In the retrieval-enhanced decoder, each layer of the architecture includes a layer normalization, a cross-attention layer, an addition layer, a causal multi-head attention layer, an addition layer, a layer normalization, a feedforward layer, an addition layer, and a layer normalization connected in sequence, and the 15-layer architecture is preceded by an embedding layer and followed by a language model head.

[0101] The advantages of cross-modal retrievers are as follows:

[0102] Causal attention ensures generation consistency: The key encoder and decoder use causal multi-head attention to strictly maintain temporal dependencies, avoid future information leakage, and improve the logical coherence of the generated text.

[0103] Flexibility of Retrieval Enhancement: The value encoder uses ordinary multi-head attention to fully model the global semantics of the retrieval context and provide rich background knowledge for the retrieval enhancement decoder.

[0104] Efficient multimodal alignment: The key encoder (causal) and value encoder (non-causal) have clear division of labor. The former captures temporal features, while the latter extracts contextual semantics, jointly optimizing cross-modal alignment.

[0105] Key-value collaborative encoding: The key encoder and retrieval-enhanced decoder share model parameters, achieving end-to-end optimization of the encoding-retrieval-decoding process and reducing computational redundancy.

[0106] Multimodal context awareness: The retrieval value contains a mixed sequence of speech and text. The lightweight design of the value encoder effectively captures cross-modal dependencies while reducing computational overhead.

[0107] The speech language model 4 is used to obtain text hypotheses corresponding to the speech tokens based on the speech tokens.

[0108] The Speech Language Model (SLM) is a core component of this system. Its primary function is to generate high-quality text representations based on input speech tokens. These text representations are hypotheses of the text corresponding to the speech tokens. Unlike traditional generation methods based on automatic speech recognition (ASR), the SLM processes the speech data directly, completely bypassing the ASR system. This avoids the common error propagation problems in ASR systems, significantly improving the accuracy and reliability of the generated text.

[0109] The process of SLM generating text representations is as follows: SLM extracts semantic information from audio segments through a speech encoder and a speech adapter. The speech encoder converts the audio data into high-dimensional speech representations, while the speech adapter projects these representations into a text embedding space, aligning them with the text modality. SLM fuses the semantics of the query with the content of the audio segment to construct a unified contextual representation. This representation not only contains the detailed information of the audio segment but also reflects the intent and requirements of the query. Through multi-level attention mechanisms (such as the self-attention mechanism in the Transformer), SLM can capture the complex relationship between the audio segment and the query, thereby generating more accurate text representations. Based on this contextual information, SLM generates text representations relevant to the query. The generation process uses an autoregressive generative model to generate coherent text by gradually predicting the next word.

[0110] Because SLM processes speech data directly, it avoids the error propagation problem inherent in traditional ASR systems during speech-to-text conversion. Traditional ASR systems can introduce errors due to factors such as speech quality, accent, and background noise, which can further impact the performance of downstream tasks. SLM generates text directly from speech in an end-to-end manner, reducing errors in intermediate steps and improving the accuracy and robustness of the generated text. Furthermore, SLM can further enhance the quality of the generated text by introducing external knowledge bases or post-processing modules (semantic correction).

[0111] The output of the SLM is the final text representation (literal hypothesis), which can be directly used in the subsequent query generation module and decision support. The text representation has the following characteristics: High accuracy: Since it avoids the propagation of ASR errors, the generated text representation is more accurate. Semantic richness: The text generated based on context information not only contains factual content but also reflects the semantic intent of the query. Diversity: According to the task requirements, the SLM can generate various forms of text representations, such as answers, summaries, or explanations.

[0112] The query generation module 5 is used to extract query fragments that may contain entity names from the literal hypothesis.

[0113] The main function of the query generation module is to extract query fragments that may contain entity names based on the literal hypothesis. The query generation module analyzes the semantic information in the literal hypothesis, identifies and extracts key entities, thereby generating structured query fragments. The literal hypothesis is usually input in the form of natural language text, which may contain incomplete sentences, grammar errors, or ambiguous expressions. Query fragments that may contain entity names are phrases or keywords extracted from the literal hypothesis that may contain entity names. These phrases or keywords are the core content for subsequent retrieval or processing. It can be a single entity name, such as place name A, or a phrase containing an entity name, such as tourism in place name A. The query fragments have high accuracy and semantic relevance, and can accurately reflect the user's intent and support downstream tasks, such as information retrieval.

[0114] The processing process of the query generation module is as follows:

[0115] (1) Preprocessing of the literal hypothesis. Text cleaning: Clean the literal hypothesis, remove irrelevant characters, punctuation marks, and stop words, such as "de", "shi", etc., and retain the key semantic content. Word segmentation and词性标注: Use a word segmentation tool to segment the literal hypothesis and perform词性标注on each word, such as noun, verb, adjective, etc., for subsequent entity recognition.

[0116] (2) Named entity recognition (NER). Use a pre-trained named entity recognition (NER) model to perform entity tagging on the literal hypothesis after词性标注. The NER model can identify entity categories in the text, such as person names, place names, organization names, time, etc. The NER model tags each word in the literal hypothesis, identifies the entity names and their categories therein. According to the tagging results of the NER model, extract the fragments that may contain entity names.

[0117] It should be noted that the term "词性标注" in Chinese is not accurately translated as it seems to be a placeholder in the original text. You may need to provide the correct English term for it if it has a specific meaning in the context.(3) Query fragment generation. The extracted fragments that may contain entity names are combined with relevant context words to generate query fragments. The generated query fragments are filtered to remove redundant or irrelevant query fragments. The query fragments are sorted according to their importance or relevance, ensuring that the most important or most relevant fragments are output first, and query fragments that may contain entity names are output.

[0118] The advantages of the query building module are as follows:

[0119] Domain Adaptation: Based on domain-specific textual hypotheses, the NER model is fine-tuned to improve its performance in specific domains.

[0120] Multi-task learning: When training the NER model, other related tasks such as part-of-speech tagging and semantic role labeling are optimized simultaneously to improve the generalization ability of the model.

[0121] Data enhancement: Through data enhancement techniques such as synonym replacement and noise addition, more training data is generated to improve the robustness of the model.

[0122] Diversity of query snippets: Single-entity snippets: snippets containing only a single entity name. Multi-entity snippets: snippets containing multiple entity names. Context-enhanced snippets: snippets containing an entity name and its context.

[0123] Error correction and post-processing: Syntax check: Perform syntax check on the generated query fragments to ensure that they comply with language specifications. Semantic correction: Perform semantic correction on the query fragments through external knowledge base or context information to ensure that they are consistent with the user's intentions. Figure 1 To.

[0124] Through the above design, the query generation module can efficiently extract key entities from text hypotheses and generate high-quality query fragments, providing strong support for downstream tasks.

[0125] The entity retrieval module 6 is used to search the entity database according to the query fragment that may contain the entity name, and obtain the relevant entity name.

[0126] The entity retrieval module's primary function is to retrieve entity information related to a query fragment from an entity database (a large-scale knowledge base). This module combines acoustic neighbor embedding, semantic embedding, and traditional text retrieval methods to perform multi-level similarity calculations, thereby improving retrieval accuracy and robustness.

[0127] The implementation process of the entity retrieval module is as follows:

[0128] (1) Acoustic Neighbor Embeddings

[0129] Acoustic Neighbor Embedding is used to capture the acoustic similarity of text fragments and is particularly suitable for processing speech-related query fragments. The specific implementation is as follows: Embedding vectors are generated through acoustic, speech, and spelling encoders. The acoustic encoder extracts the acoustic features of the speech signal, the speech encoder converts the speech into a high-dimensional speech representation, and the spelling encoder maps the text fragments into the spelling space. The generated embedding vectors are used to calculate the acoustic similarity between text fragments. The acoustic similarity between the query fragment and the entity names in the knowledge base is evaluated using metrics such as cosine similarity or Euclidean distance. Acoustic Neighbor Embedding is particularly suitable for processing entity names that have similar pronunciations but different spellings, thereby improving the recall rate of retrieval.

[0130] (2) Semantic Embeddings

[0131] Semantic embedding is used to capture the semantic similarity of text fragments and ensure that the retrieval results are consistent with the semantic meaning of the query fragment. Figure 1 The specific implementation is as follows: Semantic embedding vectors are generated using the pre-trained T5 model (Text-To-Text Transfer Transformer). The T5 model generates high-dimensional semantic representations by inputting text snippets into an encoder. The generated semantic embedding vectors are used to calculate the semantic similarity between text snippets. Metrics such as cosine similarity or dot product are used to evaluate the semantic similarity between query snippets and entity names in the knowledge base. Semantic embedding is particularly suitable for processing synonyms, near-synonyms, or semantically related entity names, thereby improving retrieval accuracy.

[0132] (3) Traditional text retrieval methods

[0133] This example uses Okapi BM25 to generate query and key vectors and calculate the relevance score between them. Specifically, Okapi BM25 calculates the relevance between query fragments and documents based on term frequency (TF) and inverse document frequency (IDF). Term frequency measures the frequency of query terms in a document, while inverse document frequency measures the importance of a query term within the entire document collection.

[0134] Okapi BM25 calculates the relevance score between the query fragment and the document using the following formula :

[0135]

[0136] in, It's a document. is the query fragment, is the i-th query word, yes The inverse document frequency of yes exist The word frequency in is the length of the document, is the average length of the document collection, is a parameter to adjust the word frequency saturation, It is a parameter that adjusts the degree of document length normalization.

[0137] Okapi BM25 is particularly suitable for processing short text query fragments (such as keywords or phrases) and can quickly retrieve highly relevant documents.

[0138] The entity retrieval module uses a multimodal fusion strategy: it combines the results of acoustic neighbor embedding, semantic embedding, and Okapi BM25. This fusion can be achieved by combining the scores of these three methods through weighted summation or using a machine learning model (such as a ranking model) to generate relevant entity names (retrieval results).

[0139] Sorting and filtering search results: After generating search results, you can sort and filter the results using the following strategies:

[0140] (1) Score-based ranking: retrieval results are ranked according to the scores of acoustic neighbor embedding, semantic embedding, and Okapi BM25.

[0141] (2) Context-based filtering: Filter the search results based on the contextual information of the query fragment (such as time and place) to ensure that they are consistent with the user's intention. Figure 1 To.

[0142] (3) Optimization based on user feedback: Optimize search results through user feedback (such as click-through rate and satisfaction score) to improve the user experience of the system.

[0143] Through the above design, the entity retrieval module can efficiently retrieve entity information related to the query fragment from the knowledge base, providing strong support for downstream tasks.

[0144] The context construction module 7 is used to construct context information based on relevant entity names and text assumptions.

[0145] The core task of the context-building module is to combine the retrieved relevant entity names with the text hypotheses to form contextual information suitable for processing. This process aims to provide rich background information, enabling it to better understand the semantic content of the text hypotheses and correct any errors, especially those in entity names.

[0146] The process of the context building module is:

[0147] (1) Entity filtering

[0148] Entity filtering is the first step in context building. Its goal is to filter out the entities most relevant to the text hypothesis from the relevant entity names and filter out irrelevant entities based on predefined Euclidean distance thresholds and count thresholds. The query content is obtained based on the text hypothesis.

[0149] Euclidean distance threshold: Evaluate the similarity between an entity and a query by calculating the Euclidean distance between the query vector and the entity vector. The smaller the Euclidean distance, the more similar the entity is to the query. Set a predefined threshold, such as 1.0, and only retain entities with a distance less than the threshold. For example, if the distance between the query and entity A is , the distance from entity B is , then entity A is retained and entity B is filtered out.

[0150] Count threshold: To control the number of entities returned for each query, set a count threshold. Even if multiple entities meet the Euclidean distance threshold, only the top N entities with the smallest distance are retained. For example, if this threshold is set to 3, a maximum of 3 entities will be returned for each query.

[0151] After the above filtering steps, candidate entities that are acoustically or semantically closest to the text hypothesis are obtained. These entities are not only highly relevant to the query, but also provide high-quality information for subsequent context construction.

[0152] (2) Prompt construction

[0153] Construct hint information for each candidate entity. The format of the hint information can be adjusted according to the task requirements. For example:

[0154] Simple tip: only include candidate entities. Example: [H] Place B.

[0155] Complex hint: contains candidate entities and their queries. For example: [H]Query: Place A → Entity: Place B. This embodiment uses the format of complex hint.

[0156] Hints provide additional context to the subsequent Large Language Model (LLM), helping it understand which entities may be related to the incorrect portion of the text hypothesis. Complex hints contain not only the retrieved candidate entity but also the query that generated it. This additional contextual information helps the LLM better understand the relationship between the entity and the query. Not only does it tell the LLM that "Place B" is a possible entity, but it also explicitly states that this entity was retrieved from the query "Place A." This information helps the LLM infer the incorrect portion of the text hypothesis and generate the correct correction.

[0157] (3) Contextual information construction

[0158] Combine the prompt information with the text hypothesis to form a complete context.

[0159] This example uses tags to separate prompts and textual assumptions. For example:

[0160] [H] Location B

[0161] [H] Location C

[0162] [A] Travel to location A [P]

[0163] Among them: [H] represents the prompt information; [A] represents the text hypothesis; [P] represents the prompt for LLM to start prediction.

[0164] (4) Context optimization:

[0165] To reduce computational and memory overhead, the length of context information is usually limited to a certain range. We experimentally determine the optimal hint format and context length to minimize resource consumption while maintaining high accuracy.

[0166] The large language model 8 is used to obtain semantic retrieval results based on context information.

[0167] The core task of the Large Language Model (LLM) is to generate corrected text output based on contextual information. By deeply understanding contextual cues and textual assumptions, the LLM infers the most likely correct text and corrects any errors. Large language models are particularly important in body camera scenarios, significantly improving the accuracy and semantic consistency of text generation.

[0168] The implementation process of the large language model is as follows:

[0169] (1) Model selection and adjustment:

[0170] To adapt to the specific task requirements of body camera scenarios, the LLM application module uses a pre-trained large language model as the base model and adjusts it through the following methods:

[0171] LoRA Adapter: Adds a low-rank adapter (LoRA) to the LLM. LoRA introduces a small number of trainable parameters, enabling the model to quickly adapt to specific tasks without significantly increasing computational overhead. Specifically, the LoRA adapter adjusts some of the model's weights through low-rank matrix factorization, significantly reducing the computational cost of training and inference while maintaining high model performance.

[0172] Task fine-tuning: The LLM is fine-tuned on datasets related to body camera scenarios to better understand the terminology, entity names, and contextual information in the body camera domain. For example, targeted training and optimization is performed on entities such as place names and personal names in video surveillance systems.

[0173] (2) Contextual input processing

[0174] LLM uses a self-attention mechanism and a multi-layer Transformer architecture to deeply understand the relationship between prompt information and textual hypotheses. The self-attention mechanism captures long-range dependencies in the context, while the multi-layer Transformer architecture extracts higher-level semantic information through layer-by-layer abstraction.

[0175] (3) Error Correction and Text Generation

[0176] LLM infers the incorrect parts of the text hypothesis based on contextual information and generates corrected text. The specific process is as follows:

[0177] Error Detection: The LLM compares the contextual prompt with the textual hypothesis and identifies errors in the textual hypothesis. For example, given the prompt "Place B" and the textual hypothesis "Travel to Place A," the LLM can identify "Place A" as an incorrect entity. However, if Place A is a valid entity, the entity retrieval module can retrieve Place A from the entity database and identify it as a related entity name.

[0178] Text Generation: LLM generates corrected text based on contextual information. For example, powered by a multi-layered Transformer architecture, it analyzes the current input while also incorporating historical behavioral data and preference trends. If a user has previously expressed a preference for modern cityscapes or European-style architecture, LLM may infer that they may be interested in Location B. Conversely, Location C's traditional and historical content may not meet the user's current needs and is therefore automatically excluded. Consequently, "traveling to Location A" is corrected to "traveling to Location B."

[0179] Decoding strategy: LLM typically uses greedy decoding to generate the most likely output sequence. Greedy decoding selects the word with the highest probability at each step, thereby generating coherent and semantically accurate text and obtaining semantic retrieval results.

[0180] (4) Output results

[0181] LLM outputs semantic retrieval results for subsequent semantic retrieval or other tasks.

[0182] (5) Performance evaluation:

[0183] In order to evaluate the corrective effect of LLM, the following indicators and methods were used:

[0184] Word Error Rate (WER): WER is the core indicator for measuring the effectiveness of text correction. The calculation formula is:

[0185] .

[0186] Experiments show that the Retrieval-Augmented Generation (RAG)-based approach can significantly reduce WER (a relative reduction of 39.9%) when correcting rare entity name errors, without causing performance degradation on queries in other domains.

[0187] Domain Adaptability Evaluation: We evaluated the domain adaptability of the LLM by comparing its performance in body camera scenarios with other scenarios (such as healthcare and finance). Experimental results show that the LLM tuned with the LoRA adapter significantly outperforms the untuned model in the body camera scenario.

[0188] User satisfaction evaluation: The practicality and accuracy of LLM-generated texts are evaluated through user feedback (e.g., click-through rate, satisfaction score).

[0189] This example verifies the performance of this system on a variety of dialect speech datasets, focusing on the reduction of word error rate (WER) and the improvement of recognition accuracy to ensure that this system can effectively improve the accuracy and efficiency of semantic retrieval of body camera audio.

[0190] 1. Validation Dataset

[0191] The following dialect speech datasets were used in this verification:

[0192] 1. Sichuan dialect natural dialogue voice data collected by mobile phone

[0193] Dataset Description: Recorded by 1,730 native Sichuanese speakers, these conversations take place in a natural, face-to-face manner, with no restrictions on topics. The recordings cover a wide range of topics, with natural and fluent speech, consistent with real-world conversation scenarios. The human transcriptions offer high accuracy. Data Volume: 799 hours. Data Format: 16kHz, uncompressed 16-bit, WAV, mono. Composition: 1,730 recorders, 74% of whom are female and 88% are under 25 years old. All recorders are from Sichuan or Chongqing. Recording Content: Recorders engage in casual, natural conversations between two speakers, with no specific topics.

[0194] 2. Henan dialect natural dialogue voice data collected by mobile phone

[0195] Dataset Description: Recorded by 762 native Henan speakers, covering a wide age range with a balanced gender ratio. No specific topics or pre-set text are specified, and pairs of recorders engage in free conversation. This dataset can be used for speech recognition acoustics, voiceprint recognition model training, or algorithm research. Data Volume: 377 hours. Data Format: 16kHz, uncompressed 16-bit, WAV, mono. Composition: Approximately 762 recorders, all from Henan, speak the Henan dialect. Recording Content: No specific topics are specified, and pairs of recorders engage in free conversation.

[0196] 3. Collecting Voice Data from Northeastern Dialect Mobile Phones

[0197] Dataset Overview: 67 hours of Northeastern dialect mobile phone voice data recorded by 312 Northeastern dialect speakers, most of whom are from the three northeastern provinces. All speakers read the text in Northeastern dialect, with rich content covering customer consultations and text messages from nearly 30 different sectors. Sentences were manually transcribed and proofread by professional annotators, achieving high accuracy. Data Volume: 67 hours. Data Format: 16kHz, 16-bit, uncompressed WAV, mono. Composition: 312 people from Northeast China; 218 (70%) are women. Recording Content: Text read in Northeastern dialect, covering customer consultations and text messages from nearly 30 different sectors.

[0198] 4. Suzhou dialect mobile phone voice data collection

[0199] Dataset Overview: 176 hours of Suzhou dialect mobile phone voice data collected by 250 speakers, each recording approximately 500 Suzhou dialect colloquial sentences. Local Suzhou speakers were selected for the recordings, ensuring authentic accents, colloquial and everyday content, and natural and fluent readings. Quality inspection and proofreading by Suzhou locals ensures more accurate transcription. Data Volume: 176 hours. Data Format: 16kHz, 16-bit, uncompressed WAV, mono. Speaker Composition: 250 Suzhou residents; 129 (52%) are female. Recording Content: Colloquial Suzhou dialect sentences with colloquial and everyday content.

[0200] 5. Collecting Voice Data from Kunming Dialect Mobile Phones

[0201] Dataset Description: 1002 hours of Kunming dialect mobile phone speech data, recorded by 2284 native Kunming speakers with authentic accents, covering a wide range of age groups. The transcripts cover a wide range of categories, including general use, interactive, in-car, and household use, and are rich in content. Quality inspection and proofreading by Kunming locals ensures more accurate transcriptions. Data Volume: 1002 hours. Data Format: 16kHz, 16-bit, uncompressed WAV, mono. Data Composition: 2284 individuals, 60% of whom are female; 80% of the recorders are primarily aged 16-25; and all recorders are locals from Kunming and surrounding areas. Recording Content: General use, interactive, household commands, numbers, and dialect terms.

[0202] 6. Changsha dialect mobile phone voice data collection

[0203] Dataset Overview: 997 hours of Changsha dialect mobile phone voice data collected by over 2,000 native Changsha speakers, covering a diverse age range with a balanced gender ratio and authentic accents. The recordings cover a wide range of categories, including general use, interactive, in-car, and home use, and are rich in content. Changsha locals participated in quality inspection and proofreading, achieving a sentence accuracy of 95%. Data Volume: 997 hours. Data Format: 16kHz, 16-bit, uncompressed WAV, mono. Speaker Composition: 2,301 people, 1,030 males and 1,271 females. Recording Content: Text from a variety of categories, including general use, interactive, in-car, and home use.

[0204] 7. Collecting Voice Data from Chinese Mobile Phones

[0205] Dataset Overview: 200 Chinese mobile phone voice data collected by 200 native Chinese speakers, covering all major dialect regions. The recordings were recorded in both quiet and noisy environments, making them more suitable for real-world speech recognition applications. The recordings consist of commonly used, spoken sentences, read naturally and fluently. Human transcription ensures high accuracy. Data size: 12 hours. Data format: 16kHz, 16-bit, uncompressed WAV, mono. Composition: 200 Chinese speakers, 53% of whom are female. Recording content: Commonly used, spoken sentences.

[0206] 2. Verification Method

[0207] 1. Baseline Model Comparison

[0208] Using the traditional ASR speech recognition system as the baseline model, speech recognition tests were performed on the above dialect datasets, and the word error rate (WER) and recognition accuracy were recorded.

[0209] 2. Application of the Retrieval Enhancement System (this system)

[0210] The retrieval enhancement system is applied to the same dialect dataset to preprocess and semantically associate the speech data before performing speech recognition.

[0211] 3. Performance Index Comparison

[0212] Compare the word error rate (WER) and recognition accuracy of the baseline model and the retrieval enhancement system, and calculate the improvement percentage.

[0213] 3. Verification Results

[0214] 1. Baseline model testing

[0215] 1. Sichuan Dialect Dataset

[0216] Baseline model word error rate (WER): 25%; baseline model recognition accuracy: 75%.

[0217] 2. Henan Dialect Dataset

[0218] Baseline model word error rate (WER): 26%; baseline model recognition accuracy: 74%.

[0219] 3. Northeastern Dialect Dataset

[0220] Baseline model word error rate (WER): 24%; baseline model recognition accuracy: 76%.

[0221] 4. Suzhou Dialect Dataset

[0222] Baseline model word error rate (WER): 27%; baseline model recognition accuracy: 73%.

[0223] 5Kunming Dialect Dataset

[0224] Baseline model word error rate (WER): 25%; average baseline model recognition accuracy: 75%.

[0225] 6. Changsha Dialect Dataset

[0226] Baseline model word error rate (WER): 26%; average baseline model recognition accuracy: 74%.

[0227] 7. Chinese mobile phone voice data set

[0228] Baseline model word error rate (WER): 28%; average baseline model recognition accuracy: 72%.

[0229] (2) Application of search enhancement system

[0230] 1. Sichuan Dialect Dataset

[0231] The word error rate (WER) after applying the retrieval enhancement system is 16%; the recognition accuracy after applying the retrieval enhancement system is 84%.

[0232] 2. Henan Dialect Dataset

[0233] The word error rate (WER) after applying the retrieval enhancement system is 17%; the recognition accuracy after applying the retrieval enhancement system is 83%.

[0234] 3. Northeastern Dialect Dataset

[0235] The word error rate (WER) after applying the retrieval enhancement system is 16%; the recognition accuracy after applying the retrieval enhancement system is 84%.

[0236] 4. Suzhou Dialect Dataset

[0237] The word error rate (WER) after applying the retrieval enhancement system is 15%; the recognition accuracy after applying the retrieval enhancement system is 83%.

[0238] 5. Kunming Dialect Dataset

[0239] The word error rate (WER) after applying the retrieval enhancement system is 16%; the recognition accuracy after applying the retrieval enhancement system is 84%.

[0240] 6. Changsha Dialect Dataset

[0241] Word Error Rate (WER) after applying the retrieval enhancement system: 12% Recognition Accuracy after applying the retrieval enhancement system: 88%

[0242] 7. Chinese mobile phone voice data set

[0243] The word error rate (WER) after applying the retrieval enhancement system is 18%; the recognition accuracy after applying the retrieval enhancement system is 82%.

[0244] (3) Reduction of Word Error Rate (WER)

[0245] Compared with the baseline model, the word error rate of this system is reduced and the reduction ratio is as follows:

[0246] 1. Sichuan Dialect Dataset

[0247] Word error rate reduced: 25%-16%=9%; reduction ratio: 9% / 25%×100%=36%.

[0248] 2. Henan Dialect Dataset

[0249] The word error rate decreased by: 26%-17%=9%; the reduction ratio was: 9% / 26%×100%=34.6%.

[0250] 3. Northeastern Dialect Dataset

[0251] The word error rate decreased by: 24%-16%=8%; the reduction ratio was: 8% / 24%×100%=33.3%.

[0252] 4. Suzhou Dialect Dataset

[0253] The word error rate decreased by: 27%-15%=12%; the reduction ratio was: 12% / 27%×100%=44.4%.

[0254] 5. Kunming Dialect Dataset

[0255] Word error rate reduced: 25%-16%=9%; reduction ratio: 9% / 25%×100%=36%.

[0256] 6. Changsha Dialect Dataset

[0257] The word error rate decreased: 26%-12%=14%; the reduction ratio: 14% / 26%×100%=53.8%.

[0258] 7. Chinese mobile phone voice data set

[0259] Word error rate reduced: 28%-18%=10%; reduction ratio: 10% / 28%×100%=35.7%.

[0260] (IV) Improvement in Recognition Accuracy

[0261] Compared with the baseline model, the improvement in recognition accuracy of this system and the improvement ratio are as follows:

[0262] 1. Sichuan Dialect Dataset

[0263] Improvement in recognition accuracy: 84% - 75% = 9%; improvement ratio: 9% / 75% × 100% = 12%.

[0264] 2. Henan Dialect Dataset

[0265] Improvement in recognition accuracy: 83% - 74% = 9%; improvement ratio: 9% / 74% × 100% = 12.2%.

[0266] 3. Northeastern Dialect Dataset

[0267] Improvement in recognition accuracy: 84% - 76% = 8%; improvement ratio: 8% / 76% × 100% = 10.5%.

[0268] 4. Suzhou Dialect Dataset

[0269] Improvement in recognition accuracy: 83% - 73% = 10%; improvement ratio: 10% / 73% × 100% = 13.7%.

[0270] 5. Kunming Dialect Dataset

[0271] Improvement in recognition accuracy: 84% - 75% = 9%; improvement ratio: 9% / 75% × 100% = 12%.

[0272] 6. Changsha Dialect Dataset

[0273] Improvement in recognition accuracy: 88% - 74% = 14%; improvement ratio: 14% / 74% × 100% = 18.9%.

[0274] 7. Chinese mobile phone voice data set

[0275] Improvement in recognition accuracy: 82% - 72% = 10%; improvement ratio: 10% / 72% × 100% = 13.9%.

[0276] IV. Conclusion

[0277] This validation demonstrated that the system significantly reduced the word error rate (WER) by an average of 39.9% across multiple dialect speech datasets, while simultaneously improving recognition accuracy by an average of 13.4%. In some dialect datasets, the WER reduction reached as high as 44.4%, while the recognition accuracy improved by 18.9%. This demonstrates that the system can effectively improve the accuracy and efficiency of semantic retrieval from body camera audio and holds promising application prospects.

[0278] In this embodiment, Figure 5 As shown, a method for audio semantic retrieval of law enforcement recorders based on retrieval enhancement is also provided, and the technical solution adopted is as follows: comprising the following steps:

[0279] Step 1: Obtain the audio data and text query recorded by the body camera.

[0280] Step 2: Project the audio data into the text embedding space to obtain the speech representation.

[0281] First, the audio data is fed into a speech encoder to generate a high-dimensional speech representation. The speech encoder is pre-trained using self-supervised learning on cross-dialect speech data. This high-dimensional speech representation is then fed into a downsampler and projected into a text embedding space to generate a speech representation.

[0282] Step 3: Perform cross-modal retrieval on the speech representation and text query to obtain speech tokens with text embedding and speech embedding in the same embedding space.

[0283] The speech representation and text query are input into the cross-modal retriever. The working process of the cross-modal retriever is as follows:

[0284] The prompt template (own by the cross-modal retriever), the speech representation and the text query are input into the key encoder to obtain the retrieval key. The key encoder adopts a multi-layer Transformer architecture based on causal multi-head attention;

[0285] Use the search key to search in the search database to obtain the search value;

[0286] The retrieval value is input into a value encoder to obtain a value vector, wherein the value encoder adopts a multi-level Transformer architecture;

[0287] The prompt template, speech representation, text query and value vector are input into the retrieval enhancement decoder to obtain speech tokens. The retrieval enhancement decoder adopts a multi-level Transformer architecture based on cross attention and causal multi-head attention, and the multi-level Transformer architecture is connected to the language model head.

[0288] The Transformer architectures of the key encoder, value encoder, and retrieval-enhanced decoder all use a 15-layer architecture in this embodiment. Specifically, in the key encoder, each layer of the architecture includes a causal multi-head attention layer, an addition layer, a layer normalization, a feedforward layer, an addition layer, and a layer normalization connected in sequence, and the 15-layer architecture is preceded by an embedding layer. In the value encoder, each layer of the architecture includes a multi-head attention layer, an addition layer, a layer normalization, a feedforward layer, an addition layer, and a layer normalization connected in sequence, and the 15-layer architecture is preceded by an embedding layer and followed by a pooling layer. In the retrieval-enhanced decoder, each layer of the architecture includes a layer normalization, a cross-attention layer, an addition layer, a causal multi-head attention layer, an addition layer, a layer normalization, a feedforward layer, an addition layer, and a layer normalization connected in sequence, and the 15-layer architecture is preceded by an embedding layer and followed by a language model head.

[0289] Step 4: Based on the speech tokens, obtain the text hypothesis corresponding to the speech tokens.

[0290] Input speech tokens into the speech language model to obtain text hypotheses.

[0291] Step 5: Extract query fragments that may contain entity names from the text hypotheses.

[0292] The text hypothesis is fed into the query generation module to obtain query fragments that may contain entity names.

[0293] Step 6: Based on the query fragment that may contain the entity name, search the entity database to obtain the relevant entity names.

[0294] A query fragment that may contain an entity name is input into the entity retrieval module. The entity retrieval module uses acoustic neighbor embedding, semantic embedding, and traditional text retrieval methods to search the entity database, fuse the results of the three methods, and generate relevant entity names. The number of relevant entity names can be one or more.

[0295] Step 7: Build context information based on relevant entity names and textual assumptions.

[0296] The relevant entity names and text hypotheses are input into the context construction module, irrelevant entities are filtered out according to the predefined Euclidean distance threshold and count threshold, candidate entities are screened out from the relevant entity names, and then context information containing candidate entity and text hypothesis information is constructed.

[0297] Step 8: Obtain semantic retrieval results based on context information.

[0298] The context information is input into the large language model. The large language model infers the incorrect parts of the text hypothesis based on the context information, and refers to past behavioral data and preference trends to generate corrected text. That is, the incorrect entity name in the text hypothesis is corrected to the candidate entity that best conforms to past behavioral data and preference trends, and then the semantic retrieval results are output.

[0299] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A law enforcement recorder audio semantic retrieval system based on retrieval enhancement, characterized in that: include: Data acquisition module, used to obtain audio data and text query recorded by law enforcement recorders; The speech adapter module is used to project audio data into the text embedding space to obtain speech representation; A cross-modal retriever, which is used to perform cross-modal retrieval on speech representations and text queries, and obtain speech tokens with text embedding and speech embedding in the same embedding space; The speech language model is used to obtain text hypotheses corresponding to speech tokens based on speech tokens; A query generation module, which extracts query fragments that may contain entity names from text hypotheses; An entity retrieval module is used to search the entity database and obtain relevant entity names based on query fragments that may contain entity names; The specific process is as follows: the entity retrieval module retrieves entity information related to query fragments that may contain entity names from the entity database, combining acoustic neighbor embedding, semantic embedding, and traditional text retrieval methods for retrieval; the results of acoustic neighbor embedding, semantic embedding, and traditional text retrieval methods are integrated to generate relevant entity names; The context building module is used to build context information based on relevant entity names and textual assumptions; The process of the context building module is: Based on the Euclidean distance threshold and the count threshold, the N entities most relevant to the text hypothesis are selected from the relevant entity names to obtain candidate entities. The specific process is as follows: Calculate the Euclidean distance between the query vector and the entity vector, set the Euclidean distance threshold, and retain entities whose distance is less than the Euclidean distance threshold; To control the number of entities returned for each query, a counting threshold is set. If the distances between multiple entities meet the Euclidean distance threshold, only the top N entities with the smallest distances are retained. The candidate entities that are closest to the text hypothesis in acoustics or semantics are obtained. Construct hint information for each candidate entity; The prompt information contains only the candidate entity, or the candidate entity and its query; Combine prompt information with textual hypotheses to form contextual information; A large language model is used to generate the most likely output sequence based on context information using greedy decoding to obtain semantic retrieval results. The processing process of the large language model is: The large language model uses a self-attention mechanism and a multi-layer Transformer structure to deeply understand the relationship between the prompt information in the context information and the text hypothesis; Infer errors in textual assumptions and generate corrected text; By comparing the hints in the context with the textual assumptions, the incorrect parts of the textual assumptions can be identified. Referencing past behavioral data and preference trends, it generates corrected text based on contextual information; Get semantic retrieval results.

2. The audio semantic retrieval system for law enforcement recorders based on retrieval enhancement according to claim 1, characterized in that: The voice adapter module includes a voice encoder and a downsampler, A speech encoder, used to convert audio data into a high-dimensional speech representation. The speech encoder is pre-trained on cross-dialect speech data through self-supervised learning; The downsampler is used to project the high-dimensional speech representation into the text embedding space to obtain the speech representation.

3. The audio semantic retrieval system for law enforcement recorders based on retrieval enhancement according to claim 1, characterized in that: The working process of the cross-modal retriever is: The prompt template, speech representation and text query are input into a key encoder to obtain a retrieval key. The key encoder adopts a multi-level Transformer architecture based on causal multi-head attention; Search the search database using the search key to obtain the search value; The retrieval value is input into a value encoder to obtain a value vector, wherein the value encoder adopts a multi-level Transformer architecture; The prompt template, speech representation, text query and value vector are input into the retrieval enhancement decoder to obtain speech tokens. The retrieval enhancement decoder adopts a multi-level Transformer architecture based on cross attention and causal multi-head attention, and the multi-level Transformer architecture is connected to the language model head.

4. The audio semantic retrieval system for law enforcement recorders based on retrieval enhancement according to claim 1, characterized in that: The processing process of the query generation module is: Clean the text hypothesis; use the word segmentation tool to segment the text hypothesis and tag each word with part of speech; Use the named entity recognition model to tag each word in the text hypothesis, identify the entity name and its category, and extract the fragments that may contain the entity name; Combine fragments that may contain entity names with relevant context words to generate query fragments; sort the query fragments according to their importance or relevance, ensuring that the most important or most relevant fragments are output first, and output query fragments that may contain entity names.

5. A semantic retrieval method for law enforcement recorder audio based on retrieval enhancement, characterized in that: Using the law enforcement recorder audio semantic retrieval system based on retrieval enhancement according to any one of claims 1 to 4, comprising the following steps: Obtain audio data and text queries recorded by body cameras; Project the audio data into the text embedding space to obtain speech representation; Perform cross-modal retrieval on speech representations and text queries to obtain speech tokens with text embedding and speech embedding in the same embedding space; According to the speech tokens, get the text hypothesis corresponding to the speech tokens; Extract query fragments that may contain entity names from text hypotheses; According to the query fragment that may contain the entity name, the entity database is searched to obtain the relevant entity names; Construct contextual information based on relevant entity names and textual assumptions; According to the context information, semantic retrieval results are obtained.

Images