Law enforcement recorder audio semantic retrieval method and system based on retrieval enhancement

Abstract

The invention relates to the technical field of speech recognition and natural language processing, and particularly discloses a law enforcement recorder audio semantic retrieval method and system based on retrieval enhancement, and the system comprises a data acquisition module which is used for acquiring audio data and text query; the voice adapter module is used for projecting the audio data to a text embedding space to obtain voice representation; the cross-modal retriever is used for carrying out cross-modal retrieval on the voice representation and the text query to obtain voice Tokens; the voice language model is used for obtaining character hypotheses; the query generation module is used for extracting query fragments possibly containing entity names; the entity retrieval module is used for retrieving an entity database according to the query fragment to obtain a related entity name; the context construction module is used for constructing context information; and the large language model is used for obtaining a semantic retrieval result. According to the method, the identification and retrieval accuracy of the entity name and key information in the law enforcement recorder audio is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech recognition and natural language processing, and in particular to a method and system for audio semantic retrieval of law enforcement recorders based on retrieval enhancement. Background Art

[0002] Body cameras are widely used in the law enforcement process and can record audio and video information at the scene of law enforcement. However, due to environmental noise, accent differences, and the limitations of speech recognition technology, traditional automatic speech recognition (ASR) systems have a high error rate when processing body camera audio, especially when recognizing uncommon entity names such as names of people, places, license plates, etc. Traditional ASR systems rely on large-scale training data, but in law enforcement scenarios, the entity names involved in body camera audio are often low-frequency and diverse. Traditional ASR systems find it difficult to accurately identify these entities, resulting in a large number of errors in semantic retrieval results.

[0003] In recent years, retrieval-augmented generation (RAG) technology has performed well in natural language processing tasks, especially when supported by external knowledge bases, which can significantly improve the performance of language models. However, existing RAG systems mainly rely on ASR transcriptions, which causes ASR errors to propagate to subsequent retrieval and generation steps. Due to the lack of support from external knowledge bases, it performs poorly when processing low-frequency entities and complex semantics, and it is difficult to correct errors in the ASR system.

[0004] In summary, the existing law enforcement recorder audio semantic retrieval technology has significant deficiencies in entity name recognition, semantic understanding, external knowledge base support, adaptability and scalability, and it is difficult to meet the needs of law enforcement work for efficient, accurate and real-time processing. Summary of the invention

[0005] The present invention aims to solve the above problems. To this end, the present invention provides a law enforcement recorder audio semantic retrieval method and system based on retrieval enhancement, which directly processes voice data and combines external knowledge base and large language model to avoid the error propagation problem in traditional automatic speech recognition systems, significantly improve the retrieval and generation effects, and improve the recognition and retrieval accuracy of entity names and key information in law enforcement recorder audio.

[0006] The present invention provides an audio semantic retrieval system for law enforcement recorders based on retrieval enhancement, and the technical scheme adopted is as follows: comprising: Data acquisition module, used to obtain audio data and text query recorded by law enforcement recorders; The speech adapter module is used to project the audio data into the text embedding space to obtain the speech representation; A cross-modal retriever is used to perform cross-modal retrieval on speech representations and text queries, obtaining speech tokens with text embeddings and speech embeddings in the same embedding space; The speech language model is used to obtain text hypotheses corresponding to speech tokens based on speech tokens; A query generation module, which is used to extract query fragments that may contain entity names from textual hypotheses; An entity retrieval module, used to retrieve an entity database and obtain relevant entity names according to a query fragment that may contain an entity name; A context building module, used to build context information based on relevant entity names and textual assumptions; Large language model, used to obtain semantic retrieval results based on contextual information.

[0007] Furthermore, the speech adapter module includes a speech encoder and a downsampler. A speech encoder, which is used to convert audio data into a high-dimensional speech representation. The speech encoder is pre-trained on cross-dialect speech data through self-supervised learning; The downsampler is used to project the high-dimensional speech representation into the text embedding space to obtain the speech representation.

[0008] Furthermore, the working process of the cross-modal retriever is: The prompt template, the speech representation and the text query are input into a key encoder to obtain a retrieval key, wherein the key encoder adopts a multi-level Transformer architecture based on causal multi-head attention; Using the search key to search in the search database, obtain the search value; The retrieval value is input into a value encoder to obtain a value vector, wherein the value encoder adopts a multi-level Transformer architecture; The prompt template, voice representation, text query and value vector are input into the retrieval enhancement decoder to obtain voice Tokens. The retrieval enhancement decoder adopts a multi-level Transformer architecture based on cross attention and causal multi-head attention. The multi-level Transformer architecture is connected to the language model head.

[0009] Furthermore, the processing of the query generation module is as follows: Clean the textual assumptions; use the word segmentation tool to segment the textual assumptions and perform part-of-speech tagging on each word; Use the named entity recognition model to tag each word in the text hypothesis, identify the entity name and its category, and extract the fragments that may contain the entity name; Combine fragments that may contain entity names with relevant context words to generate query fragments; sort the query fragments according to their importance or relevance to ensure that the most important or most relevant fragments are output first, and output query fragments that may contain entity names.

[0010] Furthermore, the entity retrieval module combines acoustic neighbor embedding, semantic embedding and traditional text retrieval methods for retrieval.

[0011] Furthermore, the results of acoustic neighbor embedding, semantic embedding and traditional text retrieval methods are fused to generate relevant entity names.

[0012] Furthermore, the processing of the context building module is as follows: According to the Euclidean distance threshold and the count threshold, N entities most relevant to the text hypothesis are selected from the relevant entity names to obtain candidate entities; Construct hint information for each candidate entity; Combine the prompt information with the text hypothesis to form contextual information.

[0013] Furthermore, the processing process of the large language model is: The large language model uses the self-attention mechanism and multi-layer Transformer structure to deeply understand the relationship between the prompt information in the context information and the text hypothesis; infer the incorrect part of the text hypothesis and generate the corrected text; use greedy decoding to generate the most likely output sequence to obtain semantic retrieval results.

[0014] The present invention also provides a law enforcement recorder audio semantic retrieval method based on retrieval enhancement, and the adopted technical solution is as follows: comprising the following steps: Obtain audio data and text queries recorded by law enforcement cameras; Project the audio data into the text embedding space to obtain speech representation; Perform cross-modal retrieval on speech representations and text queries to obtain speech tokens with text embedding and speech embedding in the same embedding space; According to the speech tokens, obtain the text hypothesis corresponding to the speech tokens; Extract query fragments that may contain entity names from textual hypotheses; According to the query fragment that may contain the entity name, the entity database is searched to obtain the relevant entity names; Construct contextual information based on relevant entity names and textual assumptions; According to the context information, semantic retrieval results are obtained.

[0015] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects: 1. The present invention completely bypasses the ASR system by directly processing the speech data, thus avoiding the problem of ASR error propagation. Especially in high-noise or low-quality audio scenarios, SLM can generate more accurate text representations, significantly improving the robustness and reliability of the system.

[0016] 2. The present invention realizes the alignment of text and speech in the same embedding space through a cross-modal retriever, so that the system can directly retrieve relevant information from speech data, reducing the dependence on ASR transcription, which not only improves the efficiency of the system but also enhances its applicability in complex scenarios.

[0017] 3. This paper introduces an external knowledge base and retrieval-augmented generation (RAG) technology to effectively correct the ASR system's recognition errors of uncommon entity names when processing law enforcement camera audio. Experimental results show that this method significantly reduces the word error rate (WER) in entity name recognition tasks, especially when processing low-frequency entities, the recognition accuracy is improved by 33.3%.

[0018] 4. By combining the large language model (LLM) and context building technology, the present invention can better understand the semantic information in the law enforcement recorder audio, thereby improving the accuracy of semantic retrieval. The present invention can not only identify the entity name, but also infer more key information based on the context information, such as event description, time and place, etc., to provide more comprehensive support for law enforcement analysis.

[0019] 5. The present invention adopts a two-step context generation process, first filtering the retrieved relevant entity names and then constructing the prompt information, which effectively reduces the size of the context input, thereby reducing the computational and memory overhead. In addition, the use of LoRA adapters to adjust the large language model can reduce the number of task-specific parameters while maintaining the model performance, further reducing the consumption of computing resources.

[0020] 6. The entity retrieval method of the present invention supports multiple embedding technologies, including acoustic neighbor embedding, semantic embedding and Okapi BM25, and can flexibly respond to different data characteristics and retrieval requirements.

[0021] 7. By automatically processing key information in the audio of the law enforcement recorder, the present invention can significantly reduce the workload of relevant personnel in manually transcribing and analyzing audio data, thereby improving law enforcement efficiency.

[0022] 8. The design of the present invention is highly scalable and can flexibly integrate new speech encoders, text retrievers and language models to adapt to ever-changing technical requirements and application scenarios, and can better support multi-language and multi-domain speech data processing tasks.

[0023] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 It is a structural schematic diagram of the system provided by the present invention.

[0026] Figure 2 It is a schematic diagram of the pre-training of the speech adapter module provided by the present invention.

[0027] Figure 3 This is an architectural diagram of the cross-modal retriever provided by the present invention.

[0028] Figure 4 It is a schematic diagram of the creation process of the retrieval database provided by the present invention.

[0029] Figure 5 It is a flow chart of the method provided by the present invention.

[0030] Reference numerals: 1. Data acquisition module; 2. Speech adapter module; 3. Cross-modal retriever; 4. Speech language model; 5. Query generation module; 6. Entity retrieval module; 7. Context construction module; 8. Large language model. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0032] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0033] Combine the following Figures 1 to 5 The present invention is further described in detail, and a method and system for audio semantic retrieval of law enforcement recorders based on retrieval enhancement are described as follows: In this embodiment, Figure 1 As shown, a law enforcement recorder audio semantic retrieval system based on retrieval enhancement is provided, including: a data acquisition module 1, a speech adapter module 2, a cross-modal retriever 3, a speech language model 4, a query generation module 5, an entity retrieval module 6, a context construction module 7 and a large language model 8.

[0034] The data acquisition module 1 is used to acquire the audio data and text query recorded by the law enforcement recorder, and input the audio data into the voice adapter module and input the text query into the cross-modal retriever.

[0035] The audio data here is the original audio data recorded by the law enforcement recorder and has not been processed in any way.

[0036] The text query is what the user wants to query. For example, if you want to query the entity name of a place in the audio data, the text query can be described as: "Where did Li go?" The speech adapter module 2 is used to project the audio data into the text embedding space to obtain the speech representation.

[0037] The audio data is input into the speech adapter module to calculate the speech representation projected into the text embedding space, which is used as an input to the cross-modal retriever.

[0038] The core function of the speech adapter module is to adapt the speech representation of the input audio data to the text embedding space of the downstream text model, thereby achieving cross-modal alignment between speech and text. The speech adapter module consists of two parts: a speech encoder and a downsampler, and is trained end-to-end through the cosine similarity loss propagated from the text model to ensure the consistency of the speech representation and the text embedding space, such as Figure 2 shown.

[0039] (1) The speech encoder is the front-end component of the speech adapter module, responsible for converting the input audio data into a high-dimensional speech representation. In the specific implementation, it is pre-trained on cross-dialect speech data through self-supervised learning. The key embedding extracted by the speech encoder not only contains common dialect features, such as phoneme distribution, but also retains dialect specificity, such as tone patterns, and supports robust alignment of multiple dialects. The last layer of hidden states of the speech encoder is extracted as the input of the downsampler. These hidden states not only contain rich speech information, such as phonemes, intonation, rhythm, etc., but also capture the contextual semantic information of the speech through self-supervised learning, providing high-quality input for subsequent cross-modal adaptation.

[0040] (2) The main function of the downsampler is to project the high-dimensional speech representation generated by the speech encoder into the text embedding space, thereby achieving alignment between speech and text. Since the sequence length of speech representation is usually much longer than that of text representation, for example, the speech frame rate is usually 100 frames / second, while the token length of text is much shorter, the sequence length needs to be adapted. The downsampler achieves this goal through the following two steps.

[0041] Average pooling layer in the time dimension: The downsampler downsamples the speech representation through an average pooling layer in the time dimension. The average pooling layer compresses the sequence length of the speech representation to a length similar to that of the text representation while retaining the key features of the speech. The average pooling operation can effectively reduce redundant information while avoiding information loss caused by direct truncation or random sampling.

[0042] Projection layer: After downsampling, the downsampler maps the dimensions of the speech representation to the same embedding dimensions as the text through a projection layer. The projection layer is usually implemented by a fully connected neural network, whose weights are optimized through end-to-end training to ensure the alignment of the speech representation with the text embedding space. The output of the projection layer is the adapted speech representation, which can be directly used for downstream tasks such as speech-to-text retrieval, speech translation, etc.

[0043] Throughout the training process, the downsampler is optimized using the cosine similarity loss. This loss function calculates the cosine similarity between speech embeddings and corresponding text embeddings, guiding the speech representation to move closer to the text embedding space, thereby achieving cross-modal alignment. Through end-to-end training, the downsampler can learn the complex mapping relationship between speech and text, providing high-quality speech representation for multimodal tasks.

[0044] The advantages of the voice adapter module are as follows: Advantages of self-supervised learning: The speech encoder uses self-supervised learning pre-training, which can learn robust speech feature representation without relying on a large amount of labeled data. This representation not only contains low-level acoustic features, but also captures high-level semantic information, providing a solid foundation for cross-modal adaptation.

[0045] Flexibility of temporal pooling: The average pooling layer in the temporal dimension can adjust the size and step size of the pooling window according to the task requirements, thereby flexibly controlling the degree of downsampling. For example, in tasks that need to retain more details, a smaller pooling window can be used; while in tasks that need to compress the sequence length, a larger pooling window can be used.

[0046] Nonlinear capabilities of the projection layer: The projection layer usually contains nonlinear activation functions, such as ReLU, which can enhance the expressive power of the model and better capture the complex relationship between speech and text.

[0047] The cross-modal retriever 3 is used to perform cross-modal retrieval on speech representations and text queries to obtain speech tokens of text embedding and speech embedding in the same embedding space.

[0048] The speech representation and text query are input into the cross-modal retriever, which realizes cross-modal retrieval by mapping the text query and speech representation into the same embedding space, and obtains speech tokens of text embedding and speech embedding in the same embedding space.

[0049] The core goal of the cross-modal retriever is to achieve efficient retrieval and alignment between text and speech. The architecture is as follows: Figure 3 As shown in the figure, the key encoder, value encoder and retrieval enhancement decoder work together to significantly improve the cross-modal retrieval performance through multimodal alignment.

[0050] (1) The retrieval-enhanced decoder is the core reasoning component that receives the prompt template, speech representation, and text query, and retrieves the relevant context based on the value vector. The prompt template is the preset content, and the content of the prompt template input each time is the same. The retrieval-enhanced decoder integrates a cross-attention mechanism, allowing dynamic attention to the retrieved multimodal context when generating each text token.

[0051] The retrieval-enhanced decoder is built based on cross-attention and causal multi-head attention. The prompt template, speech representation, text query and value vector are processed by cross-attention and causal multi-head attention, and then passed through the language model head to obtain speech tokens.

[0052] The role of Cross-Attention is to integrate the retrieved value vector into the generation process. The Cross-Attention layer uses ordinary multi-head attention (without causal mask) because the retrieved context is complete and there is no need to mask future information. The Query comes from the hidden state of the retrieval enhancement decoder, and the Key / Value comes from the output of the value encoder.

[0053] The role of causal multi-head attention is to ensure that when autoregressively generating text, the output of each position depends only on the input before the current position. In the causal multi-head attention layer, the lower triangular mask matrix is ​​used to mask the attention weights of future positions.

[0054] The language model head is connected after the last hidden state of the retrieval enhancement decoder. Its structure is a linear layer + Softmax, which maps the hidden state to the vocabulary probability distribution information. According to the above information, the corresponding speech paragraph is found, and the above information and the corresponding speech paragraph form speech tokens.

[0055] (2) The key encoder is built on causal multi-head attention and shares the same multimodal language model architecture as the retrieval-enhanced decoder. It is responsible for encoding the input sequence into a retrieval key. The key corresponds to the position of the text token in the retrieval database. Its encoding process integrates speech and text modality information to ensure that the retrieval key contains cross-modal semantic features.

[0056] Among them, the role of causal multi-head attention is to process sequences (the sequence obtained by the prompt template, voice representation and text query through the embedding layer), and strict time dependency must be maintained when generating key embeddings. When the input is a sequence of voice frames or a sequence of text, the causal mask is used to ensure that the key embedding at each position only depends on the current and previous inputs. For example, when processing audio, the key embedding of the tth frame is calculated only based on the previous t frames to avoid information leakage. Its advantage is to generate time-aligned key embeddings and improve retrieval accuracy, especially in speech recognition, where the order of phonemes is crucial.

[0057] The search key is used to search in the search database, and the search value (the context window of the key tokens around the search key in the search database) is obtained, and then input into the value encoder.

[0058] (3) The value encoder uses a lightweight multi-level Transformer architecture to encode the search value. Each search value contains a key tag, and after encoding, it generates a fixed-dimensional value vector through pooling.

[0059] Among them, the role of ordinary multi-head attention is to process the retrieved context window, such as a text paragraph or a voice clip, with complete context and no need to block future information. By inputting a fixed-length context window, such as a token sequence of [t−7, t+8], the global dependency is captured through multi-head attention. The output is a value embedding of the same dimension as the key embedding, which is used to retrieve the cross attention of the enhanced decoder. Complex relationships within the context, such as long-distance dependencies and semantic cohesion, can be fully modeled.

[0060] (4) The process of creating a search database is as follows: Figure 4 As shown in the figure. Key generation: The hidden state corresponding to each text tag in the retrieval database is used as the key, and the associated speech context, such as the complete audio paragraph, is integrated during encoding to ensure that the key contains acoustic-semantic joint features. Value extraction: With the key tag as the center, the multimodal tag sequence (including speech and text tags) in the front and back windows is extracted and stored in the retrieval database after being processed by the value encoder. Figure 4 In the figure, p0 represents the prompt template, x1 and x2 represent the voice representation, y0, y1, y2, y3 represent the text query, and k0, k1, k2, k3 represent the search key.

[0061] (5) The working process of the cross-modal retriever is: The prompt template, the voice representation and the text query are input into a key encoder to obtain a search key, wherein the key encoder adopts a multi-level Transformer architecture based on causal multi-head attention; the prompt template is a preset content, and the content of the prompt template input each time is the same; Using the search key to search in the search database, obtain the search value; The retrieval value is input into a value encoder to obtain a value vector, wherein the value encoder adopts a lightweight multi-level Transformer architecture; The prompt template, voice representation, text query and value vector are input into the retrieval enhancement decoder to obtain voice Tokens. The retrieval enhancement decoder adopts a multi-level Transformer architecture based on cross attention and causal multi-head attention, and the language model head is connected after the multi-level Transformer architecture.

[0062] The Transformer architecture of the key encoder, value encoder and retrieval enhancement decoder, in this embodiment, all use a 15-layer architecture. Specifically, in the key encoder, each layer of the architecture includes a causal multi-head attention layer, an addition layer, a layer normalization, a feedforward layer, an addition layer and a layer normalization connected in sequence, and the 15-layer architecture is connected before the embedding layer. In the value encoder, each layer of the architecture includes a multi-head attention layer, an addition layer, a layer normalization, a feedforward layer, an addition layer and a layer normalization connected in sequence. The 15-layer architecture is connected before the embedding layer and then the pooling layer. In the retrieval enhancement decoder, each layer of the architecture includes a layer normalization, a cross attention layer, an addition layer, a causal multi-head attention layer, an addition layer, a layer normalization, a feedforward layer, an addition layer and a layer normalization connected in sequence. The 15-layer architecture is connected before the embedding layer and then the language model head.

[0063] The advantages of cross-modal retrievers are as follows: Causal attention ensures generation consistency: The key encoder and decoder use causal multi-head attention to strictly maintain temporal dependencies, avoid future information leakage, and improve the logical coherence of generated text.

[0064] Flexibility of retrieval enhancement: The value encoder uses ordinary multi-head attention to fully model the global semantics of the retrieval context and provide rich background knowledge for the retrieval enhancement decoder.

[0065] Efficient multimodal alignment: The key encoder (causal) and value encoder (non-causal) have clear division of labor. The former captures temporal features, while the latter extracts contextual semantics, jointly optimizing cross-modal alignment.

[0066] Key-value collaborative encoding: The key encoder and the retrieval-enhanced decoder share model parameters to achieve end-to-end optimization of the encoding-retrieval-decoding process and reduce computational redundancy.

[0067] Multimodal context-aware: The retrieval value contains a mixed sequence of speech and text, and the lightweight design of the value encoder effectively captures cross-modal dependencies while reducing computational overhead.

[0068] The speech language model 4 is used to obtain text hypotheses corresponding to the speech tokens according to the speech tokens.

[0069] The Speech Language Model (SLM) is one of the core components of this system. Its main function is to generate high-quality text representations based on the input speech tokens. The text representation is a text hypothesis corresponding to the speech tokens. Unlike traditional generation methods based on automatic speech recognition (ASR), SLM directly processes speech data and completely bypasses the ASR system, avoiding the common error propagation problem in the ASR system, thereby significantly improving the accuracy and reliability of the generated text.

[0070] The process of SLM generating text representation is as follows: SLM extracts semantic information of audio segments through speech encoders and speech adapters. The speech encoder converts audio data into high-dimensional speech representations, while the speech adapter projects these representations into the text embedding space to align them with the text modality. SLM fuses the semantics of the query with the content of the audio segment to construct a unified contextual representation. This representation not only contains the detailed information of the audio segment, but also reflects the intent and requirements of the query. Through multi-level attention mechanisms (such as the self-attention mechanism in Transformer), SLM can capture the complex relationship between the audio segment and the query, thereby generating a more accurate text representation. Based on the contextual information, SLM generates text representations related to the query. The generation process uses an autoregressive generation model to generate coherent text by gradually predicting the next word.

[0071] Since SLM processes speech data directly, it avoids the error propagation problem in the speech-to-text conversion process in traditional ASR systems. Traditional ASR systems may introduce errors due to factors such as speech quality, accent, or background noise, which further affect the performance of downstream tasks. SLM generates text directly from speech in an end-to-end manner, reducing errors in the intermediate links, thereby improving the accuracy and robustness of the generated text. In addition, SLM can further improve the quality of the generated text by introducing an external knowledge base or post-processing module (semantic correction).

[0072] The output of SLM is the final text representation (text hypothesis), which can be directly used in subsequent query generation modules and decision support. The text representation has the following characteristics: High accuracy: The generated text representation is more accurate due to avoiding ASR error propagation. Semantic richness: The text generated based on contextual information not only contains factual content, but also reflects the semantic intent of the query. Diversity: Depending on the task requirements, SLM can generate text representations in multiple forms, such as answers, summaries, or explanations.

[0073] The query generation module 5 is used to extract query fragments that may contain entity names from the text hypothesis.

[0074] The main function of the query generation module is to extract query fragments that may contain entity names based on the text hypothesis. The query generation module analyzes the semantic information in the text hypothesis, identifies and extracts key entities, and thus generates structured query fragments. The text hypothesis is usually input in the form of natural language text, which may contain incomplete sentences, grammar errors, or ambiguous expressions. Query fragments that may contain entity names are phrases or keywords extracted from the text hypothesis that may contain entity names. These phrases or keywords are the core content for subsequent retrieval or processing. It can be a single entity name, such as place name A, or a phrase containing an entity name, such as tourism in place name A. Query fragments have high accuracy and semantic relevance, can accurately reflect the user's intention, and support downstream tasks such as information retrieval.

[0075] The processing process of the query generation module is as follows: (1)Preprocessing of the text hypothesis. Text cleaning: Clean the text hypothesis, remove irrelevant characters, punctuation marks, and stop words, such as "of", "is", etc., and retain the key semantic content. Word segmentation and part-of-speech tagging: Use a word segmentation tool to segment the text hypothesis and perform part-of-speech tagging on each word, such as noun, verb, adjective, etc., for subsequent entity recognition.

[0076] (2)Named Entity Recognition (NER). Use a pre-trained Named Entity Recognition (NER) model to perform entity tagging on the text hypothesis after part-of-speech tagging. The NER model can identify entity categories in the text, such as person names, place names, organization names, time, etc. The NER model tags each word in the text hypothesis, identifies the entity names and their categories therein. According to the tagging results of the NER model, extract the fragments that may contain entity names.

[0077] (3)Query fragment generation. Combine the extracted fragments that may contain entity names with relevant context words to generate query fragments. Filter the generated query fragments to remove redundant or irrelevant query fragments. Sort the query fragments according to their importance or relevance to ensure that the most important or relevant fragments are output first, and output query fragments that may contain entity names.

[0078] The advantages of the query generation module are as follows: Domain adaptation: For text hypotheses in a specific domain, fine-tune the NER model to improve its performance in the specific domain.

[0079] Multi-task learning: When training the NER model, optimize other related tasks at the same time, such as part-of-speech tagging, semantic role tagging, so as to improve the generalization ability of the model.

[0080] Data augmentation: Through data augmentation techniques, such as synonym replacement, noise addition, generate more training data, thereby improving the robustness of the model.

[0081] Diversity of query fragments: Single-entity fragments: fragments containing only a single entity name. Multi-entity fragments: fragments containing multiple entity names. Context-enhanced fragments: fragments containing entity names and their context.

[0082] Error correction and post-processing: Syntax check: Perform syntax check on the generated query fragments to ensure that they comply with language specifications. Semantic correction: Perform semantic correction on the query fragments through external knowledge bases or context information to ensure that they are consistent with the user's intentions. Figure 1 To.

[0083] Through the above design, the query generation module can efficiently extract key entities from text hypotheses and generate high-quality query fragments, providing strong support for downstream tasks.

[0084] The entity retrieval module 6 is used to retrieve the entity database according to the query fragment that may contain the entity name, and obtain the relevant entity name.

[0085] The main function of the entity retrieval module is to retrieve entity information related to the query fragment from the entity database (a large-scale knowledge base). The entity retrieval module combines acoustic neighbor embedding, semantic embedding and traditional text retrieval methods to perform retrieval and realize multi-level similarity calculation, thereby improving the accuracy and robustness of retrieval.

[0086] The implementation process of the entity retrieval module is as follows: (1) Acoustic Neighbor Embeddings Acoustic neighbor embedding is used to capture the acoustic similarity of text fragments, and is particularly suitable for processing speech-related query fragments. The specific implementation is as follows: the embedding vector is generated through the acoustic, speech, and spelling encoders. The acoustic encoder extracts the acoustic features of the speech signal, the speech encoder converts the speech into a high-dimensional speech representation, and the spelling encoder maps the text fragments to the spelling space. The generated embedding vector is used to calculate the acoustic similarity between text fragments. The acoustic similarity of the query fragment and the entity name in the knowledge base is evaluated through metrics such as cosine similarity or Euclidean distance. Acoustic neighbor embedding is particularly suitable for processing entity names that have similar pronunciations but different spellings, thereby improving the recall rate of retrieval.

[0087] (2) Semantic Embeddings Semantic embedding is used to capture the semantic similarity of text fragments and ensure that the retrieval results are consistent with the semantic meaning of the query fragment. Figure 1The specific implementation is as follows: Use the pre-trained T5 model (Text-To-Text Transfer Transformer) to generate semantic embedding vectors. The T5 model generates high-dimensional semantic representations by inputting text fragments into the encoder. The generated semantic embedding vectors are used to calculate the semantic similarity between text fragments. The semantic similarity between the query fragment and the entity name in the knowledge base is evaluated through metrics such as cosine similarity or dot product. Semantic embedding is particularly suitable for processing synonyms, near-synonyms, or semantically related entity names, thereby improving the accuracy of retrieval.

[0088] (3) Traditional text retrieval methods This embodiment uses Okapi BM25 to generate query and key vectors and calculate the relevance score between them. The specific implementation is as follows: Okapi BM25 calculates the relevance of query fragments and documents based on term frequency (TF) and inverse document frequency (IDF). Term frequency measures the frequency of occurrence of query terms in documents, and inverse document frequency measures the importance of query terms in the entire document collection.

[0089] Okapi BM25 calculates the relevance score between the query fragment and the document using the following formula : in, It's a document. is the query fragment, is the i-th query word, yes The inverse document frequency of yes exist The word frequency in is the length of the document, is the average length of the document collection, is a parameter to adjust the word frequency saturation. It is a parameter that adjusts the degree of document length normalization.

[0090] Okapi BM25 is particularly suitable for processing short text query fragments (such as keywords or phrases) and can quickly retrieve documents with high relevance.

[0091] The entity retrieval module adopts a multimodal fusion strategy: fusing the results of acoustic neighbor embedding, semantic embedding, and Okapi BM25. The fusion method can be: integrating the scores of the three methods through weighted summation or machine learning models (such as ranking models) to generate relevant entity names (retrieval results).

[0092] Sorting and filtering search results: After generating search results, you can sort and filter the results using the following strategies: (1) Score-based ranking: The retrieval results are ranked according to the scores of acoustic neighbor embedding, semantic embedding, and Okapi BM25.

[0093] (2) Context-based filtering: Filter the search results based on the context information of the query fragment (such as time and location) to ensure that they are consistent with the user's intention. Figure 1 To.

[0094] (3) Optimization based on user feedback: Optimize search results through user feedback (such as click-through rate, satisfaction score), thereby improving the user experience of the system.

[0095] Through the above design, the entity retrieval module can efficiently retrieve entity information related to the query fragment from the knowledge base, providing strong support for downstream tasks.

[0096] The context building module 7 is used to build context information according to the relevant entity names and text assumptions.

[0097] The core task of the context building module is to combine the retrieved relevant entity names with the text hypothesis to form context information suitable for processing. This process aims to provide rich background information so that it can better understand the semantic content in the text hypothesis and correct possible errors, especially entity name errors.

[0098] The process of context building module is: (1) Entity filtering Entity filtering is the first step in context construction, and its goal is to filter out the entities most relevant to the text hypothesis from the relevant entity names, and filter out irrelevant entities based on the predefined Euclidean distance threshold and count threshold. The query content is obtained based on the text hypothesis.

[0099] Euclidean distance threshold: Evaluate the similarity between an entity and a query by calculating the Euclidean distance between the query vector and the entity vector. The smaller the Euclidean distance, the more similar the entity is to the query. Set a predefined threshold, such as 1.0, and only keep entities whose distance is less than the threshold. For example, if the distance between the query and entity A is , the distance from entity B is , then entity A is retained and entity B is filtered out.

[0100] Count threshold: To control the number of entities returned for each query, set a count threshold. Even if there are multiple entities whose distances meet the Euclidean distance threshold, only the first N entities with the smallest distances are retained. For example, if it is set to 3, each query will return at most 3 entities.

[0101] After the above filtering steps, the candidate entities that are closest to the text hypothesis in acoustics or semantics are obtained. These entities are not only highly relevant to the query, but also can provide high-quality information for subsequent context construction.

[0102] (2) Prompt construction Construct a prompt for each candidate entity. The format of the prompt can be adjusted according to the task requirements. For example: Simple hint: only include candidate entities. Example: [H] Place B.

[0103] Complex hint: contains candidate entities and their queries. For example: [H] Query: Place A → Entity: Place B. This embodiment uses the format of complex hint.

[0104] The purpose of the hint information is to provide additional context to the subsequent large language model (LLM) to help it understand which entities may be related to the incorrect part of the text hypothesis. Complex hints contain not only the retrieved candidate entity, but also the query information that generated the candidate entity. This additional contextual information can help the LLM better understand the relationship between the entity and the query. Not only does it tell the LLM that "Place B" is a possible entity, but it also explicitly points out that this entity was retrieved from the query "Place A". This information can help the LLM infer the incorrect part of the text hypothesis and generate the correct correction result.

[0105] (3) Contextual information construction Combine the prompt information with the text hypothesis to form a complete contextual information.

[0106] This example uses tags to separate prompts and textual assumptions. For example: [H] Location B [H] Location C [A] Travel to location A [P] Among them: [H] represents prompt information; [A] represents text hypothesis; [P] represents the prompt for LLM to start prediction.

[0107] (4) Context optimization: To reduce computational and memory overhead, the length of context information is usually limited to a certain range. The optimal prompt format and context length are determined through experiments to minimize resource consumption while maintaining high accuracy.

[0108] The large language model 8 is used to obtain semantic retrieval results based on context information.

[0109] The core task of the Large Language Model (LLM) is to generate corrected text output based on context information. LLM infers the most likely correct text and corrects the errors by deeply understanding the prompt information and text assumptions in the context information. Large language models are particularly important in law enforcement camera scenarios, and can significantly improve the accuracy and semantic consistency of text generation.

[0110] The implementation process of the large language model is: (1) Model selection and adjustment: In order to adapt to the specific task requirements of the law enforcement recorder scenario, the LLM application module uses a pre-trained large language model as the basic model and adjusts it through the following methods: LoRA Adapter: Add Low-Rank Adaptation (LoRA) on the basis of LLM. LoRA enables the model to quickly adapt to specific tasks without significantly increasing the computational overhead by introducing a small number of trainable parameters. In the specific implementation, the LoRA adapter adjusts some weights of the model through low-rank matrix decomposition, thereby significantly reducing the computational cost of training and inference while maintaining the high performance of the model.

[0111] Task fine-tuning: Fine-tune the LLM on relevant datasets of law enforcement camera scenarios to enable it to better understand the terms, entity names, and contextual information in the field of law enforcement cameras. For example, conduct targeted training and optimization for entities such as place names and personal names in video surveillance systems.

[0112] (2) Contextual input processing LLM uses the self-attention mechanism and multi-layer Transformer structure to deeply understand the relationship between prompt information and textual hypotheses. The self-attention mechanism can capture long-distance dependencies in the context, while the multi-layer Transformer structure extracts higher-level semantic information through layer-by-layer abstraction.

[0113] (3) Error Correction and Text Generation LLM infers the wrong part of the text hypothesis based on the context information and generates the corrected text. The specific process is as follows: Error detection: LLM identifies the incorrect part of the text hypothesis by comparing the prompt information and the text hypothesis in the context information. For example, in the prompt information "Place B" and the text hypothesis "Travel to Place A", LLM can identify that "Place A" is an incorrect entity. Of course, if Place A is a correct entity, then the entity retrieval module can retrieve Place A from the entity database and use it as a related entity name.

[0114] Text generation: LLM generates corrected text based on contextual information. For example, supported by a multi-layer Transformer structure, by analyzing the current input information, past behavior data and preference trends are also referenced. If the user shows a preference for modern urban landscapes or European-style buildings in other queries, LLM tends to infer that he or she may be interested in location B. In contrast, the traditional and historical content of location C may not meet the current user needs and is therefore automatically excluded. Therefore, "travel to location A" is corrected to "travel to location B".

[0115] Decoding strategy: LLM usually uses greedy decoding to generate the most likely output sequence. Greedy decoding selects the word with the highest probability at each step, thereby generating coherent and semantically accurate text and obtaining semantic retrieval results.

[0116] (4) Output results LLM outputs semantic retrieval results for subsequent semantic retrieval or other tasks.

[0117] (5) Performance evaluation: In order to evaluate the corrective effect of LLM, the following indicators and methods were used: Word Error Rate (WER): WER is the core indicator for measuring the effect of text correction. The calculation formula is: .

[0118] Experiments show that the Retrieval-Augmented Generation (RAG)-based method can significantly reduce WER (relative reduction of 39.9%) when correcting rare entity name errors, without causing performance degradation on queries in other fields.

[0119] Domain adaptability evaluation: By comparing the performance of LLM in the law enforcement camera scenario with other scenarios (such as medical and financial), its domain adaptability is evaluated. Experimental results show that the LLM adjusted by the LoRA adapter performs significantly better in the law enforcement camera scenario than the unadjusted model.

[0120] User satisfaction evaluation: The practicality and accuracy of LLM-generated texts are evaluated through user feedback (e.g., click-through rate, satisfaction score).

[0121] This embodiment verifies the performance of the system on a variety of dialect speech data sets, focusing on the reduction of word error rate (WER) and the improvement of recognition accuracy, to ensure that the system can effectively improve the accuracy and efficiency of audio semantic retrieval of law enforcement recorders.

[0122] 1. Validation Dataset The following dialect speech datasets were used in this verification: 1. Sichuan dialect natural dialogue voice data collected by mobile phone Dataset introduction: 1,730 Sichuan natives participated in the recording, and they communicated face to face in a natural way. They were free to express themselves without restrictions on topics. The fields were wide, and the voices were natural and fluent, which was in line with actual conversation scenarios. The accuracy of manual transcription was high. Data volume: 799 hours. Data format: 16kHz, uncompressed 16bit, wav, mono. Personnel composition: There were 1,730 recorders in total, of which 74% were women and 88% were under 25 years old. All the recorders were from Sichuan or Chongqing. Recording content: No topic was specified, and the recorders talked to each other in a natural way.

[0123] 2. Henan dialect natural dialogue voice data collected by mobile phone Dataset introduction: Recorded by 762 native Henan people, covering multiple age groups, with a balanced ratio of men to women. No topic is specified, no preset text, and the recorders talk freely in pairs. It can be used for speech recognition acoustics, voiceprint recognition model training or algorithm research. Data volume: 377 hours. Data format: 16kHz, uncompressed 16bit, wav, mono. Personnel composition: About 762 recorders, all of whom are from Henan area and speak Henan dialect. Recording content: No topic is specified, and the recorders talk freely in groups of two.

[0124] 3. Collecting voice data with mobile phones in Northeastern dialect Dataset introduction: 67 hours of Northeastern dialect mobile phone voice data collected by 312 Northeastern dialect speakers, most of whom are from the three northeastern provinces. All speakers read the text in Northeastern dialect, with rich content, covering customer consultations and text messages in nearly 30 fields. The sentences are manually transcribed and proofread by professional annotators with high accuracy. Data volume: 67 hours. Data format: 16kHz, 16bit, uncompressed wav, mono. Personnel composition: 312 Northeastern people; 218 women, accounting for 70%. Recording content: Text read in Northeastern dialect, covering customer consultations and text messages in nearly 30 fields.

[0125] 4. Suzhou dialect mobile phone voice collection voice data Dataset introduction: 176 hours of Suzhou dialect mobile phone voice data collected by 250 speakers, each of whom recorded about 500 Suzhou dialect colloquial sentences. Local Suzhou speakers were selected to participate in the recording, with authentic accents, colloquial and life-like content, and natural and fluent reading. Suzhou locals participated in quality inspection and proofreading, and the text transcription was more accurate. Data volume: 176 hours. Data format: 16kHz, 16bit, uncompressed wav, mono. Personnel composition: 250 Suzhou people; 129 women, accounting for 52%. Recording content: Suzhou dialect colloquial sentences, with colloquial and life-like content.

[0126] 5. Collecting Voice Data with Kunming Dialect Mobile Phone Dataset introduction: 1002 hours of Kunming dialect mobile phone voice data recorded by 2284 native Kunming speakers with authentic accents covering multiple age groups. The recorded texts cover multiple categories such as general, interactive, car-mounted, and home, with rich content. Kunming locals participated in quality inspection and proofreading, and the text transcription is more accurate. Data volume: 1002 hours. Data format: 16kHz, 16bit, uncompressed wav, mono. Personnel composition: 2284 people; 60% of them are women; the recorders are mainly aged 16-25, accounting for 80%; the recorders are all locals in Kunming and surrounding areas. Recording content: general, interactive, home command, digital, and dialect terms.

[0127] 6. Changsha Dialect Mobile Phone Voice Data Collection Dataset introduction: 997 hours of Changsha dialect mobile phone voice data collected by more than 2,000 Changsha dialect native speakers, covering multiple age groups, with a uniform ratio of men and women and authentic accents. The recorded texts cover multiple categories such as general, interactive, in-vehicle, and home, with rich content. Changsha locals participated in quality inspection and proofreading, with a sentence accuracy rate of 95%. Data volume: 997 hours. Data format: 16kHz, 16bit, uncompressed wav, mono. Personnel composition: 2,301 people in total, 1,030 males and 1,271 females. Recording content: multiple categories of texts such as general, interactive, in-vehicle, and home.

[0128] 7. Collecting Voice Data from Chinese Mobile Phones Dataset introduction: 200 Chinese mobile phone voice data collected by 200 native Chinese speakers, covering major dialect areas. The recordings cover quiet and noisy environments, which are more suitable for actual application scenarios of speech recognition. The recordings are all commonly used colloquial sentences, and the reading is natural and fluent. Manual transcription, high accuracy. Data volume: 12 hours. Data format: 16kHz, 16bit, uncompressed wav, mono. Personnel composition: 200 Chinese, of which women account for 53%. Recording content: commonly used colloquial sentences.

[0129] 2. Verification Method 1. Baseline model comparison Using the traditional ASR speech recognition system as the baseline model, speech recognition tests were performed on the above dialect datasets, and the word error rate (WER) and recognition accuracy were recorded.

[0130] 2. Application of the Retrieval Enhancement System (this system) The retrieval enhancement system is applied to the same dialect dataset to preprocess and semantically associate the speech data before performing speech recognition.

[0131] 3. Performance index comparison Compare the word error rate (WER) and recognition accuracy of the baseline model and the retrieval enhancement system, and calculate the improvement ratio.

[0132] 3. Verification Results 1.1 Baseline model testing 1. Sichuan Dialect Dataset Baseline model word error rate (WER): 25%; Baseline model recognition accuracy: 75%.

[0133] 2. Henan Dialect Dataset Baseline model word error rate (WER): 26%; Baseline model recognition accuracy: 74%.

[0134] 3. Northeastern Dialect Dataset Baseline model word error rate (WER): 24%; Baseline model recognition accuracy: 76%.

[0135] 4. Suzhou Dialect Dataset Baseline model word error rate (WER): 27%; Baseline model recognition accuracy: 73%.

[0136] 5Kunming Dialect Dataset Baseline model word error rate (WER): 25%; average baseline model recognition accuracy: 75%.

[0137] 6. Changsha Dialect Dataset Baseline model word error rate (WER): 26%; average baseline model recognition accuracy: 74%.

[0138] 7. Chinese mobile phone voice data set Baseline model word error rate (WER): 28%; average baseline model recognition accuracy: 72%.

[0139] 2. Application of search enhancement system 1. Sichuan Dialect Dataset Word error rate (WER) after applying the retrieval enhancement system: 16%; recognition accuracy after applying the retrieval enhancement system: 84%.

[0140] 2. Henan Dialect Dataset Word error rate (WER) after applying the retrieval enhancement system: 17%; recognition accuracy after applying the retrieval enhancement system: 83%.

[0141] 3. Northeastern Dialect Dataset Word error rate (WER) after applying the retrieval enhancement system: 16%; recognition accuracy after applying the retrieval enhancement system: 84%.

[0142] 4. Suzhou Dialect Dataset Word error rate (WER) after applying the retrieval enhancement system: 15%; recognition accuracy after applying the retrieval enhancement system: 83%.

[0143] 5. Kunming Dialect Dataset Word error rate (WER) after applying the retrieval enhancement system: 16%; recognition accuracy after applying the retrieval enhancement system: 84%.

[0144] 6. Changsha Dialect Dataset Word Error Rate (WER) after applying the retrieval enhancement system: 12% Recognition Accuracy after applying the retrieval enhancement system: 88% 7. Chinese mobile phone voice data set Word error rate (WER) after applying the retrieval enhancement system: 18%; recognition accuracy after applying the retrieval enhancement system: 82%.

[0145] 3. Reduction of Word Error Rate (WER) Compared with the baseline model, the word error rate of this system is reduced and the reduction ratio is: 1. Sichuan Dialect Dataset The word error rate decreased: 25%-16%=9%; the reduction ratio: 9% / 25%×100%=36%.

[0146] 2. Henan Dialect Dataset The word error rate decreased: 26%-17%=9%; the reduction ratio: 9% / 26%×100%=34.6%.

[0147] 3. Northeastern Dialect Dataset The word error rate decreased: 24%-16%=8%; the reduction ratio: 8% / 24%×100%=33.3%.

[0148] 4. Suzhou Dialect Dataset The word error rate decreased: 27%-15%=12%; the reduction ratio: 12% / 27%×100%=44.4%.

[0149] 5. Kunming Dialect Dataset The word error rate decreased: 25%-16%=9%; the reduction ratio: 9% / 25%×100%=36%.

[0150] 6. Changsha Dialect Dataset The word error rate decreased: 26%-12%=14%; the reduction ratio: 14% / 26%×100%=53.8%.

[0151] 7. Chinese mobile phone voice data set The word error rate decreased: 28%-18%=10%; the reduction ratio: 10% / 28%×100%=35.7%.

[0152] (IV) Improvement in recognition accuracy Compared with the baseline model, the improvement of recognition accuracy of this system and the improvement ratio are as follows: 1. Sichuan Dialect Dataset Improvement in recognition accuracy: 84%-75%=9%; improvement ratio: 9% / 75%×100%=12%.

[0153] 2. Henan Dialect Dataset Improvement in recognition accuracy: 83%-74%=9%; improvement ratio: 9% / 74%×100%=12.2%.

[0154] 3. Northeastern Dialect Dataset Improvement in recognition accuracy: 84%-76%=8%; improvement ratio: 8% / 76%×100%=10.5%.

[0155] 4. Suzhou Dialect Dataset Improvement in recognition accuracy: 83%-73%=10%; improvement ratio: 10% / 73%×100%=13.7%.

[0156] 5. Kunming Dialect Dataset Improvement in recognition accuracy: 84%-75%=9%; improvement ratio: 9% / 75%×100%=12%.

[0157] 6. Changsha Dialect Dataset Improvement in recognition accuracy: 88%-74%=14%; improvement ratio: 14% / 74%×100%=18.9%.

[0158] 7. Chinese mobile phone voice data set Improvement in recognition accuracy: 82%-72%=10%; improvement ratio: 10% / 72%×100%=13.9%.

[0159] IV. Conclusion Through this verification, it is proved that this system can significantly reduce the word error rate (WER) on various dialect speech data sets, with an average reduction of 39.9%, and the recognition accuracy rate increased by 13.4% on average. Among them, the word error rate of some dialect data sets has even been reduced by 44.4%, and the recognition accuracy rate has been improved by 18.9%. This shows that this system can effectively improve the accuracy and efficiency of audio semantic retrieval of law enforcement recorders, and has good application prospects.

[0160] In this embodiment, Figure 5 As shown, a method for audio semantic retrieval of law enforcement recorders based on retrieval enhancement is also provided, and the technical solution adopted is as follows: comprising the following steps: Step 1: Obtain audio data and text query recorded by the law enforcement recorder.

[0161] Step 2: Project the audio data into the text embedding space to obtain the speech representation.

[0162] First, the audio data is fed into a speech encoder to obtain a high-dimensional speech representation. The speech encoder is pre-trained on cross-dialect speech data through self-supervised learning. Then, the high-dimensional speech representation is fed into a downsampler and projected into the text embedding space to obtain the speech representation.

[0163] Step 3: Perform cross-modal retrieval on the speech representation and text query to obtain speech tokens of text embedding and speech embedding in the same embedding space.

[0164] The speech representation and text query are input into the cross-modal retriever, and the working process of the cross-modal retriever is as follows: The prompt template (own by the cross-modal retriever), the speech representation and the text query are input into the key encoder to obtain the retrieval key. The key encoder adopts a multi-layer Transformer architecture based on causal multi-head attention; Using the search key to search in the search database, obtain the search value; The retrieval value is input into a value encoder to obtain a value vector, wherein the value encoder adopts a multi-level Transformer architecture; The prompt template, voice representation, text query and value vector are input into the retrieval enhancement decoder to obtain voice Tokens. The retrieval enhancement decoder adopts a multi-level Transformer architecture based on cross attention and causal multi-head attention. The multi-level Transformer architecture is connected to the language model head.

[0165] The Transformer architecture of the key encoder, value encoder and retrieval enhancement decoder, in this embodiment, all use a 15-layer architecture. Specifically, in the key encoder, each layer of the architecture includes a causal multi-head attention layer, an addition layer, a layer normalization, a feedforward layer, an addition layer and a layer normalization connected in sequence, and the 15-layer architecture is connected before the embedding layer. In the value encoder, each layer of the architecture includes a multi-head attention layer, an addition layer, a layer normalization, a feedforward layer, an addition layer and a layer normalization connected in sequence. The 15-layer architecture is connected before the embedding layer and then the pooling layer. In the retrieval enhancement decoder, each layer of the architecture includes a layer normalization, a cross attention layer, an addition layer, a causal multi-head attention layer, an addition layer, a layer normalization, a feedforward layer, an addition layer and a layer normalization connected in sequence. The 15-layer architecture is connected before the embedding layer and then the language model head.

[0166] Step 4: Based on the speech tokens, obtain the text hypotheses corresponding to the speech tokens.

[0167] Input speech tokens into the speech language model to obtain text hypotheses.

[0168] Step 5: Extract query fragments that may contain entity names from text hypotheses.

[0169] The textual hypothesis is input into the query generation module to obtain query fragments that may contain entity names.

[0170] Step 6: According to the query fragment that may contain the entity name, search the entity database to obtain the relevant entity names.

[0171] The query fragment that may contain entity names is input into the entity retrieval module, which uses acoustic neighbor embedding, semantic embedding and traditional text retrieval methods to search the entity database, fuse the results of the three methods, and generate relevant entity names. The number of relevant entity names is 1 or more.

[0172] Step 7: Build context information based on relevant entity names and textual assumptions.

[0173] The relevant entity names and text hypotheses are input into the context building module, irrelevant entities are filtered out according to the predefined Euclidean distance threshold and count threshold, candidate entities are screened out from the relevant entity names, and then context information containing candidate entity and text hypothesis information is built.

[0174] Step 8: Obtain semantic retrieval results based on context information.

[0175] The context information is input into the large language model. The large language model infers the incorrect parts of the text hypothesis based on the context information, and generates corrected text with reference to past behavioral data and preference trends. That is, the incorrect entity name in the text hypothesis is corrected to the candidate entity that best matches the past behavioral data and preference trends, and then the semantic retrieval results are output.

[0176] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A law enforcement recorder audio semantic retrieval system based on retrieval enhancement, characterized in that: include: Data acquisition module, used to obtain audio data and text query recorded by law enforcement recorders; The speech adapter module is used to project the audio data into the text embedding space to obtain the speech representation; A cross-modal retriever is used to perform cross-modal retrieval on speech representations and text queries, obtaining speech tokens with text embeddings and speech embeddings in the same embedding space; The speech language model is used to obtain text hypotheses corresponding to speech tokens based on speech tokens; A query generation module, which is used to extract query fragments that may contain entity names from textual hypotheses; An entity retrieval module, used to retrieve an entity database and obtain relevant entity names according to a query fragment that may contain an entity name; A context building module, used to build context information based on relevant entity names and textual assumptions; Large language model, used to obtain semantic retrieval results based on contextual information.

2. The audio semantic retrieval system for law enforcement recorders based on retrieval enhancement as claimed in claim 1, characterized in that: The speech adapter module includes a speech encoder and a downsampler, A speech encoder, which is used to convert audio data into a high-dimensional speech representation. The speech encoder is pre-trained on cross-dialect speech data through self-supervised learning; The downsampler is used to project the high-dimensional speech representation into the text embedding space to obtain the speech representation.

3. The audio semantic retrieval system for law enforcement recorders based on retrieval enhancement as claimed in claim 1, characterized in that: The working process of the cross-modal retriever is: The prompt template, the speech representation and the text query are input into a key encoder to obtain a retrieval key, wherein the key encoder adopts a multi-level Transformer architecture based on causal multi-head attention; Using the search key to search in the search database, obtain the search value; The retrieval value is input into a value encoder to obtain a value vector, wherein the value encoder adopts a multi-level Transformer architecture; The prompt template, voice representation, text query and value vector are input into the retrieval enhancement decoder to obtain voice Tokens. The retrieval enhancement decoder adopts a multi-level Transformer architecture based on cross attention and causal multi-head attention. The multi-level Transformer architecture is connected to the language model head.

4. The audio semantic retrieval system for law enforcement recorders based on retrieval enhancement as claimed in claim 1, characterized in that: The processing of the query generation module is as follows: Clean the textual assumptions; use the word segmentation tool to segment the textual assumptions and perform part-of-speech tagging on each word; Use the named entity recognition model to tag each word in the text hypothesis, identify the entity name and its category, and extract the fragments that may contain the entity name; Combine fragments that may contain entity names with relevant context words to generate query fragments; sort the query fragments according to their importance or relevance to ensure that the most important or most relevant fragments are output first, and output query fragments that may contain entity names.

5. The audio semantic retrieval system for law enforcement recorders based on retrieval enhancement as claimed in claim 1, characterized in that: The entity retrieval module combines acoustic neighbor embedding, semantic embedding and traditional text retrieval methods for retrieval.

6. The audio semantic retrieval system for law enforcement recorders based on retrieval enhancement as claimed in claim 5, characterized in that: The results of acoustic neighbor embedding, semantic embedding and traditional text retrieval methods are fused to generate relevant entity names.

7. The audio semantic retrieval system for law enforcement recorders based on retrieval enhancement as claimed in claim 1, characterized in that: The process of context building module is: According to the Euclidean distance threshold and the count threshold, N entities most relevant to the text hypothesis are selected from the relevant entity names to obtain candidate entities; Construct hint information for each candidate entity; Combine the prompt information with the text hypothesis to form contextual information.

8. The audio semantic retrieval system for law enforcement recorders based on retrieval enhancement as claimed in claim 1, characterized in that: The processing process of the large language model is: The large language model uses the self-attention mechanism and multi-layer Transformer structure to deeply understand the relationship between the prompt information in the context information and the text hypothesis; infer the wrong part of the text hypothesis and generate the corrected text; Greedy decoding is used to generate the most likely output sequence and obtain the semantic retrieval result.

9. A semantic retrieval method for audio of law enforcement recorders based on retrieval enhancement, characterized in that: Using a law enforcement recorder audio semantic retrieval system based on retrieval enhancement as described in any one of claims 1 to 8, comprising the following steps: Obtain audio data and text queries recorded by law enforcement cameras; Project the audio data into the text embedding space to obtain speech representation; Perform cross-modal retrieval on speech representations and text queries to obtain speech tokens with text embedding and speech embedding in the same embedding space; According to the speech tokens, obtain the text hypothesis corresponding to the speech tokens; Extract query fragments that may contain entity names from textual hypotheses; According to the query fragment that may contain the entity name, the entity database is searched to obtain the relevant entity names; Construct contextual information based on relevant entity names and textual assumptions; According to the context information, semantic retrieval results are obtained.

Images