Domain-based speech recognition method and system based on RAG

Abstract

The present invention discloses a domain-based speech recognition method and system based on RAG, which includes obtaining an original domain document set, constructing a hierarchical document structure through subject classification, paragraph segmentation and sentence segmentation, performing vectorization processing to generate a multi-granularity vector matrix set, and establishing a hierarchical knowledge vector index structure; performing multi-scale time-frequency analysis on real-time audio streams, extracting and enhancing acoustic features, and generating multi-scale feature tensors; performing adaptive block segmentation and progressive attention processing on feature tensors, constructing a probability estimation model, and generating candidate text sequences; performing multi-layer retrieval in knowledge indexes based on multi-dimensional query vectors to generate relevant knowledge fragments; performing spatial mapping and dynamic attention processing on features, and performing probability optimization decoding to obtain the final recognized text. The present invention improves the accuracy and robustness of speech recognition in professional fields through knowledge enhancement and multi-level feature processing.

Description

Technical Field

[0001] The present invention belongs to the field of speech recognition, and in particular to a RAG-based domain-based speech recognition method and system. Background Art

[0002] In recent years, with the rapid development of scenarios such as smart manufacturing, telemedicine, and smart office, the demand for the application of domain-based speech recognition technology in professional environments has increased significantly. In these professional scenarios, the speech recognition system not only needs to accurately understand various professional terms and non-standard expressions, but also needs to be able to adapt to complex acoustic environments, such as industrial noise, multi-person conversations, and far-field pickup. Especially in the fields of medical diagnosis, engineering maintenance, and legal consulting, accurate speech recognition is of great significance to improving work efficiency and service quality. By introducing retrieval augmentation generation (RAG) technology and deeply integrating the domain knowledge base with the speech recognition system, the system's ability to understand professional scenarios and recognition accuracy can be significantly improved.

[0003] At present, mainstream speech recognition technologies are mainly based on deep learning methods, such as end-to-end sequence transcription models and Transformer-based encoding-decoding architectures. These methods have achieved good recognition results in general scenarios through large-scale data training. Some studies have tried to improve the performance of models in specific fields through technologies such as domain adaptation and transfer learning. In terms of acoustic feature extraction, traditional methods mainly rely on MFCC features and acoustic event detection, and perform feature fusion through simple feature concatenation or weighted summation. In terms of language modeling, commonly used methods include n-gram statistical models and simple neural network language models. These models mainly focus on the statistical laws of word sequences and have limited semantic understanding capabilities in professional fields.

[0004] However, there are still many challenges for existing technologies when dealing with speech recognition tasks in professional fields. First, in terms of acoustic feature processing, the traditional fixed block method is difficult to adapt to the dynamic changes of speech signals, especially when dealing with fast speech and non-standard pronunciation, which easily leads to inaccurate feature segmentation. Secondly, existing feature fusion methods often use simple feature splicing or linear combination of fixed weights, which cannot effectively deal with the interaction and importance differences between different features. Thirdly, in terms of knowledge retrieval, traditional vector retrieval methods face the problem of low retrieval efficiency when dealing with large-scale professional knowledge bases, and it is difficult to achieve multi-granularity precise matching. In addition, when dealing with long sentence recognition, the existing system is prone to semantic breaks and context inconsistency due to the lack of an effective long-range dependency modeling mechanism. Finally, in the feature enhancement stage, simple feature superposition cannot fully utilize the guiding role of domain knowledge on acoustic features, resulting in insufficient recognition accuracy of the system when dealing with professional terms and non-standard expressions. Summary of the invention

[0005] The purpose of the invention is to provide a RAG-based domain-based speech recognition method and system to solve the above-mentioned problems existing in the prior art.

[0006] The technical solution is a domain-based speech recognition method based on RAG, which includes the following steps:

[0007] S1. Obtain the original domain document set, classify the original domain document set by subject, and obtain the subject category set; segment the subject category set into paragraphs to obtain the paragraph set; segment the paragraph set into sentences to obtain the sentence set; integrate the subject category set, paragraph set, and sentence set to generate a hierarchical document structure; vectorize the texts of each layer in the hierarchical document structure to generate a multi-granularity vector matrix set; construct a retrieval index based on the multi-granularity vector matrix set to obtain a hierarchical index structure; combine the hierarchical index structure with the hierarchical document structure, construct a hierarchical mapping relationship, and output a hierarchical knowledge vector index structure;

[0008] S2. Obtain a real-time audio stream, perform multi-scale time-frequency analysis on the real-time audio stream, and generate multi-scale spectrum features; extract acoustic features based on the multi-scale spectrum features to obtain an enhanced feature set; perform context encoding on the enhanced feature set to obtain a context feature matrix; fuse and compress the context feature matrix with the multi-scale spectrum features to output a multi-scale feature tensor;

[0009] S3, perform adaptive block processing on the multi-scale feature tensor to obtain a feature block sequence; calculate progressive attention on the feature block sequence to obtain an attention feature sequence; based on the attention feature sequence, build a probability estimation model to generate a hierarchical probability matrix set; perform multi-beam decoding on the hierarchical probability matrix set to obtain a candidate text sequence set;

[0010] S4, performing multi-dimensional feature representation on the candidate text sequence set to generate a multi-dimensional query vector; based on the multi-dimensional query vector, performing multi-layer retrieval in the hierarchical knowledge vector index structure to obtain a multi-layer retrieval result set; based on the multi-layer retrieval result set and the candidate text sequence set, calculating the multi-dimensional relevance to obtain a sorted knowledge fragment set; performing context optimization on the sorted knowledge fragment set to generate a related knowledge fragment set;

[0011] S5. Based on the multi-scale feature tensor and the set of related knowledge fragments, spatial mapping is performed to obtain a unified feature space matrix; dynamic attention calculation is performed on the unified feature space matrix to obtain a fused feature tensor; context enhancement encoding is performed on the fused feature tensor to obtain an enhanced feature matrix; probability optimization decoding is performed on the enhanced feature matrix to obtain the final recognized text.

[0012] The RAG-based domain-based speech recognition system includes:

[0013] at least one processor; and,

[0014] a memory communicatively connected to at least one of the processors; wherein,

[0015] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the RAG-based domain-based speech recognition method.

[0016] Beneficial effects: The present invention establishes a knowledge base foundation that can be efficiently searched, solves the problem of insufficient understanding of professional domain knowledge by traditional speech recognition systems; enhances the system's adaptability to complex acoustic environments, improves the system's recognition ability for long sentences and fast speech, and overcomes the limitations of traditional fixed block methods; achieves high efficiency and accuracy in knowledge retrieval, enables deep fusion of acoustic features and knowledge features, and improves the system's understanding of domain-specific scenarios; improves recognition accuracy while maintaining low computational complexity and resource consumption, effectively solving the problem of insufficient understanding of domain knowledge by traditional speech recognition systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flow chart of the method of the present invention.

[0018] Figure 2 This is a flow chart of step S1 of the present invention.

[0019] Figure 3 This is a flow chart of step S2 of the present invention.

[0020] Figure 4 This is a flow chart of step S3 of the present invention.

[0021] Figure 5 This is a flow chart of step S4 of the present invention.

[0022] Figure 6 This is a flow chart of step S5 of the present invention. DETAILED DESCRIPTION

[0023] like Figure 1 As shown, the present application proposes a domain-based speech recognition method based on RAG, comprising the following steps:

[0024] S1. Obtain the original domain document set, classify the original domain document set by subject, and obtain the subject category set; segment the subject category set into paragraphs to obtain the paragraph set; segment the paragraph set into sentences to obtain the sentence set; integrate the subject category set, paragraph set, and sentence set to generate a hierarchical document structure; vectorize the texts of each layer in the hierarchical document structure to generate a multi-granularity vector matrix set; construct a retrieval index based on the multi-granularity vector matrix set to obtain a hierarchical index structure; combine the hierarchical index structure with the hierarchical document structure, construct a hierarchical mapping relationship, and output a hierarchical knowledge vector index structure;

[0025] S2. Obtain a real-time audio stream, perform multi-scale time-frequency analysis on the real-time audio stream, and generate multi-scale spectrum features; extract acoustic features based on the multi-scale spectrum features to obtain an enhanced feature set; perform context encoding on the enhanced feature set to obtain a context feature matrix; fuse and compress the context feature matrix with the multi-scale spectrum features to output a multi-scale feature tensor; wherein the dimension of the multi-scale feature tensor is t×f×d, where t represents the time dimension, f represents the frequency dimension, and d represents the feature dimension;

[0026] S3. Perform adaptive block processing on the multi-scale feature tensor to obtain a feature block sequence; calculate progressive attention on the feature block sequence to obtain an attention feature sequence; build a probability estimation model based on the attention feature sequence to generate a hierarchical probability matrix set; perform multi-beam decoding on the hierarchical probability matrix set to obtain a candidate text sequence set; wherein the feature tensor dimension is t×f×d, t represents the time dimension, f represents the frequency dimension, and d represents the feature dimension;

[0027] S4, performing multi-dimensional feature representation on the candidate text sequence set to generate a multi-dimensional query vector; based on the multi-dimensional query vector, performing multi-layer retrieval in the hierarchical knowledge vector index structure to obtain a multi-layer retrieval result set; based on the multi-layer retrieval result set and the candidate text sequence set, calculating the multi-dimensional relevance to obtain a sorted knowledge fragment set; performing context optimization on the sorted knowledge fragment set to generate a related knowledge fragment set; wherein the candidate text sequence set includes m sequences, and m represents the number of candidate sequences;

[0028] S5. Based on the multi-scale feature tensor and the set of related knowledge fragments, spatial mapping is performed to obtain a unified feature space matrix; dynamic attention calculation is performed on the unified feature space matrix to obtain a fused feature tensor; context enhancement encoding is performed on the fused feature tensor to obtain an enhanced feature matrix; probability optimization decoding is performed on the enhanced feature matrix to obtain the final recognized text.

[0029] like Figure 2 As shown, according to one aspect of the present application, step S1 is further:

[0030] S11, obtaining an original domain document set, classifying the original domain document set using a topic clustering algorithm, generating a topic category set containing n topic categories; segmenting each document in the topic category set using a semantic segmentation algorithm, generating a paragraph set containing p paragraphs; segmenting each paragraph in the paragraph set using a syntactic analysis method, generating a sentence set containing s sentences; integrating the topic category set, paragraph set, and sentence set according to a hierarchical relationship, and outputting a hierarchical document structure; wherein n represents the number of topics, p represents the total number of paragraphs, and s represents the total number of sentences;

[0031] S12, the topic level text in the hierarchical document structure is processed by a preconfigured topic vectorization model to generate a topic vector matrix with k rows and dt columns; the paragraph level text in the hierarchical document structure is processed by a preconfigured semantic vectorization model to generate a paragraph vector matrix with m rows and dp columns; the sentence level text in the hierarchical document structure is processed by a preconfigured context vectorization model to generate a sentence vector matrix with n rows and ds columns; the topic vector matrix, the paragraph vector matrix and the sentence vector matrix are integrated into a vector matrix set, and a multi-granularity vector matrix set is output; wherein k represents the number of topics, dt represents the topic vector dimension, m represents the number of paragraphs, dp represents the paragraph vector dimension, n represents the number of sentences, and ds represents the sentence vector dimension;

[0032] S13. Use the local sensitive hashing algorithm to construct an index for the topic vector matrix in the multi-granularity vector matrix set, and generate a topic-level index; use the hierarchical nearest neighbor search algorithm to construct an index for the paragraph vector matrix in the multi-granularity vector matrix set, and generate a paragraph-level index; use the inverted index algorithm to construct an index for the sentence vector matrix in the multi-granularity vector matrix set, and generate a sentence-level index; integrate the topic-level index, paragraph-level index, and sentence-level index, and output a hierarchical index structure;

[0033] S14. Based on the hierarchical index structure and hierarchical document structure, calculate the association between the topic level and the paragraph level, and generate a topic-paragraph mapping matrix with t rows and p columns; based on the hierarchical index structure and hierarchical document structure, calculate the association between the paragraph level and the sentence level, and generate a paragraph-sentence mapping matrix with p rows and s columns; combine the topic-paragraph mapping matrix and the paragraph-sentence mapping matrix into a hierarchical conversion matrix; integrate the hierarchical conversion matrix with the hierarchical index structure to generate a hierarchical knowledge vector index structure. Where t represents the number of topics, p represents the number of paragraphs, and s represents the number of sentences.

[0034] In one embodiment of the present application, the topic clustering algorithm is specifically as follows: the document topic similarity S(i, j) represents the comprehensive similarity between document i and document j, and the calculation formula is S(i, j) = α·cos(TF(i), TF(j)) + β·sim(LDA(i), LDA(j)) + γ·MI(KW(i), KW(j)); wherein cos(TF(i), TF(j)) = TF(i)·TF(j) / (||TF(i)||·||TF(j)||) is the TF-IDF feature cosine similarity; sim(LDA(i), LDA(j)) = exp(-KL(θi||θj)) is the LDA topic distribution similarity; MI(KW(i), KW(j)) = Σ p(x, y)log(p(x, y) / (p(x)p(y))) is the keyword mutual information measure; α, β, γ are adaptive weight coefficients, and α+β+γ=1; TF(i) represents the TF-IDF feature vector of document i; LDA(i) represents the topic distribution vector of document i; KW(i) represents the keyword set of document i; θi represents the topic distribution parameter of document i; p(x, y) represents the joint probability of the keyword pair; p(x) and p(y) represent the marginal probabilities of a single keyword, respectively.

[0035] The semantic segmentation algorithm is as follows: the paragraph boundary score B(t) represents the probability score of the text position t as the paragraph boundary, and the calculation formula is B(t) = λ·CosSim(V(tw), V(t+w)) + μ·LexCoh(t) + ν·TopicShift(t); where CosSim(V(tw), V(t+w)) = V(tw)·V(t+w) / (||V(tw)||·||V(t+w)||) is the word embedding cosine similarity of the previous and next text blocks in window w; LexCoh(t) = Σ(overlap(tk, t+k)) / k is the local lexical cohesion; TopicShift(t) = KL(θt-1||θt) is the topic transfer metric; λ, μ, ν are adaptive weight coefficients, and λ+μ+ν=1; V(t) represents the word embedding vector of the text block at position t; overlap(i, j) represents the word overlap of the text at positions i and j; θt represents the local topic distribution at position t; w represents the context window size; k represents the local connection calculation range.

[0036] The specific syntactic analysis method is as follows: the sentence grammatical structure score G(s) represents the grammatical completeness score at the boundary s of the candidate sentence, and the calculation formula is G(s) = δ·DepTree(s) + ε·SynCmp(s) + ζ·BoundMark(s); where DepTree(s) = Complete(T(s)) / Size(T(s)) is the dependency tree completeness score, and T(s) is the dependency syntactic tree at position s; SynCmp(s) = Σ(Match(Ri, Pi)) / |R| is the syntactic rule matching degree, Ri represents the predefined syntactic rule set, and Pi represents the syntactic pattern of the current sentence fragment; BoundMark(s) = σ(W(s)·H + b) is the boundary marking score, W(s) is the context feature vector at position s; δ, ε, ζ are adaptive weight coefficients, and δ+ε+ζ=1; Complete(T) represents the completeness score of the dependency tree T; Size(T) represents the size of the dependency tree T; Match(R, P) represents the matching degree between rule R and pattern P; σ(x) represents the sigmoid activation function; H is the hidden layer parameter matrix for boundary detection; b is the bias vector.

[0037] Locality sensitive hashing algorithm: The hash coding score H(x) represents the multi-level hash coding of vector x, and the calculation formula is H(x) = κ·LSH(x) +ξ·MLH(x) +χ*·AH(x); where LSH(x) = sign(Rx + b) represents random projection hash coding, R is the random projection matrix, and b is the random bias vector; MLH(x) = sign(Wx + Σ(y_i·U_i)) represents metric learning hash coding, W is the transformation matrix, y_i is the label information, and U_i is the label correlation matrix; AH(x) = sign(NN(x)) represents adaptive hash coding, NN represents neural network transformation; κ, ξ, χ* are adaptive weight coefficients, and κ+ξ+χ*= 1; sign represents the sign function; x represents the input feature vector.

[0038] Hierarchical nearest neighbor search algorithm: The nearest neighbor retrieval score N(q, p) represents the hierarchical similarity between the query vector q and the data point p, and the calculation formula is N(q, p) = μ·TreeDist(q, p) + ν·GraphDist(q, p) + ξ·LocalDist(q, p); where TreeDist(q, p) = min(Path(q, p)) represents the KD tree path distance, Path(q, p) represents the set of all possible paths from q to p; GraphDist(q, p) = ShortestPath(G, q, p) represents the shortest path distance of the adjacency graph, and G represents the nearest neighbor graph structure; LocalDist(q, p) = ||f(q)-f(p)|| represents the local feature space distance, and f(·) represents the feature mapping function; μ, ν, ξ are adaptive weight coefficients, and μ+ν+ξ=1.

[0039] Inverted index algorithm: The retrieval relevance R(q, d) represents the relevance between the query q and the document d, and the calculation formula is R(q, d) = α·TF(q, d) + β·IDF(q) + γ·PL(q, d); where TF(q, d) = Σ tf(t, d) represents the term frequency score, and tf(t, d) is the frequency of term t in document d; IDF(q) = log(N / df(q)) represents the inverse document frequency, N is the total number of documents, and df(q) is the number of documents containing the query term q; PL(q, d) = norm(Σ pos(q, d)) represents the position score, and pos(q, d) is the position information of the query term in the document; α, β, and γ are adaptive weight coefficients, and α+β+γ=1.

[0040] This embodiment constructs a hierarchical document structure by adopting hypergraph partitioning and multi-level processing, and realizes multi-granular representation and efficient retrieval of documents. Specifically, first, in the document preprocessing stage, regular expression cleaning and part-of-speech tagging are used to ensure the basic guarantee of text quality; secondly, the hypergraph partitioning algorithm is used for topic clustering, which not only captures the many-to-many complex relationship between documents, but also realizes the adaptive division of topic categories through the dynamic adjustment of hyperedge weights; thirdly, in the paragraph segmentation link, the semantic similarity sequence of word embedding features is calculated and combined with the dynamic programming algorithm to realize the determination of semantically coherent paragraph boundaries; finally, in the sentence segmentation stage, the semantic integrity and context relevance of sentences are guaranteed by dependency syntax tree analysis and coreference resolution. This embodiment not only improves the accuracy of subsequent retrieval, but also enhances the system's adaptability to queries of different lengths and complexities through multi-granular representation. Due to the use of optimization algorithms such as dynamic programming, the entire processing process achieves efficient computing performance while ensuring quality, especially when processing large-scale domain documents, showing efficiency advantages.

[0041] According to one aspect of the present application, step S11 is further:

[0042] S111, obtaining an original domain document set, using regular expressions to clean special characters and redundant formats in the text, and generating a cleaned document set; performing word segmentation and part-of-speech tagging on the cleaned document set, and generating a tagged document set; extracting TF-IDF features from the tagged document set, and generating a document feature matrix; constructing a document similarity hypergraph based on the document feature matrix, and generating a similarity hypergraph structure;

[0043] S112. Based on the similarity hypergraph structure, calculate the hyperedge weight, construct the Laplace matrix, and generate the hypergraph Laplace matrix; perform spectral clustering optimization on the hypergraph Laplace matrix to generate a cluster feature matrix; use a dynamic threshold to classify the cluster feature matrix and generate a topic label vector; based on the topic label vector, group the annotated document set and output a topic category set;

[0044] S113, based on the subject category set, extract word embedding features for each document to generate a document embedding matrix; calculate the semantic similarity of adjacent text segments in the document embedding matrix to generate a semantic similarity sequence; use a dynamic programming algorithm to find the optimal segmentation point on the semantic similarity sequence to generate a paragraph boundary set; based on the paragraph boundary set, segment the subject category set to obtain a paragraph set;

[0045] S114, based on the paragraph set, construct a dependency syntax tree for each paragraph to generate a syntax tree set; identify sentence boundary markers in the syntax tree set to generate a sentence boundary sequence; based on the sentence boundary sequence, segment the paragraph text in the paragraph set to generate an initial sentence set; perform coreference resolution and omission restoration on the initial sentence set to output a sentence set;

[0046] S115. Based on the subject category set, paragraph set and sentence set, a hierarchical index table is constructed to generate a hierarchical index structure; based on the hierarchical index structure, the mapping relationship between each level is calculated to generate a hierarchical association matrix; the hierarchical index structure and the hierarchical association matrix are combined to output a hierarchical document structure.

[0047] This embodiment achieves accurate division and semantic preservation of document structure through the design of hypergraph partitioning and hierarchical semantic analysis. First, in the text preprocessing stage, the quality of the basic text is guaranteed through regular expression cleaning and part-of-speech tagging, and the interference of special characters and redundant formats on subsequent processing is effectively removed; secondly, by constructing a document similarity hypergraph, the complex association relationship between documents is captured, which can better represent the topic association between multiple documents compared with the traditional bipartite graph method; then, through the spectral clustering optimization of the hypergraph Laplacian matrix, the adaptive division of topics is achieved, avoiding the limitations of the traditional fixed threshold method; then, through the paragraph segmentation of word embedding features and dynamic programming algorithms, the semantic coherence of paragraph division is guaranteed; finally, through the sentence normalization of dependency syntactic analysis and coreference resolution, the semantic integrity of the sentence level is ensured. This embodiment improves the quality and accuracy of document structuring through multi-level semantic analysis and optimization, especially when processing long documents in professional fields, it can accurately capture the topic structure and semantic hierarchy of documents, providing a reliable foundation for subsequent knowledge retrieval. At the same time, through the application of optimization algorithms such as dynamic programming, efficient processing performance is achieved, which reduces the consumption of computing resources while ensuring processing quality.

[0048] According to one aspect of the present application, step S12 is further:

[0049] S121. Obtain topic-level text in a hierarchical document structure, construct a term frequency-inverse document frequency matrix, and generate a TF-IDF feature matrix; perform singular value decomposition on the TF-IDF feature matrix to generate a topic semantic space; apply non-negative matrix decomposition in the topic semantic space to generate a topic base matrix; perform sparse coding optimization on the topic base matrix and output a topic vector matrix with k rows and dt columns.

[0050] S122. Obtain the topic vector matrix at the topic level, build a hierarchical word embedding model, and generate a word embedding sequence; apply the attention mechanism to the word embedding sequence to calculate the weights and generate a word weight matrix; perform weighted feature aggregation based on the word weight matrix to generate an aggregated feature vector; perform dimension calibration and normalization on the aggregated feature vector, and output the optimized k-row dt-column topic vector matrix.

[0051] S123. Obtain paragraph-level text in a hierarchical document structure, use a bidirectional encoder to extract features, and generate a context feature sequence; apply a self-attention mechanism to the context feature sequence to generate an attention feature matrix; perform multi-level feature fusion to generate a fused feature vector; perform dimensionality reduction compression on the fused feature vector, and output a paragraph vector matrix of m rows and dp columns.

[0052] S124. Obtain sentence-level text in a hierarchical document structure, build a deep language model, and generate a language representation sequence; apply a context-enhanced encoder to generate an enhanced feature matrix; perform multi-head attention calculations to generate a multi-head feature group; perform feature integration and dimensionality transformation on the multi-head feature group, and output a sentence vector matrix of n rows and ds columns.

[0053] S125. Obtain a topic vector matrix, a paragraph vector matrix, and a sentence vector matrix, calculate the correlation scores of each level vector, and generate a level correlation matrix; perform feature alignment based on the level correlation matrix to generate an alignment feature group; apply adaptive feature fusion to the alignment feature group to generate a fused feature set; normalize the fused feature set and output a multi-granularity vector matrix set.

[0054] This embodiment realizes the high-dimensional feature representation of documents through the design of deep topic vectorization and multi-level feature fusion. First, in the topic vectorization stage, the initial topic semantic space is constructed through the combination of TF-IDF feature extraction and singular value decomposition, and then a more accurate topic feature representation is extracted through non-negative matrix decomposition and sparse coding optimization; secondly, through hierarchical word embedding and attention weight calculation, the expressive power of word-level features is enhanced, especially when dealing with professional terms and field-specific expressions, it shows excellent feature capture ability; then, through the paragraph vectorization of bidirectional encoder and self-attention mechanism, the context-aware feature extraction at the paragraph level is realized, and the long-distance semantic dependency is effectively captured; then, through the sentence vectorization of deep language model and multi-head attention calculation, a fine-grained semantic representation is constructed; finally, through hierarchical relevance calculation and adaptive feature fusion, the effective integration of features of different granularities is realized. This embodiment improves the expressive power and distinguishability of features through the combination of deep learning model and attention mechanism, especially when dealing with documents in professional fields, it can accurately capture the semantic features and hierarchical structure of documents. At the same time, through feature alignment and normalization processing, the consistency and comparability of features of different granularities are ensured, providing a reliable feature basis for subsequent knowledge retrieval.

[0055] According to one aspect of the present application, step S13 is further:

[0056] S131. Obtain the topic vector matrix in the multi-granularity vector matrix set, perform random projection transformation, and generate a projection vector group; construct a hash function family for the projection vector group to generate a hash function set; calculate the vector fingerprint based on the hash function set to generate a vector fingerprint matrix; divide the vector fingerprint matrix into multiple hash buckets, and output the topic layer index.

[0057] S132. Obtain the topic layer index, calculate the similarity distribution of the vectors in the hash bucket, and generate a similarity distribution matrix; optimize the hash bucket division based on the similarity distribution matrix to generate an optimized hash table; construct a multi-level hash index structure to generate a multi-level index tree; perform balanced optimization on the multi-level index tree, and output the optimized topic layer index.

[0058] S133. Obtain a paragraph vector matrix from a multi-granularity vector matrix set, construct a hierarchical KD tree structure, and generate an initial KD tree; perform node splitting optimization on the initial KD tree to generate an optimized tree structure; calculate the nearest neighbor relationship between nodes to generate an adjacency relationship graph; combine the optimized tree structure with the adjacency relationship graph, and output a paragraph-level index.

[0059] S134. Obtain a sentence vector matrix from a multi-granularity vector matrix set, extract keyword features, and generate a keyword sequence; construct an inverted list structure and generate an inverted linked list; calculate term weights and generate a weight vector; optimize and sort the inverted linked list based on the weight vector, and output a sentence-level index.

[0060] S135. Obtain the topic layer index, paragraph layer index and sentence layer index, construct the mapping relationship between the layers, and generate an index mapping matrix; calculate the layer transition probability and generate a transition probability vector; optimize the layer access path based on the transition probability vector and generate an access path tree; integrate the index mapping matrix with the access path tree and output a hierarchical index structure.

[0061] This embodiment realizes efficient multi-granularity knowledge retrieval through the design of hierarchical index structure. First, in the subject layer index construction stage, the rapid encoding of the subject vector is realized through the combination of random projection and local sensitive hashing, and the storage structure of the index is optimized through the adaptive partitioning of the hash bucket; secondly, the accuracy and efficiency of the retrieval are improved through the similarity distribution analysis and the optimization of the multi-level hash index, especially when processing large-scale document sets, the time complexity of the retrieval is reduced; then, through the paragraph layer index of the hierarchical KD tree and the adjacency relationship graph, efficient neighbor search is realized, and the balance of the index is improved through node splitting optimization; then, through the design of the enhanced inverted index, accurate retrieval at the sentence level is realized, and the relevance of the retrieval results is improved through weight optimization; finally, through hierarchical mapping and access path optimization, efficient access to the multi-layer index structure is realized. This embodiment improves the efficiency and accuracy of retrieval through a multi-level index structure and optimization strategy, especially when processing large-scale professional knowledge bases, it can achieve fast and accurate knowledge positioning. At the same time, through conversion probability optimization and access path optimization, efficient access to the multi-layer index structure is ensured, while ensuring the retrieval quality, the response time of the system is reduced.

[0062] According to one aspect of the present application, step S14 is further:

[0063] S141. Obtain the topic-level text in the hierarchical index structure and the hierarchical document structure, construct a deep semantic network, and generate a semantic association graph; execute a community detection algorithm on the semantic association graph to generate a topic community set; calculate the text similarity within the community and generate a community similarity matrix; construct a hierarchical mapping relationship based on the community similarity matrix, and output a topic-paragraph mapping matrix with t rows and p columns.

[0064] S142. Obtain the topic-paragraph mapping matrix, apply the attention mechanism to calculate the mapping weights, and generate a mapping weight vector; perform sparse processing on the mapping weight vector to generate a sparse weight matrix; optimize the mapping relationship based on the sparse weight matrix to generate an optimized mapping matrix; perform normalization processing on the optimized mapping matrix, and output the optimized t-row and p-column topic-paragraph mapping matrix.

[0065] S143. Obtain paragraph-level and sentence-level texts in a hierarchical document structure, construct a hierarchical dependency syntax tree, and generate a dependency tree set; analyze the semantic dependencies in the dependency tree set to generate a dependency graph; perform path analysis on the dependency graph to generate a semantic path matrix; establish a hierarchical mapping based on the semantic path matrix, and output a paragraph-sentence mapping matrix with p rows and s columns.

[0066] S144. Obtain the topic-paragraph mapping matrix and the paragraph-sentence mapping matrix, construct a hierarchical conversion network, and generate a conversion network structure; calculate the conversion probability between levels and generate a conversion probability matrix; optimize the hierarchical conversion path and generate a path optimization matrix; combine the conversion probability matrix and the path optimization matrix and output the hierarchical conversion matrix.

[0067] S145. Obtain the hierarchical conversion matrix and the hierarchical index structure, construct a multi-level index mapping model, and generate an index mapping structure; calculate the association strength of the index nodes and generate an association strength matrix; optimize the index access path and generate an access path tree; integrate the index mapping structure, the association strength matrix, and the access path tree, and output a hierarchical knowledge vector index structure.

[0068] In one embodiment of the present application, the community detection algorithm: the topic community association degree CS(c_i, c_j) represents the association strength between communities c_i and c_j, and the calculation formula is CS(c_i, c_j) = ι·NodeSim(c_i, c_j) + κ·EdgeDen(c_i, c_j) + λ·CommFlow(c_i, c_j); where NodeSim(c_i, c_j) = |V(c_i)∩V(c_j)| / |V(c_i)∪V(c_j)| represents node similarity, and V(c) represents the node set in community c; EdgeDen(c_i, c_j) = |E(c_i, c_j)| / sqrt(|V(c_i)|·|V(c_j)|) represents the edge density, E(c_i, c_j) represents the set of edges connecting two communities; CommFlow(c_i, c_j) = MaxFlow(G(c_i), G(c_j)) represents the maximum flow between communities, G(c) represents the subgraph structure of community c; ι, κ, λ are adaptive weight coefficients, and ι+κ+λ=1.

[0069] This embodiment achieves accurate expression and efficient access to knowledge structure through the design of deep association analysis and hierarchical mapping optimization. First, in the topic-paragraph mapping construction stage, the combination of deep semantic network and community detection algorithm accurately captures the topic structure and hierarchical relationship of the document, especially when dealing with complex professional documents, it can effectively identify the potential associations between topics; secondly, through the mapping optimization of attention mechanism and sparse processing, the accuracy and interpretability of hierarchical mapping are improved, and through the adaptive adjustment of weights, a more accurate topic-paragraph association is achieved; then, through hierarchical dependency syntactic analysis and semantic path optimization, an accurate mapping relationship between paragraphs and sentences is constructed, effectively maintaining the semantic coherence of the text; then, through hierarchical conversion network and probability optimization, a smooth transition between different levels is achieved, and the accuracy of multi-level retrieval is improved; finally, through multi-level index mapping and access path optimization, an efficient knowledge access mechanism is constructed. This embodiment improves the expression ability and access efficiency of knowledge structure through multi-level association analysis and optimization strategies, especially when dealing with large-scale professional knowledge bases, it can achieve fast and accurate multi-level knowledge positioning. At the same time, through the optimization of association strength and access path, efficient access to the knowledge structure is ensured, while ensuring the retrieval quality, the response delay of the system is reduced.

[0070] like Figure 3 As shown, according to one aspect of the present application, step S2 is further:

[0071] S21, obtain a real-time audio stream, process the real-time audio stream using short-time Fourier transform, and generate a basic spectrum feature with a time-frequency resolution of t1×f1; use wavelet packet transform to perform multi-scale decomposition on the real-time audio stream, and generate a detailed spectrum feature with a time-frequency resolution of t2×f2; use a gammatone filter group to extract auditory features from the real-time audio stream, and generate an auditory spectrum feature with a time-frequency resolution of t3×f3; combine the basic spectrum feature, the detailed spectrum feature, and the auditory spectrum feature, and output a multi-scale spectrum feature; wherein t1, t2, and t3 represent different time scales, and f1, f2, and f3 represent different frequency resolutions;

[0072] S22. Based on multi-scale spectral features, the Mel-frequency cepstral coefficient extraction algorithm is applied to calculate acoustic features to generate an MFCC feature matrix with a dimension of t×m; the autocorrelation analysis method is used to extract the fundamental frequency trajectory to generate a fundamental frequency feature sequence with a dimension of t×p; the acoustic event detection algorithm is used to identify feature points to generate acoustic event features with a dimension of t×e; the MFCC feature matrix, the fundamental frequency feature sequence and the acoustic event features are combined to output an enhanced feature set; wherein t represents the number of time frames, m represents the number of MFCC coefficients, p represents the fundamental frequency parameter dimension, and e represents the acoustic event feature dimension;

[0073] S23. Based on the enhanced feature set, a bidirectional gated recurrent unit network is used to process the feature sequence to generate a forward feature matrix with a dimension of t×h; a causal convolutional network is used to process the feature sequence to generate a backward feature matrix with a dimension of t×h; the forward feature matrix is ​​concatenated with the backward feature matrix to output a context feature matrix with a dimension of t×2h; where t represents the number of time frames and h represents the number of hidden layer units;

[0074] S24. Based on the context feature matrix and multi-scale spectral features, a multi-head attention mechanism is used to calculate feature weights and generate an attention weight matrix with a dimension of t×h; based on the attention weight matrix, the features are weighted and fused to generate a fused feature matrix with a dimension of t×f; a deep autoencoder is used to reduce the dimension of the fused feature matrix and output a multi-scale feature tensor with a dimension of t×f×d. Where t represents the number of time frames, h represents the number of attention heads, f represents the feature dimension, and d represents the compressed feature dimension.

[0075] In one embodiment of the present application, the process of extracting the fundamental frequency trajectory is specifically as follows: the fundamental frequency trajectory score F(t, f) represents the comprehensive score of the frequency f at time t as the fundamental frequency, and the calculation formula is F(t, f) = η·ACF(t, f) + θ·HPS(t, f)+ ρ·CPE(t, f); wherein ACF(t, f) = Σ x(t+τ)x(t) represents the autocorrelation function score, and x(t) is the audio signal at time t; HPS(t, f) = ∏(|X(t, nf)| / n) represents the harmonic product spectrum score, X(t, f) is the time-frequency spectrum, n is the harmonic number, and ∏ is the product operator; CPE(t, f) = -Σ p(h|f)log(p(h|f)) represents the harmonic conditional entropy, and p(h|f) is the probability distribution of the harmonic h under a given fundamental frequency f; η, θ, ρ are adaptive weight coefficients, and η + θ + ρ = 1; τ represents the time delay; |X(t, f)| represents the amplitude of the time-frequency spectrum; h represents the harmonic component index.

[0076] The acoustic event detection algorithm is as follows: the acoustic event probability P(e|x) represents the posterior probability of event category e under a given acoustic feature x, and the calculation formula is P(e|x) = softmax(φ·CNN(x) + ψ·RNN(x) + ω·ATT(x)); where CNN(x) = Pool(Conv(x)) represents the convolutional network feature, Conv represents the convolution operation, and Pool represents the pooling operation; RNN(x) = LSTM(BiGRU(x)) represents the recurrent network feature, BiGRU represents the bidirectional gated recurrent unit, and LSTM represents the long short-term memory network; ATT(x) = Σ(α(t)·h(t)) represents the attention feature, h(t) is the hidden layer state, and α(t) is the attention weight; φ, ψ, and ω are adaptive fusion weights; x represents the input acoustic feature sequence; e represents the acoustic event category; and softmax represents the normalized exponential function.

[0077] Mel frequency cepstral coefficient extraction algorithm: MFCC feature M(t, f) represents the cepstral coefficient of time t and Mel frequency f, and the calculation formula is M(t, f) = ω·PreSpec(t, f) + ε·MelBank(t, f) + δ·DCT(t, f); where PreSpec(t, f) = |FFT(w(n)·x(n))| 2represents the power spectrum after pre-emphasis, w(n) is the Hamming window, and x(n) is the input signal; MelBank(t, f) = Σ H_m(k)·S(t, k) represents the Mel filter bank output, H_m(k) is the frequency response of the mth Mel filter, and S(t, k) is the short-time power spectrum; DCT(t, f) = Σ log(MelBank(t, k))·cos(πf(k+0.5) / M) represents discrete cosine transform, and M is the number of filter banks; ω, ε, δ are adaptive weight coefficients, and ω+ε+δ=1.

[0078] This embodiment improves the expressiveness and robustness of acoustic features through multi-level time-frequency analysis and feature enhancement processing. The system first captures the acoustic features at different time and frequency resolutions through three complementary time-frequency analysis methods: short-time Fourier transform, wavelet packet transform, and Gammatone filter group; then, through the adaptive pre-emphasis MFCC algorithm, the multi-objective optimized fundamental frequency estimation method, and the deep acoustic event detection algorithm, the multi-dimensional extraction of acoustic features is realized; finally, through the bidirectional feature encoding of the bidirectional gated recurrent unit network and the causal convolutional network, and the feature fusion of the multi-head attention mechanism, a multi-scale feature representation with powerful expressiveness is constructed. This embodiment is particularly suitable for processing speech signals in practical application scenarios such as background noise, speaker changes, and non-standard pronunciation. Through the diversity and complementarity of features, the system's adaptability to complex acoustic environments is improved, while maintaining low computational complexity and resource occupancy. In particular, the attention mechanism used in the feature fusion stage can adaptively adjust the weights of different features, further improving the robustness and generalization ability of the system.

[0079] According to one aspect of the present application, step S22 is further:

[0080] S221, based on the multi-scale spectrum characteristics, perform pre-emphasis filtering to generate a pre-emphasis signal sequence; perform frame windowing processing on the pre-emphasis signal sequence to generate a windowed signal matrix; calculate the short-time Fourier transform of the windowed signal matrix to generate a spectrum energy matrix; map the spectrum energy matrix to the Mel frequency scale to generate a Mel filter bank output;

[0081] S222, based on the output of the Mel filter group, calculate the logarithmic energy spectrum and generate a logarithmic energy matrix; based on the logarithmic energy matrix, perform discrete cosine transform to generate initial MFCC coefficients; based on the initial MFCC coefficients, use cepstrum mean subtraction to perform channel compensation to generate compensated MFCC coefficients; based on the compensated MFCC coefficients, calculate the first-order and second-order differential features to generate a dynamic feature matrix; splice the compensated MFCC coefficients with the dynamic feature matrix to form an MFCC feature matrix;

[0082] S223, based on the multi-scale spectrum features, using the autocorrelation analysis method to calculate the periodicity and generate a periodic score sequence; based on the periodic score sequence, using the adaptive time domain threshold detection to generate a candidate fundamental frequency point set; based on the candidate fundamental frequency point set, analyzing the harmonic structure; based on the harmonic structure, performing error correction on the candidate fundamental frequency point set to generate a corrected fundamental frequency sequence; based on the corrected fundamental frequency sequence, calculating the pitch contour parameters and generating a fundamental frequency parameter matrix; combining the corrected fundamental frequency sequence with the fundamental frequency parameter matrix to generate a fundamental frequency feature sequence;

[0083] S224. Based on multi-scale spectral features, a deep convolutional neural network is constructed to extract local features and generate a convolutional feature map; based on the convolutional feature map, a cyclic attention mechanism is applied to construct temporal dependencies and generate a temporal feature sequence; based on the temporal feature sequence, multi-task learning is performed and event classification is performed to generate an event category probability matrix; based on the event category probability matrix, event boundaries and durations are calculated to generate an event description vector; the event category probability matrix is ​​fused with the event description vector to obtain acoustic event features;

[0084] S225. Based on the MFCC feature matrix, the fundamental frequency feature sequence and the acoustic event feature, the feature vector is normalized to generate a normalized feature set; based on the normalized feature set, the feature weight coefficient is calculated to generate a feature weight vector; based on the feature weight vector, the normalized feature set is weighted combined to output an enhanced feature set.

[0085] In one embodiment of the present application, the cepstral mean subtraction method: the enhanced MFCC feature M'(t, d) represents the enhanced cepstral coefficient of time t and dimension d, and the calculation formula is M'(t, d) = ρ·CMN(M(t, d)) + τ·CVN(M(t, d)) +υ·CMS(M(t, d)); wherein CMN(M(t, d)) = M(t, d) - (1 / T)·Σ M(t, d) represents cepstral mean normalization, and T is the time window length; CVN(M(t, d)) = (M(t, d) - μ_d) / σ_d represents cepstral variance normalization, and μ_d and σ_d are the mean and standard deviation of dimension d, respectively; CMS(M(t, d)) = M(t, d) - α·MA(M(t, d)) represents cepstral mean subtraction, MA represents moving average, α represents smoothing coefficient; ρ, τ, υ represent adaptive weight coefficients, and ρ+τ+υ=1; M(t, d) represents original MFCC features.

[0086] This embodiment achieves high-quality acoustic feature representation through multi-dimensional acoustic feature extraction and deep feature fusion. First, in the MFCC feature extraction stage, the system's ability to express acoustic signals is enhanced through a combination of adaptive pre-emphasis and dynamic feature calculation, especially when dealing with different speakers and acoustic environments, showing excellent feature stability; secondly, through multi-objective optimized fundamental frequency estimation and harmonic structure analysis, the accuracy of pitch features is improved, effectively solving the problems of traditional methods in dealing with fast speech and non-standard pronunciation; then, through deep convolutional networks and acoustic event detection of cyclic attention mechanisms, fine-grained acoustic feature capture is achieved, and various acoustic events and boundaries are accurately identified; then, through feature optimization of multi-task learning and variational inference, the expressive power and generalization performance of features are improved; finally, through feature normalization and weight adaptive feature fusion, a robust acoustic feature representation is constructed. This embodiment improves the quality and robustness of acoustic features through the combination of deep learning and optimization algorithms, especially when dealing with complex practical application scenarios, it can accurately capture various features of acoustic signals. At the same time, through feature fusion and normalization processing, the balance and consistency between different features are ensured, providing a reliable feature basis for subsequent speech recognition.

[0087] like Figure 4 As shown, according to one aspect of the present application, step S3 is further:

[0088] S31, calculating the inter-frame feature change rate of the multi-scale feature tensor, generating a change rate matrix with a dimension of t×d; based on the change rate matrix, executing a dynamic programming algorithm, determining the optimal block length, and generating a block length sequence with a length of l; based on the block length sequence, dividing the multi-scale feature tensor into n overlapping blocks, and outputting a feature block sequence; wherein t represents the number of time frames, d represents the feature dimension, l represents the number of blocks, and n represents the total number of blocks generated;

[0089] S32. Based on the feature block sequence, apply self-attention calculation to each block to generate a local attention matrix with a dimension of n×h×l; based on the feature block sequence, calculate the attention relationship between adjacent blocks to generate a cross-block attention matrix with a dimension of n×h×(2l); fuse the local attention matrix with the cross-block attention matrix to output an attention feature sequence with a dimension of n×h×(3l); where n represents the number of blocks, h represents the number of attention heads, and l represents the block length;

[0090] S33. Based on the attention feature sequence, a preconfigured acoustic model is applied to calculate the phoneme level probability, and a phoneme probability matrix with a dimension of t×p is generated; based on the phoneme probability matrix, a preconfigured dictionary model is used to calculate the word level probability, and a word level probability matrix with a dimension of t×w is generated; based on the word level probability matrix, context information is extracted, the language model probability is calculated, and a context probability matrix with a dimension of t×c is generated; the phoneme probability matrix, the word level probability matrix and the context probability matrix are combined, and a hierarchical probability matrix set is output; wherein t represents the number of time frames, p represents the number of phonemes, w represents the vocabulary size, and c represents the number of context states;

[0091] S34. Based on the hierarchical probability matrix set, a variational inference algorithm is used to generate k decoding paths to form a candidate path matrix with a dimension of k×t; based on the candidate path matrix, the confidence score of each path is calculated to generate a confidence vector with a dimension of k; based on the confidence vector, the m paths with the highest scores are selected for text conversion, and a candidate text sequence set containing m sequences is output. Where k represents the number of initial candidate paths, t represents the number of time frames, and m represents the number of candidate sequences finally retained.

[0092] In one embodiment of the present application, the dynamic programming algorithm: the block boundary score D(t) represents the optimal score of the block boundary at time t, and the calculation formula is D(t) = max{D(t-1) + η·Var(t) + θ·Len(t) + ρ·Coh(t)}; where Var(t) = σ 2 (F[tw:t]) represents the feature change rate, F represents the feature sequence, and w represents the window size; Len(t) = norm(t-last(t)) represents the block length score, and last(t) represents the previous boundary position; Coh(t) = sim(B[tw:t], B[t:t+w]) represents the cohesion within the block, and B represents the feature block; η, θ, ρ are adaptive weight coefficients, and η+θ+ρ=1.

[0093] The specific calculation of applying self-attention is as follows: the attention feature score A(q, k) represents the correlation score between the query vector q and the key vector k, and the calculation formula is A(q, k) = π·LocalAttn(q, k) + τ·CrossAttn(q, k) + υ·ScaleAttn(q, k); where LocalAttn(q, k) = softmax(q·k T / sqrt(d_k))·v represents the local attention score, d_k is the feature dimension; CrossAttn(q, k) = softmax(f(q, k))·g(v) represents the cross-block attention score, f(q, k) is the correlation function, and g(v) is the value vector transformation function; ScaleAttn(q, k) = Σ(β_s·softmax(q·W_s·k T )) represents the multi-scale attention score, W_s is the transformation matrix at scale s; π, τ, υ are adaptive weight coefficients, and π+τ+υ=1; v represents the value vector; β_s represents the weight coefficient of scale s; softmax represents the normalized exponential function.

[0094] This embodiment achieves high-precision recognition of speech signal to text conversion through adaptive block and progressive attention mechanism. Specifically, firstly, by calculating the inter-frame feature change rate and using a dynamic programming algorithm to determine the optimal block length, the problem that the traditional fixed block method is difficult to adapt to the dynamic changes of the speech signal is solved; secondly, the dual mechanism of local attention and cross-block attention is adopted in the attention calculation link, which not only captures the detailed features within the block, but also establishes the long-range dependency relationship between adjacent blocks; then in the probability estimation stage, a complete set of hierarchical probability matrices is constructed through the multi-level probability fusion of acoustic models, dictionary models and language models; finally, variational inference and multi-beam decoding are used to generate candidate text sequences. This embodiment improves the system's recognition ability for fast speech, accent changes and non-standard expressions, especially when dealing with professional terms and long sentences, showing excellent accuracy and stability. Through the design of adaptive block, the system effectively balances the relationship between computing resource consumption and recognition accuracy, so that high recognition performance can be maintained under limited computing resource conditions.

[0095] According to one aspect of the present application, step S32 is further:

[0096] S321. Obtain a feature block sequence, perform a linear projection transformation on each block, and generate a projection feature sequence; calculate the scale-adaptive attention score of the projection feature sequence to generate an attention score matrix; perform softmax normalization on the attention score matrix to generate normalized attention weights; perform weighted aggregation on the features based on the normalized attention weights, and output the initial local features.

[0097] S322. Obtain initial local features, construct a multi-head attention layer for parallel feature extraction, and generate a multi-head feature group; perform residual connection and layer normalization on the multi-head feature group to generate a normalized feature sequence; apply a feedforward neural network to perform feature transformation and generate a transformed feature matrix; combine the normalized feature sequence with the transformed feature matrix and output a local attention matrix with a dimension of n×h×l.

[0098] S323. Obtain a feature block sequence, construct a feature matrix of adjacent block pairs, and generate a block pair feature set; perform bidirectional attention calculation on the block pair feature set to generate a bidirectional attention weight; extract cross-block interaction features based on the bidirectional attention weight to generate an interaction feature sequence; apply a gated update mechanism to fuse context information and output a cross-block attention matrix with a dimension of n×h×(2l).

[0099] S324. Obtain the local attention matrix and the cross-block attention matrix, calculate the feature correlation coefficient, and generate a feature correlation vector; perform adaptive feature selection based on the feature correlation vector to generate a selection feature matrix; use a nonlinear activation function to perform feature transformation to generate a transformed feature sequence; perform dimensionality rearrangement and concatenation operations on the transformed feature sequence, and output an attention feature sequence with a dimension of n×h×(3l).

[0100] S325. Obtain an attention feature sequence, construct a feature quality assessment model, and generate a feature quality score; perform feature importance sorting based on the feature quality score and generate a feature importance vector; optimize and reorganize the feature sequence according to the feature importance vector, and output an optimized attention feature sequence with a dimension of n×h×(3l).

[0101] This embodiment achieves high-quality temporal feature representation through multi-scale self-attention and cross-block feature fusion. First, in the feature extraction stage, the system's ability to capture features of different time scales is enhanced through the combination of linear projection transformation and scale adaptive attention, especially when processing variable-speed speech and non-standard speech, showing excellent feature adaptability; secondly, through the deep feature extraction of multi-head attention layer and residual connection, the feature expression ability and training stability are improved, and the gradient vanishing problem of traditional methods when processing long sequences is effectively solved; then, through the cross-block feature extraction of bidirectional attention calculation and gated update mechanism, the modeling of long-distance dependencies is realized, and the semantic association across time blocks is accurately captured; then, through the feature fusion of feature correlation calculation and adaptive feature selection, the distinguishability and effectiveness of features are improved; finally, through the feature optimization of feature quality evaluation and importance ranking, a robust attention feature representation is constructed. This embodiment improves the quality and robustness of temporal features through the combination of deep learning and optimization strategies, especially when processing complex real speech scenes, it can accurately capture speech features and semantic information at different time scales. At the same time, through feature selection and optimization processing, the simplicity and effectiveness of feature representation are ensured, providing reliable feature support for subsequent speech recognition.

[0102] like Figure 5 As shown, according to one aspect of the present application, step S4 is further:

[0103] S41. Based on the candidate text sequence set, a semantic encoder is used to encode each sequence to generate a semantic feature vector with a dimension of m×d_s; based on the semantic feature vector, a topic model is used to extract topic features to generate a topic feature vector with a dimension of m×d_t; based on the topic feature vector, keyword features are extracted to generate a keyword feature vector with a dimension of m×d_k; the semantic feature vector, the topic feature vector and the keyword feature vector are combined to output a multidimensional query vector; wherein m represents the number of candidate sequences, d_s represents the semantic feature dimension, d_t represents the topic feature dimension, and d_k represents the keyword feature dimension;

[0104] S42. Based on the multidimensional query vector and the hierarchical knowledge vector index structure, perform approximate nearest neighbor search in the topic layer index to generate a topic candidate set containing p topics; perform precise retrieval in the paragraph layer index to generate a paragraph candidate set containing q paragraphs; perform context matching in the sentence layer index to generate a sentence candidate set containing r sentences; combine the topic candidate set, the paragraph candidate set and the sentence candidate set to generate a multi-layer retrieval result set; wherein p represents the number of topic candidates, q represents the number of paragraph candidates, and r represents the number of sentence candidates;

[0105] S43. Based on the multi-layer retrieval result set and the candidate text sequence set, calculate the similarity at the semantic level and generate a semantic similarity matrix with a dimension of n×m; based on the semantic similarity matrix, calculate the consistency at the topic level and generate a topic consistency matrix with a dimension of n×m; based on the topic consistency matrix, calculate the context relevance and generate a context relevance matrix with a dimension of n×m; perform weighted fusion on the semantic similarity matrix, the topic consistency matrix and the context relevance matrix to generate a comprehensive relevance matrix with a dimension of n×m; based on the comprehensive relevance matrix, reorder the multi-layer retrieval result set and output the ordered knowledge fragment set; wherein n represents the number of retrieval results and m represents the number of candidate sequences;

[0106] S44. Based on the sorted knowledge fragment set, the information entropy calculation method is used to identify redundant content and generate a deduplicated knowledge fragment set; based on the deduplicated knowledge fragment set, the dynamic programming algorithm is used to optimize the fragment boundaries and generate a boundary-optimized fragment set; based on the boundary-optimized fragment set, the context relevance is calculated; based on the context relevance, the context information is supplemented to obtain a related knowledge fragment set. Each knowledge fragment contains the original text content and its position information in the document.

[0107] In one embodiment of the present application, a semantic encoder is used to encode each sequence as follows: the semantic feature vector E(x) represents the deep semantic representation of the text sequence x, and the calculation formula is E(x) = σ·Trans(x) +Φ·Deep(x) + Ξ·Fusion(x); wherein Trans(x) = MultiHead(Q(x), K(x), V(x)) represents Transformer encoding features, Q, K, and V represent query, key, and value transformation, respectively; Deep(x) = FFN(LayerNorm(x)) represents deep feedforward features, FFN represents feedforward network, and LayerNorm represents layer normalization; Fusion(x) = Gate(BiLSTM(x)) represents fusion features, Gate represents gating mechanism, and BiLSTM represents bidirectional long short-term memory network; σ, Φ, and Ξ are adaptive weight coefficients, and σ+Φ+Ξ=1; x represents the input text sequence.

[0108] Information entropy calculation method: The knowledge fragment redundancy R(s) represents the information redundancy degree of fragment s, and the calculation formula is R(s) = μ·ContentEnt(s) + ν·ContextEnt(s) + ξ·CrossEnt(s); where ContentEnt(s) = -Σ p(w|s)log(p(w|s)) represents content entropy, and p(w|s) represents the probability distribution of word w in fragment s; ContextEnt(s) = -Σ p(c|s)log(p(c|s)) represents context entropy, and p(c|s) represents the context distribution of fragment s; CrossEnt(s) = -Σ p(w|s)log(q(w)) represents cross entropy, and q(w) represents reference distribution; μ, ν, ξ are adaptive weight coefficients, and μ+ν+ξ=1; s represents the input knowledge fragment.

[0109] This embodiment improves the accuracy and efficiency of knowledge retrieval through multi-dimensional feature representation and multi-layer retrieval. First, in the feature representation stage, a comprehensive query vector representation is constructed by using a deep semantic encoder, a hierarchical topic model and a multi-level keyword feature extraction method; secondly, in the retrieval process, a multi-level retrieval strategy of hierarchical local sensitive hashing, precise semantic retrieval and context-enhanced matching is used to achieve a progressive search from coarse-grained to fine-grained; then, through the multi-dimensional correlation calculation of semantic similarity, topic consistency and context relevance, the comprehensiveness and accuracy of the retrieval results are ensured; finally, through information entropy calculation and dynamic programming optimization, the deduplication and boundary optimization of the retrieval results are achieved. This embodiment not only improves the accuracy of retrieval, but also reduces the computational complexity through a progressive search strategy, especially when dealing with large-scale knowledge bases, showing excellent efficiency and scalability. Through the design of multi-dimensional feature representation, the system enhances the depth of understanding of domain knowledge and effectively improves the relevance and completeness of retrieval results.

[0110] According to one aspect of the present application, step S41 is further:

[0111] S411. Obtain a set of candidate text sequences, perform standardized preprocessing on the text sequences, and generate a standard text set; build a context-aware word embedding model to generate a word embedding sequence; apply a bidirectional Transformer encoder to extract features and generate a context feature matrix; perform an attention pooling operation to aggregate features and output a semantic feature vector with a dimension of m×d_s.

[0112] S412. Obtain a standard text set, build a multi-layer Bayesian topic model, and generate an initial topic distribution; perform variational inference to optimize the topic distribution and generate an optimized topic distribution; calculate the document-topic relevance matrix and generate a topic relevance vector; perform sparse processing on the topic relevance vector and output a topic feature vector with a dimension of m×d_t.

[0113] S413. Obtain a standard text set, build a word graph network model, and generate a word graph structure; calculate node importance scores and generate word importance vectors; extract key phrases based on word graph path analysis and generate a key phrase set; combine the word importance vector with the key phrase set features and output a keyword feature vector with a dimension of m×d_k.

[0114] S414, obtain semantic feature vectors, topic feature vectors and keyword feature vectors, perform feature cross-fusion calculations, and generate a cross-feature matrix; apply nonlinear transformation to the cross-feature matrix to generate a transformation feature sequence; construct an attention mechanism for feature selection to generate a selection feature matrix; perform dimensional reorganization on the selection feature matrix to output a multi-dimensional query vector.

[0115] S415, obtain a multidimensional query vector, build a feature quality assessment model, and generate a feature scoring vector; perform feature enhancement based on the feature scoring vector to generate an enhanced feature matrix; apply adaptive normalization processing to generate a normalized feature vector; perform quantization encoding on the normalized feature vector, and output an optimized multidimensional query vector.

[0116] This embodiment achieves high-quality query vector generation through deep semantic coding and multi-dimensional feature fusion. First, in the semantic feature extraction stage, the system's ability to understand text semantics is enhanced through the combination of context-aware word embedding and bidirectional Transformer encoding, especially when dealing with professional terms and complex expressions, it shows excellent semantic capture capabilities; secondly, through the multi-layer Bayesian topic model and variational inference optimized topic feature extraction, the accuracy and interpretability of topic representation are improved, and the problems of traditional methods in dealing with multi-topic texts are effectively solved; then, through the keyword feature extraction of word graph network and path analysis, multi-level key information capture is achieved, and important concepts and phrases in the text are accurately identified; then, through feature cross-fusion and feature integration of attention selection, the expressiveness and distinguishability of features are improved; finally, through feature quality evaluation and adaptive normalization feature optimization, a robust query vector representation is constructed. This embodiment improves the quality and expressiveness of query vectors through the combination of deep learning and probabilistic models, especially when dealing with retrieval tasks in professional fields, it can accurately capture all aspects of query intent. At the same time, through feature fusion and optimization processing, the accuracy and availability of query vectors are ensured, providing a reliable query basis for subsequent knowledge retrieval.

[0117] According to one aspect of the present application, step S42 is further;

[0118] S421. Obtain the topic feature vector in the multidimensional query vector, perform local sensitive hash coding on the vector, and generate a query hash code; search for similar hash buckets in the topic layer index based on the query hash code to generate a candidate hash bucket set; calculate the similarity scores of the vectors in the candidate hash bucket set to generate a topic similarity matrix; select the p topics with the highest scores based on the topic similarity matrix, and output the topic candidate set.

[0119] S422. Obtain a topic candidate set, construct a topic relevance network, and generate a topic association graph; perform path extension on the topic association graph to generate an extended topic set; calculate importance weights for the extended topic set to generate a topic weight vector; reorder candidate topics based on the topic weight vector, and output an optimized topic candidate set.

[0120] S423, obtaining the paragraph feature vector in the multidimensional query vector, performing a vector space search in the paragraph layer index, and generating an initial paragraph set; calculating the semantic similarity between the query vector and the initial paragraph set, and generating a paragraph similarity matrix; applying a semantic reordering algorithm, and generating a reordered paragraph set; selecting q paragraphs with the highest similarity from the reordered paragraph set, and outputting a paragraph candidate set.

[0121] S424, obtain sentence feature vectors in the multidimensional query vector, build a context-aware matching model, and generate matching model parameters; perform retrieval in the sentence layer index based on the matching model parameters to generate an initial sentence set; calculate the context relevance score to generate a sentence relevance matrix; select the r sentences with the highest scores from the sentence relevance matrix and output the sentence candidate set.

[0122] S425. Obtain topic candidate sets, paragraph candidate sets and sentence candidate sets, construct a hierarchical association model, and generate a hierarchical association graph; calculate the consistency scores between levels and generate a consistency matrix; optimize the hierarchical relationship based on the consistency matrix and generate an optimized hierarchical structure; integrate the optimized hierarchical structure with the candidate results of each level and output a multi-layer retrieval result set.

[0123] This embodiment achieves efficient and accurate knowledge retrieval through hierarchical retrieval strategy and multi-dimensional correlation optimization. First, in the subject layer retrieval stage, fast topic positioning is achieved through the combination of local sensitive hash coding and similarity calculation. Compared with the traditional linear search method, the retrieval efficiency is improved by several orders of magnitude; secondly, the candidate topic optimization through the subject association network and path extension improves the recall rate and diversity of the retrieval results, effectively solving the problem that the traditional method easily misses related topics; then, through the paragraph retrieval of vector space search and semantic reordering, accurate paragraph matching is achieved, and the text fragments related to the query are accurately located; then, through the sentence retrieval of context-aware matching and correlation calculation, the accuracy and contextual coherence of the retrieval results are improved; finally, through the integration of the results of hierarchical association model and consistency optimization, a complete multi-layer retrieval result is constructed. This embodiment improves the efficiency and accuracy of retrieval through the combination of multi-level retrieval strategies and optimization algorithms, especially when processing large-scale professional knowledge bases, it can achieve fast and accurate knowledge positioning. At the same time, through hierarchical association optimization and consistency constraints, the integrity and coherence of the retrieval results are ensured, providing reliable knowledge support for knowledge-enhanced speech recognition.

[0124] like Figure 6 As shown, according to one aspect of the present application, step S5 is further:

[0125] S51. Based on the multi-scale feature tensor and the set of related knowledge fragments, a linear projection transformation is performed on the acoustic feature space to generate an acoustic projection matrix with a dimension of t×d_a; a nonlinear projection transformation is performed on the semantic feature space to generate a semantic projection matrix with a dimension of n×d_a; the transposed product of the acoustic projection matrix and the semantic projection matrix is ​​calculated to obtain a unified feature space matrix with a dimension of t×n; wherein t represents the number of time frames, n represents the number of knowledge fragments, and d_a represents the unified feature dimension;

[0126] S52. Based on the unified feature space matrix, calculate the feature correlation within the modality and generate the intra-modality attention weight with the dimension of h×t×t; based on the unified feature space matrix, calculate the interaction relationship between the modalities and generate the inter-modality attention weight with the dimension of h×t×n; combine the intra-modality attention weight and the inter-modality attention weight to generate a comprehensive attention tensor with the dimension of h×t×(t+n); based on the comprehensive attention tensor, apply the attention distillation algorithm to perform feature reorganization and output a fused feature tensor with the dimension of t×d_f; where h represents the number of attention heads, t represents the number of time frames, n represents the number of knowledge fragments, and d_f represents the fused feature dimension;

[0127] S53. Based on the fused feature tensor, a hierarchical Transformer structure is used for encoding to generate a hierarchical coding matrix with a dimension of t×d_h; based on the hierarchical coding matrix, a residual gating mechanism is used for feature enhancement to generate a residual feature matrix with a dimension of t×d_h; the hierarchical coding matrix is ​​added to the residual feature matrix to form an enhanced feature matrix with a dimension of t×d_h; where t represents the number of time frames and d_h represents the hidden layer feature dimension;

[0128] S54. Based on the enhanced feature matrix, an acoustic-semantic joint probability model is constructed to generate a joint probability matrix of dimension t×v; based on the joint probability matrix, a variational inference algorithm is applied to perform probability optimization to generate an optimized path matrix of dimension b×t; based on the optimized path matrix, beam search decoding is performed to obtain the final recognized text. Where t represents the number of time frames, v represents the vocabulary size, and b represents the beam width.

[0129] In one embodiment of the present application, the feature correlation within the modality is calculated as follows: the feature correlation strength C(f_i, f_j) represents the degree of correlation between features f_i and f_j, and the calculation formula is C(f_i, f_j) = α·PearsonCorr(f_i, f_j) + β·MutualInfo(f_i, f_j) + γ·DeepCorr(f_i, f_j); wherein PearsonCorr(f_i, f_j) = cov(f_i, f_j) / (σ_i·σ_j) represents the Pearson correlation coefficient, cov represents the covariance, and σ represents the standard deviation; MutualInfo(f_i, f_j) = Σ p(x, y)log(p(x, y) / (p(x)p(y))) represents the mutual information; DeepCorr(f_i, f_j) = NN(concat(f_i, f_j)) represents deep correlation, NN represents neural network; α, β, γ are adaptive weight coefficients, and α+β+γ=1; f_i, f_j represent the feature vectors to be analyzed.

[0130] The variational inference algorithm is applied for probability optimization as follows: the recognition path probability P(Y|X) represents the posterior probability of the text sequence Y given the acoustic feature sequence X, and the calculation formula is P(Y|X) = ζ·AM(Y|X) + η·LM(Y) + θ·KM(Y); where AM(Y|X) =Πp(y_t|x_t) represents the acoustic model score, and p(y_t|x_t) represents the output probability of the acoustic model at time t; LM(Y) =Πp(y_t|y_{tn:t-1}) represents the language model score, and p(y_t|y_{tn:t-1}) represents the conditional probability based on the first n words; KM(Y) = sim(Y, K) represents the knowledge model score, and sim(Y, K) represents the similarity between the text sequence Y and the knowledge base K; ζ, η, θ are adaptive weight coefficients, and ζ+η+θ=1; X represents the input acoustic feature sequence; Y represents the candidate text sequence.

[0131] This embodiment realizes the deep fusion of acoustic features and knowledge features through spatial mapping and dynamic attention mechanism. First, through linear and nonlinear projection transformations, the features of different modalities are mapped to a unified feature space; secondly, the dual mechanisms of intra-modal attention and inter-modal attention are adopted to capture the internal correlation and cross-modal interaction of features; then, through the deep feature enhancement of hierarchical Transformer and residual gating mechanism, the expressive power of features is improved; finally, through the optimized decoding of acoustic-semantic joint probability model and variational inference algorithm, the final recognition result is generated. This embodiment improves the system's ability to understand complex speech scenes, especially when dealing with terms and expressions in professional fields, and greatly improves the accuracy of recognition through knowledge enhancement. Through the design of dynamic attention mechanism, the system realizes adaptive adjustment of the importance of different features, effectively improving the robustness and generalization ability of the system.

[0132] According to one aspect of the present application, step S53 is further:

[0133] S531. Obtain the fused feature tensor, construct a multi-layer position encoder, and generate a position encoding sequence; combine the fused feature tensor with the position encoding sequence to generate an augmented feature matrix; perform multi-head self-attention calculation on the augmented feature matrix to generate an attention feature sequence; apply a feedforward neural network to perform feature conversion and output the initial encoding feature.

[0134] S532. Obtain initial coding features, build a hierarchical coding module, and generate a hierarchical feature group; perform cross-layer feature interaction on the hierarchical feature group to generate an interactive feature matrix; apply an adaptive feature aggregation mechanism to generate an aggregated feature sequence; perform normalization on the aggregated feature sequence, and output a hierarchical coding matrix with a dimension of t×d_h.

[0135] S533. Obtain the fused feature tensor, build a deep residual network, and generate a residual feature map; apply a dynamic gating mechanism to the residual feature map to generate a gated feature sequence; perform multi-scale feature extraction to generate a multi-scale feature group; adaptively fuse the multi-scale feature group and output a residual feature matrix with a dimension of t×d_h.

[0136] S534, obtain the hierarchical coding matrix and the residual feature matrix, calculate the feature correlation score, and generate a correlation weight vector; perform weighted feature fusion based on the correlation weight vector to generate a fused feature sequence; apply a nonlinear activation function to perform feature transformation to generate a transformed feature matrix; perform dimensionality reorganization on the transformed feature matrix, and output an enhanced feature matrix with a dimension of t×d_h.

[0137] S535. Obtain an enhanced feature matrix, construct a feature evaluation model, and generate a feature quality index; perform feature optimization based on the feature quality index to generate an optimized feature sequence; apply adaptive quantization coding to generate a quantized feature matrix; perform normalization processing on the quantized feature matrix, and output an optimized enhanced feature matrix with a dimension of t×d_h.

[0138] This embodiment achieves high-quality feature enhancement through the design of hierarchical Transformer encoding and deep residual optimization. First, in the feature encoding stage, the system's modeling ability for sequence features is enhanced through the combination of multi-layer position encoding and multi-head self-attention, especially when dealing with long sequences and complex dependencies, showing excellent feature capture capabilities; secondly, through hierarchical encoding and feature optimization of cross-layer feature interaction, the expressiveness and hierarchy of features are improved, effectively solving the information loss problem of traditional methods when dealing with multi-level features; then, through the feature enhancement of deep residual network and dynamic gating mechanism, effective feature extraction and selection are achieved, and feature information of different scales is accurately captured; then, through weighted feature fusion and feature integration of nonlinear transformation, the expressiveness and distinguishability of features are improved; finally, through feature evaluation and adaptive quantization feature optimization, a robust enhanced feature representation is constructed. This embodiment improves the quality and expressiveness of features through the combination of advanced deep learning architecture and optimization strategy, especially when dealing with complex speech recognition tasks, it can accurately capture all aspects of acoustic and semantic features. At the same time, through feature fusion and optimization processing, the effectiveness and reliability of enhanced features are ensured, providing high-quality feature support for the final speech recognition.

[0139] According to one aspect of the present application, it also includes:

[0140] S6. Based on the final recognition text and the candidate text sequence set, perform multi-dimensional error analysis on the text sequence to obtain an error feature vector; based on the error feature vector, perform system performance analysis to obtain a performance feature vector; based on the error feature vector and the performance feature vector, generate an optimization plan to obtain an optimization strategy set; based on the optimization strategy set, adjust system parameters to obtain an optimization parameter set. The text sequence includes the word sequence of the recognition result and its timestamp information.

[0141] S61, obtain the final recognition text and the candidate text sequence set, calculate the word-level edit distance, generate a word error matrix with a dimension of m×n, where m represents the final text length and n represents the number of candidate sequences; use the semantic similarity calculation model to generate a semantic deviation vector with a dimension of d; construct a confidence statistical model to generate a confidence distribution matrix with a dimension of k×b; combine the word error matrix, the semantic deviation vector, and the confidence distribution matrix, and output an error feature vector. Where d represents the semantic feature dimension, k represents the number of confidence levels, and b represents the number of statistical intervals.

[0142] S62, obtain the error characteristic vector, use the performance monitoring algorithm to calculate the processing delay of each module, generate a performance monitoring matrix with a dimension of c×t; collect system resource occupancy data, generate a resource utilization vector with a dimension of r; locate the system bottleneck position, generate a bottleneck location matrix with a dimension of p×q; combine the performance monitoring matrix, resource utilization vector, and bottleneck location matrix, and output a performance characteristic vector. Where c represents the number of system modules, t represents the number of time windows, r represents the number of resource types, p represents the number of processing nodes, and q represents the number of performance indicators.

[0143] S63, obtain the error characteristic vector and the performance characteristic vector, build a decision tree model for strategy generation, generate a decision matrix with dimension a×b; apply the reinforcement learning algorithm to optimize strategy selection, generate a learning strategy matrix with dimension l×m; integrate the two strategy matrices, and output the optimized strategy set. Where a represents the number of decision nodes, b represents the number of decision options, l represents the number of learning states, and m represents the dimension of the action space.

[0144] S64, obtain an optimization strategy set, update the weight parameters of the attention mechanism according to the strategy, generate an attention weight matrix with a dimension of h×d; adjust the threshold parameters of the knowledge retrieval module, generate a retrieval threshold vector with a dimension of k; optimize the configuration parameters of the decoder, generate a decoding parameter vector with a dimension of s; combine the attention weight matrix, the retrieval threshold vector, and the decoding parameter vector, and output the optimized parameter set. Among them, h represents the number of attention heads, d represents the feature dimension, k represents the number of retrieval parameters, and s represents the number of decoding parameters.

[0145] This embodiment can accurately locate and correct recognition errors by calculating the edit distance and semantic similarity at the word level, thereby improving the accuracy of the final recognized text; the use of a confidence statistical model helps to identify and filter low-confidence recognition results, thereby further improving the reliability of recognition; the processing delay of each module is calculated by a performance monitoring algorithm, and the performance of the system can be monitored in real time; the system resource occupancy data is collected, and system resources can be effectively managed and allocated to avoid resource waste; locating the system bottleneck position to generate a bottleneck positioning matrix helps to identify and solve the performance bottleneck in the system and improve the overall processing efficiency; constructing a decision tree model for strategy generation can generate the optimal strategy according to different recognition and performance conditions; applying a reinforcement learning algorithm to optimize strategy selection, through continuous learning and adjustment, further optimize the system's strategy selection, and improve the system's adaptability; optimizing the configuration parameters of the decoder can enable the system to maintain optimal performance in different tasks and environments; by integrating the attention weight matrix, the retrieval threshold vector and the decoding parameter vector, the system can better handle complex text recognition tasks and provide higher quality recognition results. This embodiment improves the accuracy, performance and adaptability of the text recognition system through multi-level error analysis, performance monitoring, intelligent strategy generation and comprehensive optimization, and can provide a more reliable and efficient solution in practical applications.

[0146] According to one aspect of the present application, a domain-based speech recognition method based on RAG comprises the following steps:

[0147] Step 1: Document processing: Collect and organize existing documents from related fields, follow the Retrieval-Augmented Generation (RAG) processing method, and structure and pre-process the documents for subsequent retrieval.

[0148] The input document is preprocessed, including text cleaning, formatting, and segmentation, to ensure a clear content structure. The preprocessed text is vectorized using a word embedding model such as BERT or GPT. The vectorized text is converted into a high-dimensional vector that can effectively express the semantic information of the document. These high-dimensional vectors represent the subject and core content of the text, which helps to efficiently retrieve relevant information in subsequent steps. After vectorization, the system stores these high-dimensional vectors in a vector database. The database has fast retrieval capabilities and can find the best matching text content through vector similarity calculations when a query request is received. This process ensures that text data that is highly relevant to the audio content can be obtained in the subsequent speech transcription step.

[0149] Step 2: Audio input and automatic speech recognition (ASR) result acquisition: Input the audio data to be recognized into the ASR model to obtain preliminary ASR transcription results. Input the user's audio file into the ASR model for preliminary transcription to obtain preliminary text results. This transcription may contain some recognition errors, especially in areas involving professional terms and proper nouns, so it is only used as the basic input for subsequent processing.

[0150] Step 3: Vector database search: Input the above ASR results as queries into the vector database, and retrieve relevant text to obtain document content that is highly relevant to the ASR results. Match the preliminary transcription text with the text content in the vector database previously generated by RAG, and retrieve document content related to the transcription text.

[0151] Step 4, fusion input and re-identification: The retrieved relevant text is further extracted to obtain hot word entities that are highly relevant to the audio content, and then the hot word entities are input into the ASR model as context to perform the final ASR recognition task. The retrieved document content is used as supplementary information and input into the ASR model again together with the original audio. At this point, the ASR model can refer to these related document contents to more accurately identify the professional terms and expressions in the audio and generate the final transcription. This process of associative retrieval and fine transcription helps to reduce the misrecognition rate, especially when it comes to professional terms in specific fields, and can ensure that the final transcription is more accurate.

[0152] In another embodiment of the present application, some variables are set for the representation of the subsequent processing process, namely: input audio file A, preliminary transcription text T_init, input document D, vectorization function V(∙), vector database V, retrieval function R(∙), ASR model ASR(∙), and final transcription text T_final. The input document D is converted into a vector representation: V(D)=v_1, v_2,..., v_n, where v_i represents the vector representation of the i-th block in the document. All document vectors V(D) are stored in the vector database V for subsequent retrieval. The input audio 𝐴 is input into the ASR model for preliminary transcription to obtain the preliminary transcription text T_init: T_init=ASR(A). The preliminary transcription text T_init is used to retrieve the text vector related to it in the vector database V: D_rel=R(T_init, V), where D_rel represents the document content related to T_init. The preliminary transcription text T_init and the related text D_rel are input into the ASR model together with the audio A to generate the final transcription text T_final: T_final=ASR(A, D_rel). The final transcription text T_final is the high-precision text result obtained after the assistance of related documents: T_final=ASR(A, R(ASR(A),V)).

[0153] By combining the ASR transcription results with related documents, this embodiment can use contextual information and domain knowledge to improve the accuracy of speech recognition. It is expected that in applications in specific fields, the recognition accuracy will be improved by about 5% to 10%; real-time information retrieval is achieved through the vector database, so that the model can quickly obtain relevant information when facing complex domain speech, overcoming the limitations of traditional static models; due to the use of RAG technology, while achieving high recognition accuracy, it optimizes resource usage and reduces dependence on a large amount of labeled data, thereby reducing manpower and time costs. This embodiment can adapt to the needs of various industries and has strong flexibility. It can adjust the retrieval and recognition strategies according to the characteristics of different fields to meet specific application scenarios.

[0154] According to one aspect of the present application, a domain-based speech recognition system based on RAG includes:

[0155] at least one processor; and,

[0156] a memory communicatively connected to at least one of the processors; wherein,

[0157] The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the RAG-based domain-based speech recognition method described in any of the above embodiments.

[0158] The present invention achieves high precision and high reliability of domain-based speech recognition by constructing a complete RAG enhanced speech recognition framework. First, through the construction and vectorization of a hierarchical document structure, an efficient and retrievable knowledge base is established, solving the problem of insufficient understanding of professional domain knowledge by traditional speech recognition systems; secondly, through multi-scale spectrum analysis and deep feature extraction, the system's adaptability to complex acoustic environments is enhanced, especially when dealing with practical application scenarios such as background noise and speaker changes, showing excellent robustness; thirdly, through adaptive block and progressive attention mechanism, the system's recognition ability for long sentences and fast speech is improved, overcoming the limitations of traditional fixed block methods; then, through multi-dimensional feature representation and multi-layer retrieval strategy, the efficiency and accuracy of knowledge retrieval are achieved, especially when dealing with professional terms and non-standard expressions, showing significant advantages; finally, through spatial mapping and dynamic attention mechanism, the deep fusion of acoustic features and knowledge features is achieved, improving the system's ability to understand domain-specific scenarios. The present invention not only improves the recognition accuracy, but also maintains low computational complexity and resource consumption, showing excellent performance and scalability in practical applications. Especially when dealing with speech recognition tasks in professional fields, the problem of insufficient understanding of domain knowledge by traditional speech recognition systems is effectively solved through knowledge enhancement, providing reliable technical support for speech recognition applications in professional fields.

[0159] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.

Claims

1. A domain-based speech recognition method based on RAG, characterized in that: The steps include: S1. Obtain an original domain document set, perform subject classification on the original domain document set, and obtain a subject category set; perform paragraph segmentation on the subject category set, and obtain a paragraph set; perform sentence segmentation on the paragraph set, and obtain a sentence set; Integrate the subject category set, paragraph set, and sentence set to generate a hierarchical document structure; Vectorize the texts of each layer in the hierarchical document structure to generate a multi-granularity vector matrix set; construct a retrieval index based on the multi-granularity vector matrix set to obtain a hierarchical index structure; Combine the hierarchical index structure with the hierarchical document structure, build a hierarchical mapping relationship, and output a hierarchical knowledge vector index structure; S2, obtaining a real-time audio stream, performing multi-scale time-frequency analysis on the real-time audio stream, and generating multi-scale spectrum features; Based on multi-scale spectrum features, acoustic features are extracted to obtain an enhanced feature set; Perform context encoding on the enhanced feature set to obtain a context feature matrix; The context feature matrix is ​​fused and compressed with the multi-scale spectrum features to output a multi-scale feature tensor; S3, performing adaptive block processing on the multi-scale feature tensor to obtain a feature block sequence; Calculate progressive attention on the feature block sequence to obtain an attention feature sequence; Based on the attention feature sequence, a probability estimation model is constructed to generate a hierarchical probability matrix set; multi-beam decoding is performed on the hierarchical probability matrix set to obtain a candidate text sequence set; S4, performing multi-dimensional feature representation on the candidate text sequence set to generate a multi-dimensional query vector; Based on the multi-dimensional query vector, multi-layer retrieval is performed in the hierarchical knowledge vector index structure to obtain a multi-layer retrieval result set; Based on the multi-layer retrieval result set and the candidate text sequence set, the multi-dimensional relevance is calculated to obtain the sorted knowledge fragment set; context optimization is performed on the sorted knowledge fragment set to generate the relevant knowledge fragment set; S5. Based on the multi-scale feature tensor and the set of related knowledge fragments, spatial mapping is performed to obtain a unified feature space matrix; Perform dynamic attention calculation on the unified feature space matrix to obtain the fused feature tensor; Perform context enhancement encoding on the fused feature tensor to obtain an enhanced feature matrix; Probabilistic optimization decoding is performed on the enhanced feature matrix to obtain the final recognized text.

2. The RAG-based domain-based speech recognition method according to claim 1, characterized in that: Step S1 is further as follows: S11, obtaining an original domain document set, classifying the original domain document set using a topic clustering algorithm to generate a topic category set; segmenting each document in the topic category set using a semantic segmentation algorithm to generate a paragraph set; segmenting each paragraph in the paragraph set using a syntactic analysis method to generate a sentence set; Integrate the subject category set, paragraph set, and sentence set according to the hierarchical relationship and output a hierarchical document structure; S12, processing the topic-level text in the hierarchical document structure using a preconfigured topic vectorization model to generate a topic vector matrix; processing the paragraph-level text in the hierarchical document structure using a preconfigured semantic vectorization model to generate a paragraph vector matrix; processing the sentence-level text in the hierarchical document structure using a preconfigured context vectorization model to generate a sentence vector matrix; Integrate the topic vector matrix, paragraph vector matrix and sentence vector matrix into a vector matrix set, and output a multi-granularity vector matrix set; S13. Use the local sensitive hashing algorithm to construct an index for the topic vector matrix in the multi-granularity vector matrix set, and generate a topic-level index; use the hierarchical nearest neighbor search algorithm to construct an index for the paragraph vector matrix in the multi-granularity vector matrix set, and generate a paragraph-level index; use the inverted index algorithm to construct an index for the sentence vector matrix in the multi-granularity vector matrix set, and generate a sentence-level index; integrate the topic-level index, paragraph-level index, and sentence-level index, and output a hierarchical index structure; S14, based on the hierarchical index structure and the hierarchical document structure, calculating the correlation between the topic level and the paragraph level, and generating a topic-paragraph mapping matrix; Based on the hierarchical index structure and hierarchical document structure, the correlation between the paragraph level and the sentence level is calculated to generate a paragraph-sentence mapping matrix; Combining the topic-paragraph mapping matrix and the paragraph-sentence mapping matrix into a hierarchical transformation matrix; The hierarchical transformation matrix is ​​integrated with the hierarchical index structure to generate a hierarchical knowledge vector index structure.

3. The RAG-based domain-based speech recognition method according to claim 2, characterized in that: Step S2 is further as follows: S21, obtaining a real-time audio stream, processing the real-time audio stream by short-time Fourier transform, generating basic spectrum features; performing multi-scale decomposition of the real-time audio stream by wavelet packet transform, generating detailed spectrum features; performing auditory feature extraction on the real-time audio stream by Gammatone filter group, generating auditory spectrum features; Combine basic spectrum features, detailed spectrum features and auditory spectrum features to output multi-scale spectrum features; S22. Based on multi-scale spectrum features, the Mel frequency cepstral coefficient extraction algorithm is applied to calculate acoustic features and generate MFCC feature matrix; the autocorrelation analysis method is used to extract the fundamental frequency trajectory and generate the fundamental frequency feature sequence; the acoustic event detection algorithm is used to identify feature points and generate acoustic event features; Combine the MFCC feature matrix, fundamental frequency feature sequence and acoustic event features to output an enhanced feature set; S23, based on the enhanced feature set, a bidirectional gated recurrent unit network is used to process the feature sequence to generate a forward feature matrix; Use a causal convolutional network to process the feature sequence and generate a backward feature matrix; concatenate the forward feature matrix with the backward feature matrix to output the context feature matrix; S24, based on the context feature matrix and multi-scale spectrum features, a multi-head attention mechanism is used to calculate feature weights and generate an attention weight matrix; Based on the attention weight matrix, the features are weighted and fused to generate a fused feature matrix; A deep autoencoder is used to reduce the dimension of the fused feature matrix and output a multi-scale feature tensor.

4. The RAG-based domain-based speech recognition method according to claim 3, characterized in that: Step S3 is further as follows: S31, calculating the inter-frame feature change rate of the multi-scale feature tensor and generating a change rate matrix; based on the change rate matrix, executing a dynamic programming algorithm to determine the optimal block length and generating a block length sequence; Based on the block length sequence, the multi-scale feature tensor is divided into a predetermined number of overlapping blocks, and a feature block sequence is output; S32, based on the feature block sequence, apply self-attention calculation to each block to generate a local attention matrix; Based on the feature block sequence, the attention relationship between adjacent blocks is calculated to generate a cross-block attention matrix; the local attention matrix is ​​fused with the cross-block attention matrix to output an attention feature sequence; S33, based on the attention feature sequence, applying a preconfigured acoustic model to calculate the phoneme level probability and generate a phoneme probability matrix; Based on the phoneme probability matrix, the pre-configured dictionary model is used to calculate the word-level probability and generate the word-level probability matrix; based on the word-level probability matrix, the context information is extracted, the language model probability is calculated, and the context probability matrix is ​​generated; the phoneme probability matrix, the word-level probability matrix and the context probability matrix are combined to output a hierarchical probability matrix set; S34, based on the hierarchical probability matrix set, using a variational inference algorithm to generate a decoding path to form a candidate path matrix; Based on the candidate path matrix, the confidence score of each path is calculated to generate a confidence vector; Based on the confidence vector, a predetermined path with the highest score is selected for text conversion, and a set of candidate text sequences is output.

5. The RAG-based domain-based speech recognition method according to claim 4, characterized in that: Step S4 is further as follows: S41, based on the candidate text sequence set, using a semantic encoder to encode each sequence to generate a semantic feature vector; based on the semantic feature vector, using a topic model to extract topic features to generate a topic feature vector; based on the topic feature vector, extract keyword features to generate a keyword feature vector; combine the semantic feature vector, the topic feature vector and the keyword feature vector to output a multi-dimensional query vector; S42, based on the multi-dimensional query vector and the hierarchical knowledge vector index structure, perform approximate nearest neighbor search in the topic layer index to generate a topic candidate set; perform precise search in the paragraph layer index to generate a paragraph candidate set; perform context matching in the sentence layer index to generate a sentence candidate set; combine the topic candidate set, the paragraph candidate set and the sentence candidate set to generate a multi-layer search result set; S43, based on the multi-layer retrieval result set and the candidate text sequence set, calculating the similarity at the semantic level, and generating a semantic similarity matrix; based on the semantic similarity matrix, calculating the consistency at the topic level, and generating a topic consistency matrix; Based on the topic consistency matrix, the context relevance degree is calculated to generate the context relevance matrix; Perform weighted fusion on the semantic similarity matrix, topic consistency matrix and context relevance matrix to generate a comprehensive relevance matrix; Based on the comprehensive relevance matrix, the multi-layer retrieval result set is re-ranked and the ranked knowledge fragment set is output; S44, based on the sorted knowledge fragment set, using the information entropy calculation method to identify redundant content and generate a deduplicated knowledge fragment set; based on the deduplicated knowledge fragment set, using the dynamic programming algorithm to optimize the fragment boundaries and generate a boundary optimized fragment set; based on the boundary optimized fragment set, calculating the context relevance; Based on the context relevance, the context information is supplemented to obtain a set of related knowledge fragments.

6. The RAG-based domain-based speech recognition method according to claim 5, characterized in that: Step S5 is further as follows: S51. Based on the multi-scale feature tensor and the set of related knowledge fragments, a linear projection transformation is performed on the acoustic feature space to generate an acoustic projection matrix; a nonlinear projection transformation is performed on the semantic feature space to generate a semantic projection matrix; Calculate the transposed product of the acoustic projection matrix and the semantic projection matrix to obtain a unified feature space matrix; S52, based on the unified feature space matrix, calculating the feature correlation within the modality and generating the intra-modality attention weight; Based on the unified feature space matrix, the interaction relationship between modalities is calculated to generate inter-modal attention weights; The intra-modality attention weights and inter-modality attention weights are combined to generate a comprehensive attention tensor. Based on the comprehensive attention tensor, the attention distillation algorithm is applied to perform feature reorganization and output a fused feature tensor. S53, based on the fused feature tensor, a hierarchical Transformer structure is used for encoding to generate a hierarchical encoding matrix; Based on the hierarchical coding matrix, the residual gating mechanism is applied to perform feature enhancement and generate a residual feature matrix; Add the hierarchical coding matrix to the residual feature matrix to form an enhanced feature matrix; S54, based on the enhanced feature matrix, construct an acoustic-semantic joint probability model and generate a joint probability matrix; Based on the joint probability matrix, the variational inference algorithm is applied to perform probability optimization and generate the optimization path matrix; Based on the optimized path matrix, beam search decoding is performed to obtain the final recognized text.

7. The RAG-based domain-based speech recognition method according to claim 6, characterized in that: Step S11 is further as follows: S111, obtaining an original domain document set, using regular expressions to clean special characters and redundant formats in the text, and generating a cleaned document set; performing word segmentation and part-of-speech tagging on the cleaned document set, and generating a tagged document set; Extract TF-IDF features from the annotated document set and generate a document feature matrix; Based on the document feature matrix, a document similarity hypergraph is constructed to generate a similarity hypergraph structure; S112. Based on the similarity hypergraph structure, calculate the hyperedge weights, construct the Laplacian matrix, and generate the hypergraph Laplacian matrix; Perform spectral clustering optimization on the hypergraph Laplacian matrix to generate a clustering feature matrix; A dynamic threshold is used to classify the cluster feature matrix and generate a topic label vector; Based on the topic label vector, the annotated document set is grouped and the topic category set is output; S113, based on the topic category set, extract word embedding features for each document and generate a document embedding matrix; Calculate the semantic similarity of adjacent text segments in the document embedding matrix to generate a semantic similarity sequence; use a dynamic programming algorithm to find the optimal segmentation point on the semantic similarity sequence to generate a paragraph boundary set; based on the paragraph boundary set, segment the topic category set to obtain a paragraph set; S114, based on the paragraph set, construct a dependency syntax tree for each paragraph to generate a syntax tree set; Identify sentence boundary markers in the syntactic tree set and generate sentence boundary sequences; Based on the sentence boundary sequence, segment the paragraph text in the paragraph set to generate an initial sentence set; Perform coreference resolution and omission restoration on the initial sentence set and output the sentence set; S115. Based on the subject category set, the paragraph set and the sentence set, a hierarchical index table is constructed to generate a hierarchical index structure; Based on the hierarchical index structure, the mapping relationship between each level is calculated to generate a hierarchical association matrix; the hierarchical index structure and the hierarchical association matrix are combined to output a hierarchical document structure.

8. The RAG-based domain-based speech recognition method according to claim 6, characterized in that: Step S22 is further as follows: S221, based on the multi-scale spectrum characteristics, perform pre-emphasis filtering to generate a pre-emphasis signal sequence; perform frame windowing processing on the pre-emphasis signal sequence to generate a windowed signal matrix; calculate short-time Fourier transform on the windowed signal matrix to generate a spectrum energy matrix; Map the spectral energy matrix to the Mel frequency scale to generate the Mel filter bank output; S222, based on the output of the Mel filter bank, calculating the logarithmic energy spectrum and generating a logarithmic energy matrix; Based on the logarithmic energy matrix, a discrete cosine transform is performed to generate the initial MFCC coefficients; Based on the initial MFCC coefficients, the cepstral mean subtraction method is used to perform channel compensation to generate compensated MFCC coefficients; Based on the compensated MFCC coefficients, the first-order and second-order differential features are calculated to generate a dynamic feature matrix; The compensated MFCC coefficients are concatenated with the dynamic feature matrix to form the MFCC feature matrix; S223, based on the multi-scale spectrum characteristics, using the autocorrelation analysis method to calculate the periodicity and generate a periodic score sequence; Based on the periodic score sequence, adaptive time domain threshold detection is used to generate a candidate fundamental frequency point set; based on the candidate fundamental frequency point set, the harmonic structure is analyzed; Based on the harmonic structure, error correction is performed on the candidate fundamental frequency point set to generate a corrected fundamental frequency sequence; Based on the modified fundamental frequency sequence, the pitch contour parameters are calculated to generate a fundamental frequency parameter matrix; the modified fundamental frequency sequence is combined with the fundamental frequency parameter matrix to generate a fundamental frequency feature sequence; S224. Based on multi-scale spectral features, a deep convolutional neural network is constructed to extract local features and generate a convolutional feature map. Based on the convolutional feature map, a cyclic attention mechanism is applied to construct temporal dependencies and generate a temporal feature sequence. Based on the time series feature sequence, multi-task learning is performed and event classification is performed to generate an event category probability matrix; Based on the event category probability matrix, the event boundary and duration are calculated to generate an event description vector; the event category probability matrix is ​​fused with the event description vector to obtain the acoustic event feature; S225, based on the MFCC feature matrix, the fundamental frequency feature sequence and the acoustic event feature, normalizing the feature vector to generate a normalized feature set; Based on the normalized feature set, the feature weight coefficient is calculated to generate a feature weight vector; Based on the feature weight vector, the normalized feature set is weighted combined to output the enhanced feature set.

9. The RAG-based domain-based speech recognition method according to claim 6, characterized in that: Also includes: S6. Based on the final recognized text and the candidate text sequence set, a multi-dimensional error analysis is performed on the text sequence to obtain an error feature vector; based on Error characteristic vector, perform system performance analysis to obtain performance characteristic vector; based on the error characteristic vector and the performance characteristic vector, generate optimization plan to obtain optimization strategy set; based on the optimization strategy set, adjust system parameters to obtain optimization parameter set.

10. The domain-based speech recognition system based on RAG is characterized by: include: at least one processor; as well as, a memory communicatively connected to at least one of the processors; wherein, The memory stores instructions that can be executed by the processor, and the instructions are used to be executed by the processor to implement the RAG-based domain-based speech recognition method described in any one of claims 1 to 9.

Images