An intelligent recall strategy scheduling method, device and medium for the lao language

Abstract

The embodiment of the specification discloses a kind of Laos language intelligent recall strategy scheduling method, equipment and medium, it is related to big model technical field, method includes: receiving the Laos language query text input by user, to carry out multi-dimensional query intent analysis to Laos language query text, determine the intent analysis result of Laos language query text;According to intent analysis result, in the preset recall channel combination configuration table, determine at least one target activated recall channel and corresponding recall quantity upper limit parameter corresponding to current query, recall channel includes sparse word item recall channel, dense vector recall channel and entity directional recall channel;Based on target activated recall channel and corresponding recall quantity upper limit parameter, determine the candidate document set of each target activated recall channel and corresponding document score, to determine fusion uniform candidate set.Single recall strategy in low resource language scene solves the problem of coverage blind area, improves the coverage of recall result and the stability of candidate set.

Description

Technical Field

[0001] This specification relates to the field of large model technology, and in particular to a Lao-language intelligent recall strategy scheduling method, device and medium. Background Technology

[0002] In the search and knowledge-based question-answering scenarios of the Lao language AI service platform, recall is a core component of the information retrieval pipeline. The recall task involves rapidly filtering a set of candidate documents semantically relevant to the user's query from a large-scale document database, providing a high-quality candidate set for the downstream reranking and answer generation modules. The quantity and quality of the recall results determine the upper bound of the entire retrieval system; if relevant documents are missed during the recall phase, subsequent reranking and answer generation cannot compensate for this deficiency.

[0003] Currently, information retrieval and intelligent question-answering systems for Lao mainly employ sparse term matching recall methods or dense vector retrieval methods based on term matching. Sparse term matching recall methods, exemplified by BM25, rely on the literal overlap between query terms and document terms to calculate relevance. However, Lao uses a continuous writing system where words are typically not separated by spaces. Existing word segmentation tools have limited accuracy in segmenting word boundaries in this language, leading to misalignment between terms stored in the inverted index and the actual terms queried by the user. Furthermore, Lao has a rich vocabulary of polysemous words, synonyms, and colloquial variations. When the vocabulary used in the user's query differs from the expression in the document (i.e., a lexical mismatch exists), term matching methods cannot recall semantically relevant documents, resulting in low recall rates and unstable candidate set quality. Dense vector recall methods based on bi-encoders compress queries and documents into dense vectors of fixed dimensions before calculating similarity. They are good at capturing semantic relevance, but perform poorly in scenarios that require precise word matching (such as queries for proper nouns like names of people, organizations, and technical terms).

[0004] Therefore, in the context of low-resource Lao language, the existing recall methods have adaptation defects with the language characteristics of Lao, such as word segmentation ambiguity caused by continuous writing without spaces, which poses a risk of high missed recall rate and unstable candidate set quality. Summary of the Invention

[0005] This specification provides one or more embodiments of a Lao language intelligent recall strategy scheduling method, device, and medium to solve the following technical problem: In the low-resource language scenario of Lao, there is a lack of compatibility between existing recall methods and the language characteristics of Lao, such as word segmentation ambiguity caused by continuous writing without spaces, which poses a risk of high missing recall rate of related documents and unstable quality of candidate set.

[0006] One or more embodiments of this specification employ the following technical solutions: This specification provides one or more embodiments of a Lao language intelligent recall strategy scheduling method, the method comprising: receiving Lao language query text input by a user, performing multi-dimensional query intent analysis on the Lao language query text, and determining the intent analysis result of the Lao language query text; based on the intent analysis result, determining at least one target active recall channel and a corresponding recall quantity upper limit parameter corresponding to the current query in a preset recall channel combination configuration table, wherein the recall channel includes one or more of sparse term recall channels, dense vector recall channels, and entity-oriented recall channels; based on the target active recall channel and the corresponding recall quantity upper limit parameter, determining a candidate document set and a corresponding document score for each target active recall channel, so as to determine a fused unified candidate set through multiple candidate document sets and corresponding document scores.

[0007] This specification provides one or more embodiments of a Lao-language intelligent recall strategy scheduling device, comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the above-described method.

[0008] This specification provides one or more embodiments of a non-volatile computer storage medium storing computer-executable instructions configured to perform the above-described method.

[0009] The above-described at least one technical solution used in the embodiments of this specification can achieve the following beneficial effects: By introducing multi-dimensional query intent analysis, the system proactively identifies the semantic characteristics and low-resource features of user queries, avoiding the waste of computational resources caused by indiscriminate full-channel retrieval. Based on the intent analysis results, it dynamically determines the set of recall channels to be activated for the current query from a pre-defined recall channel combination configuration table. This allows for the selection of the most suitable recall path for lexical, semantic, entity, and mixed queries, reducing invalid retrieval overhead while ensuring recall quality. Simultaneously, based on the intent analysis results, a corresponding recall quantity limit parameter is configured for each activated recall channel, enabling dynamic adjustment of the candidate document truncation quantity for each recall channel according to query information. This avoids over-recall in simple queries and ensures sufficient coverage in complex queries. By executing each activated recall channel in parallel to obtain candidate document sets and corresponding document scores, the system leverages the complementary advantages of sparse term matching, dense semantic matching, and entity-oriented retrieval. Finally, a unified candidate set is generated through score fusion, effectively solving the coverage blind spot problem of a single recall strategy in low-resource language scenarios. This significantly improves the coverage rate and stability of the candidate set, while optimizing computational resource utilization. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 A flowchart illustrating a Lao-language intelligent recall strategy scheduling method provided in the embodiments of this specification; Figure 2 This is a schematic diagram of the structure of a Lao-language intelligent recall strategy scheduling device provided as an embodiment of this specification. Detailed Implementation

[0011] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0012] This specification provides an intelligent recall strategy scheduling method for Lao language. It should be noted that the execution entity in this specification embodiment can be a server or any device with data processing capabilities. Figure 1This is a flowchart illustrating a Lao-language intelligent recall strategy scheduling method provided in an embodiment of this specification, as shown below. Figure 1 As shown, the main steps include the following: Step S101: Receive the Lao query text input by the user, perform multi-dimensional query intent analysis on the Lao query text, and determine the intent analysis results of the Lao query text.

[0013] A multi-dimensional query intent analysis was performed on the Lao query text to determine the intent analysis results. Specifically, this included: performing multi-level word boundary segmentation on the Lao query text to obtain a first segmentation sequence, a second segmentation sequence, and a third segmentation sequence. This multi-level word boundary segmentation included a dictionary-based maximum matching segmenter, a conditional random field-based statistical segmenter, and a syllable rule-based deterministic segmenter. The union of the first, second, and third segmentation sequences was used to determine the word-level unit sequence. Based on this word-level unit sequence, the Lao query text was analyzed for intent. The Lao query text is evaluated for low-resource features to determine corresponding low-resource feature indicators, including word segmentation ambiguity index, mixed language degree index, and low-frequency word coverage. A multi-dimensional feature vector is determined using the word-level unit sequence and the low-resource feature indicators. Based on this multi-dimensional feature vector and a pre-trained intent classification model, the query intent type of the Lao query text is determined. This query intent type includes one of lexical, semantic, entity, and mixed types. The intent analysis result is then determined using the query intent type and the low-resource feature indicators.

[0014] In low-resource Lao language scenarios, there are issues such as word segmentation ambiguity due to continuous writing without spaces and the mixed use of multiple languages, as well as the frequent appearance of some low-frequency technical terms. Without multi-dimensional intent analysis of user queries, subsequent recall strategy scheduling will result in the use of a uniform recall channel combination and fixed weight configuration regardless of the query type, thus failing to achieve a dynamic balance between the accuracy of term matching and the generalization ability of semantic vectors.

[0015] In one embodiment of this specification, Lao query text input by a user is received. After receiving the Lao query text input by the user, a multi-dimensional query intent analysis is performed on the Lao query text, specifically implemented in the following manner: The first step involves multi-level word boundary segmentation of the Lao query text. This is achieved by running three word segmenters in parallel to segment the Lao query text and determine the first, second, and third segmentation sequences. It should be noted that the three word segmenters include a dictionary-based maximum matching segmenter, a conditional random field-based statistical segmenter, and a syllable rule-based deterministic segmenter.

[0016] Specifically, the dictionary-based maximum matching word segmenter utilizes a pre-built Lao basic vocabulary and employs a forward maximum matching algorithm. Starting from the beginning of the Lao query text, it sequentially attempts to match words in the Lao basic vocabulary. Let the length of the Lao query text be L. It first attempts to select a substring of length L from the current pointer position as a candidate word and matches it in the dictionary. If a match is successful, the substring is output as a segmentation unit, and the pointer is moved L positions forward. The remaining part is then shortened to the new substring after the pointer, and L is updated to the length of this new substring. If a match fails, L is decremented by one unit, for example, by one character. A new, shorter substring is extracted and a match is attempted again. This process is repeated until a match is successful or L decreases to a preset minimum word length threshold (usually 1). After a successful match, the pointer moves, the remaining part is updated, and L is updated accordingly. The process continues until the pointer reaches the end of the text, at which point the remaining part is empty, the segmentation is complete, and the first segmentation sequence is output. Each element in the first segmentation sequence is a word unit. It should be noted that the construction of the basic Lao lexicon can be achieved by collecting basic entries from publicly available Lao dictionary resources. Building upon this, high-frequency Lao everyday vocabulary from Southeast Asian daily vocabulary classification dictionaries can be further integrated. High-frequency co-occurring word strings can also be extracted from Lao news corpora using automatic word string extraction algorithms. These strings are then reviewed by linguistic experts and incorporated into the lexicon to cover commonly used expressions in the news field. For emerging Lao vocabulary (such as scientific terms and foreign transliterations), a user feedback-driven lexicon iteration mechanism is established. New words and terms are collected through a crowdsourcing platform and incorporated into the standard lexicon after expert review. After deduplication of all collected entries, a structured basic Lao lexicon is constructed, with each entry containing the original Lao word and its part-of-speech tag.

[0017] The statistical word segmenter based on Conditional Random Fields (CRF) uses a CRF model trained on a small amount of annotated Lao corpus to predict boundary marker sequences for each character position in the query text. It then segments the words based on these marker sequences and outputs a second segmentation sequence. The training corpus for the CRF model, LaoCORPUS, includes manually annotated Lao vocabulary, covering both formal and informal written Lao texts. For low-resource languages ​​like Lao, where corpus resources are extremely limited, large-scale supervised learning methods cannot be directly used for lexical analysis. Therefore, a semi-supervised learning approach is adopted to obtain annotated corpus. Utilizing the limited annotated dictionary and unannotated corpus resources, a simple probabilistic model is used to obtain a relatively complete annotated dictionary. Then, an integer programming method is used to obtain a large amount of automatically annotated corpus. After corpus preparation, the word boundary segmentation problem is formalized as a sequence labeling task. Each character position in the corpus is marked with a word boundary using the BI labeling system, where B represents the first character of a word and I represents a character within a word. Based on the BI tagging system, each Lao character in the training corpus is labeled with a corresponding boundary label. When selecting feature templates, by analyzing the characteristics of the Lao language and text, atomic features such as the Unicode code point values ​​of the current character and its preceding and following characters, character type features (consonants, vowels, tone marks, etc.), and the character's position within a syllable are selected to construct a combined feature template. Using Conditional Random Field (CRF) toolkits such as CRF++, a CRF model is trained on the labeled corpus and feature templates, and the model parameters are learned through maximum conditional likelihood estimation. After training, the CRF model trained on a small amount of labeled Lao corpus is obtained. Each character in the Lao query text is used as input to the model. Based on the learned feature weights and the transition probabilities between adjacent labels, the model predicts the boundary label (B or I) for each character. Based on the predicted boundary label sequence, consecutive characters are segmented into terms, and a second segmentation sequence is output.

[0018] In Lao syllable writing rules, each syllable consists of a consonant (which may include a prepositional consonant), a vowel, and an optional tone mark. A syllable can contain at most one vowel core, and the position of the consonant is key to determining the start of a new syllable. Lao is a syllable-based language, and each character in its writing system essentially represents a syllable, with syllable boundaries definitively determined by the position of the consonant. Therefore, a deterministic segmenter based on syllable rules can utilize Lao syllable writing rules, where syllable boundaries can be definitively determined by the position of the consonant. Without relying on any dictionary or statistical model, it directly outputs a third segmentation sequence, where each unit is a syllable. Thus, the third segmentation sequence can also be called a syllable-level unit sequence. In one example, the LaoBreakIterator class in the Lucene framework can be used. This tool is specifically designed to segment Lao text according to syllable boundaries. Its segmentation logic references the Lao script syllable segmentation algorithm, which uses a greedy matching principle. It traverses the text character sequence from front to back. When a consonant character is encountered, it is determined as the start of a new syllable, and subsequent consecutive consonants, vowels, and tone marks are grouped into the same syllable until the next consonant character is encountered or the text ends. For complex cases with ambiguous boundaries (such as multiple possible syllable boundaries in the sequence to be segmented), a backtracking adjustment method is used. First, a syllable is matched according to the greedy rule. If the subsequent remaining character sequence does not conform to the syllable rule, the character at the end of the previous syllable is moved to the current syllable. Then, the two adjusted syllables are re-verified to conform to the Lao syllable rule. This process is repeated until all syllables are valid.

[0019] The second step involves taking the union of the first, second, and third segmentation sequences to determine the word-level unit sequence. This sequence includes all possible word boundaries identified by the three segmentation methods, maximizing the preservation of lexical information in the query. It should be noted that the first, second, and third segmentation sequences are all ordered sequences, recording the order in which each term appears in the original query text. When determining the word-level unit sequence by taking the union, all terms from the three segmentation sequences are collected into a list, retaining the starting character position of each term in the original query text. The list is then sorted in ascending order based on the starting character position. Finally, adjacent duplicate terms (i.e., terms appearing at the same character position in multiple segmentation sequences) are removed, resulting in a word-level unit sequence arranged in the original text order. Based on the word-level unit sequence, a low-resource feature assessment is performed on the Lao query text to determine the corresponding low-resource feature indicators. In one embodiment, these indicators include a segmentation ambiguity index, a mixed language index, and low-frequency word coverage, implemented as follows: Based on the first, second, and third segmentation sequences, the Jaccard similarity coefficient is calculated between any two segmentation sequences. The Jaccard similarity coefficient equals the number of elements in the intersection of the sets corresponding to the two sequences divided by the number of elements in the union, measuring the degree of overlap between the two segmentation results. In languages ​​with continuous writing without spaces, the segmentation result sets generated by different segmenters may differ in the number of elements and the position of word boundaries. The Jaccard similarity coefficient is defined as the size of the intersection of the two sets divided by the size of the union, and its value ranges from 0 to 1, intuitively reflecting the proportion of overlap in word composition between the two segmentation sequences. Since the essence of word segmentation ambiguity is that the same character sequence is segmented into different sets of words, rather than a change in the order of words, and the Jaccard coefficient does not depend on the order or frequency of words, but only on the membership relationship of the word sets, it helps to measure the overall consistency of the word segmentation results. The word segmentation ambiguity index corresponding to the Lao query text is determined by the arithmetic mean of multiple Jaccard similarity coefficients. The closer the word segmentation ambiguity index is to 1, the more consistent the results of the three word segmenters are (low ambiguity), and the closer it is to 0, the more serious the divergence is (high ambiguity).

[0020] Each word is extracted from the word-level unit sequence and input into a pre-trained language classifier to determine the language label corresponding to each word. The number of times the language label switches within the word-level unit sequence is counted, and the degree of mixed language in the Lao query text is determined based on the ratio of the number of switches to the sequence length of the word-level unit sequence. Specifically, each word is extracted sequentially from the word-level unit sequence and input into a pre-trained language classifier, which is based on character-level n... Gram features and a Naive Bayes model can determine whether a term belongs to Lao, Chinese, English, or Thai, thus assigning a corresponding language label to each term. The entire sequence of word-level units is traversed, and the number of times the language label changes (i.e., the number of times the language category changes from one term to the next) is counted. Dividing this number by the sequence length (i.e., the total number of terms) yields a language mixing index. A higher index indicates more frequent language mixing in the query and greater interference with the pure semantic vector.

[0021] Each word in the word-level unit sequence is compared with a pre-constructed Lao word frequency table to count the number of low-frequency words below a preset threshold. The low-frequency word coverage rate of the Lao query text is determined based on the ratio of the number of low-frequency words to the sequence length of the word-level unit sequence. Specifically, the frequency of each word in a large-scale general Lao corpus is pre-counted, sorted in descending order, and a preset percentile is used as the threshold. Words with frequencies below this threshold are defined as low-frequency words. Each word in the word-level unit sequence is compared one by one with the Lao word frequency table. The corpus frequency corresponding to each word in the word-level unit sequence is read, and it is determined whether the predicted frequency is lower than the preset threshold. Words with frequencies below the threshold are low-frequency words, and the number of such words is accumulated to determine the number of low-frequency words. The low-frequency word coverage rate is obtained by dividing the counted number of low-frequency words by the total length of the word-level unit sequence, i.e., the total number of words contained in the query text. Low-frequency word coverage reflects the proportion of rare words, technical terms, or transliterated foreign words in the current query. The higher the value, the more terms in the query are rare in the general corpus.

[0022] The third step involves determining multidimensional feature vectors using word-level unit sequences and low-resource feature indicators. Based on these multidimensional feature vectors and a pre-trained intent classification model, the query intent type of the Lao query text is determined. The query intent type includes one of the following: lexical, semantic, entity, or mixed. The intent analysis results are then determined using the query intent type and low-resource feature indicators.

[0023] Specifically, word-level unit sequences cannot be directly used as numerical features input into classification models; therefore, they are vectorized. For example, the TF of each word in the word-level unit sequence is calculated. Alternatively, a lightweight word embedding model (such as a shallow neural network pre-trained for Lao) can be used to map the entire sequence into a dense vector of fixed dimensions. When using the TF-IDF weighted averaging method, a dictionary covering commonly used Lao words is constructed, with a dimension of 500. The inverse document frequency (IDF) of each word in the word-level unit sequence is calculated in the entire document database. Then, the TF-IDF vectors of all word terms in the sequence are weighted and averaged according to the frequency of word occurrence to obtain a fixed-dimensional vector representation of the query. This method does not rely on external word embedding models and avoids the problem of insufficient word vector training in low-resource languages.

[0024] The word segmentation ambiguity index, mixed language degree index, and low-frequency word coverage are used as three independent numerical features, concatenated with the aforementioned sequence vector to form a multi-dimensional feature vector that contains both semantic information and low-resource uncertainty indicators. This multi-dimensional feature vector is then input into a pre-trained intent classification model, which can be a support vector machine, random forest, or lightweight neural network. The training process uses a large number of Lao query samples with labeled intent types. For each sample, word-level unit sequences and low-resource feature indicators are extracted to generate a feature vector, and the model parameters are trained using supervised learning. The intent classification model outputs a probability distribution or directly outputs the predicted query intent type, which includes lexical, semantic, entity, and hybrid types. Lexical intents primarily rely on precise term matching, focusing on specific word matching, such as professional terminology queries. Semantic intents require deep semantic understanding, focusing on semantic comprehension, such as open-ended question answering. Entity intents are centered around specific named entities, such as specific person / place / organization names. Hybrid intents combine multiple characteristics. Finally, the query intent type output by the model is packaged with low resource characteristic indicators to determine the intent analysis results.

[0025] Step S102: Based on the intent analysis results, determine at least one target activation recall channel and the corresponding recall quantity limit parameter corresponding to the current query in the preset recall channel combination configuration table.

[0026] The recall channel includes one or more of the following: sparse term recall channel, dense vector recall channel, and entity-oriented recall channel; Based on the intent analysis results, in the preset recall channel combination configuration table, at least one target active recall channel and its corresponding recall quantity limit parameter are determined for the current query. Specifically, this includes: determining the query intent type and low-resource feature indicators in the intent analysis results; when the query intent type is lexical, activating the sparse term recall channel and determining the target active recall channel; when the query intent type is semantic, activating the dense vector recall channel and determining the target active recall channel; when the query intent type is entity, activating the entity-oriented recall channel and determining the target active recall channel; when the query intent type is mixed, activating the sparse term recall channel, the dense vector recall channel, and the entity-oriented recall channel and determining the target active recall channel; and based on the query intent type and low-resource feature indicators, obtaining the recall quantity limit parameter corresponding to each target active recall channel from the recall channel combination configuration table, wherein the recall quantity limit parameter is used to represent the maximum number of candidate documents output by each recall channel.

[0027] In one embodiment of this specification, the query intent type and low-resource characteristic indicators of the current Lao query text are determined. Then, based on the query intent type, a target is selected to activate the recall channel. The specific implementation process is as follows: When the query intent type is lexical, it indicates that the user's query is primarily for precise word matching, such as a technical term query or code number query. Since semantic vector retrieval and entity retrieval can easily introduce noisy documents with similar semantics but mismatched words in such queries, only the sparse term recall channel is activated, and the channel is activated only for those identified as targets. When the query intent type is semantic, it indicates that the user's query is primarily for semantic understanding, such as open-ended knowledge questions or descriptive questions. Under this intent type, term matching is difficult to capture synonymous expressions and sentence variations, so only the dense vector recall channel is activated. When the query intent type is entity, it indicates that the user's query is centered on a specific named entity, such as querying information about a person, a place, or an organization. Entity-oriented recall can accurately locate documents containing that entity through NER identification and entity linking, so only the entity-oriented recall channel is activated. When the query intent type is mixed, it means that the user query has multiple characteristics of words, semantics and entities. At this time, the sparse term recall channel, dense vector recall channel and entity-oriented recall channel are activated at the same time. All three channels are identified as the target activation channels to ensure comprehensive coverage of relevant information from multiple dimensions.

[0028] After determining the target activation recall channels, the recall quantity cap parameter corresponding to each target activation recall channel is obtained from the preset recall channel combination configuration table based on the query intent type and word segmentation ambiguity index. The recall quantity cap parameter for the sparse term recall channel is adjusted according to the word segmentation ambiguity index, where the word segmentation ambiguity index is positively correlated with the recall quantity cap parameter; the recall quantity cap parameter for the dense vector recall channel is adjusted according to the mixed language proficiency index, where the mixed language proficiency index is positively correlated with the recall quantity cap parameter; and the recall quantity cap parameter for the entity-oriented recall channel is adjusted according to the low-frequency word coverage rate, where the low-frequency word coverage rate is positively correlated with the recall quantity cap parameter.

[0029] Specifically, the recall channel combination configuration table is a predefined data structure with query intent type as the primary key. It also presets multiple threshold ranges and corresponding adjustment coefficients or incremental values ​​for each low-resource feature indicator. Using query intent type as the index, the basic recall quantity for each activated channel is read from the recall channel combination configuration table. In lexical queries, only the sparse term recall channel has a basic recall quantity; the basic recall quantities for the dense vector recall channel and the entity-oriented recall channel are set to zero. In semantic queries, only the dense vector recall channel has a basic recall quantity. In entity queries, only the entity-oriented recall channel has a basic recall quantity. In mixed queries, all three channels have their own basic recall quantities.

[0030] Then, the basic recall quantity for each channel is dynamically adjusted based on each low-resource characteristic indicator. In the sparse term recall channel, the word segmentation ambiguity index is compared with the preset first threshold range in the configuration table. If the word segmentation ambiguity index exceeds the first high threshold, it is determined that the current query term boundary ambiguity is severe and word-level matching is unreliable. In this case, the basic recall quantity needs to be multiplied by a first adjustment coefficient greater than one to expand the candidate document scale and compensate for missed recalls due to word segmentation errors. If the word segmentation ambiguity index is lower than the first low threshold, the basic recall quantity remains unchanged. If it is between the two, linear interpolation is used to determine the adjustment coefficient. In scenarios with high word segmentation ambiguity indices, the sparse term recall channel, by increasing the recall quantity, can effectively compensate for keyword loss caused by word boundary segmentation errors, allowing more segments that were originally excluded from the inverted index matching range due to inconsistent word segmentation to be included in the candidate set, thereby improving the recall coverage of relevant documents under fuzzy boundary queries.

[0031] Under the dense vector recall channel, the mixed language intensity index is compared with the second threshold range preset in the recall channel combination configuration table. When the mixed language intensity index exceeds the second high threshold, it is determined that there is frequent language switching in the query, and the single language vector model is prone to semantic breaks. Therefore, a second adjustment coefficient greater than one needs to be multiplied by the basic recall quantity to recall more potentially relevant documents in different language expressions. When the mixed language intensity index is below the second low threshold, the basic recall quantity remains unchanged; when it is between these thresholds, linear interpolation adjustment is used. In scenarios with high mixed language intensity, the dense vector recall channel, by increasing the recall quantity, can more fully cover semantically relevant documents in different language expressions, effectively alleviating the semantic break problem in cross-language queries.

[0032] Under the entity-oriented recall channel, the low-frequency word coverage rate is compared with the preset third threshold range in the configuration table. When the low-frequency word coverage rate exceeds the third high threshold, it is determined that the query contains a large number of rare words or unregistered entities, which are difficult for conventional named entity recognition models to accurately identify. Therefore, a third adjustment coefficient greater than one needs to be multiplied by the basic recall quantity to expand the candidate scale of entity recall, thereby covering more potentially relevant documents through transliteration fuzzy matching and knowledge graph neighborhood expansion. When the low-frequency word coverage rate is lower than the third low threshold, the basic recall quantity remains unchanged; when it is between these thresholds, linear interpolation adjustment is used. In scenarios with high low-frequency word coverage, the entity-oriented recall channel, by increasing the recall quantity, can provide more sufficient candidate space for transliteration fuzzy matching and knowledge graph neighborhood expansion, thereby improving the recall rate of documents related to rare entities. Through the above steps, the final recall quantity upper limit parameter for each target activation recall channel is determined, representing the maximum number of candidate documents output by this recall channel when performing a retrieval. For example, when calling the inverted index for BM25 retrieval, only the top N documents by score are returned, where N is the recall quantity upper limit parameter.

[0033] By selectively activating recall channels based on query intent type, the computational resource waste caused by performing full-channel retrieval on all queries is avoided. Lexical queries activate only the sparse term recall channel, saving computational overhead for vector encoding and entity recognition; semantic queries activate only the dense vector recall channel, avoiding inefficient term matching retrieval; and entity queries activate only the entity-oriented recall channel, achieving precise positioning. On-demand activation significantly reduces average retrieval latency. The recall depth of each channel is optimized according to different low-resource characteristics of Lao queries, maximizing recall coverage while ensuring retrieval accuracy. Dynamic scheduling and quantity control at the channel level are implemented before recall execution, ensuring retrieval quality and optimizing resource utilization, which is particularly suitable for low-resource language scenarios such as Lao, which has abundant low-frequency words, widespread language mixing, and frequent word segmentation ambiguity.

[0034] Step S103: Based on the target activation recall channel and the corresponding recall quantity limit parameter, determine the candidate document set and the corresponding document score for each target activation recall channel, so as to determine the fused unified candidate set through multiple candidate document sets and corresponding document scores.

[0035] After determining the target activation recall channel and the corresponding recall quantity limit parameter in step S102, the candidate document set and corresponding document score for each target activation recall channel are determined based on the target activation recall channel and the corresponding recall quantity limit parameter. Specifically, this includes: when the target activation recall channel includes the sparse term recall channel, obtaining the multi-granularity text unit sequence corresponding to the Lao query text, wherein the multi-granularity text unit sequence includes word-level unit sequence, syllable-level unit sequence, character-level n-gram unit sequence, and sub-word unit sequence; performing weighted BM25 retrieval based on the multi-granularity text unit sequence, and determining the dynamic weight of each granularity using the pre-obtained word segmentation ambiguity index, and determining the first candidate document set and the corresponding first... Document score; when the target activation recall channel includes the dense vector recall channel, the pre-acquired word-level unit sequence is mapped to the high-resource language semantic space to determine the query vector; using the query vector, an approximate nearest neighbor search is performed in the pre-generated document vector library to determine the second candidate document set and the corresponding second document score; when the target activation recall channel includes the entity-oriented recall channel, continuous syllable segments in the syllable-level unit sequence are enumerated and converted into transliteration codes to perform transliteration code matching in the entity tag inverted index to determine the matched candidate entity segments; based on the candidate entity segments, a one-hop neighborhood expansion is performed in the pre-constructed knowledge graph to determine the third candidate document set and the corresponding third document score.

[0036] Based on the multi-granularity text unit sequence, a weighted BM25 retrieval is performed, and the dynamic weight of each granularity is determined using the word segmentation ambiguity index. This process determines the first candidate document set and the corresponding first document score. Specifically, this includes: performing BM25 retrieval in the pre-constructed inverted index of each granularity to determine the temporary candidate document set and the initial BM25 score for each granularity; determining the dynamic weight of each granularity using the word segmentation ambiguity index, where the dynamic weight of word-level granularity is negatively correlated with the word segmentation ambiguity index, and the dynamic weights of syllable-level, character-level n-gram, and sub-word-level granularity are positively correlated with the word segmentation ambiguity index; performing a weighted summation of the initial BM25 scores for each granularity according to the dynamic weight of each granularity to determine the BM25 score for each temporary candidate document set; and determining the first candidate document set and the corresponding first document score for the sparse term recall channel according to the recall quantity upper limit parameter.

[0037] In existing hybrid retrieval technologies, sparse term recall channels typically employ the BM25 algorithm, using a word-level inverted index for keyword matching. However, single-granularity word-level BM25 retrieval cannot meet the needs of low-resource languages ​​like Lao, which are written continuously without spaces. Word segmentation ambiguity leads to frequent errors in word boundary segmentation. When a term in the query is incorrectly segmented due to a segmenter error, it cannot match the correct document term in the inverted index, resulting in the omission of relevant documents. Although some existing technologies use BM25 keyword retrieval, their retrieval granularity is limited to the word level, failing to consider the negative impact of word segmentation ambiguity on recall rates in low-resource language scenarios, and providing no compensation for segmentation uncertainty. Relying solely on word-level BM25 retrieval, in query scenarios with high word segmentation ambiguity, will result in the loss of a large number of semantically related but lexically inconsistent documents due to word boundary segmentation errors.

[0038] In one embodiment of this specification, when the target activation recall channel includes the sparse term recall channel, the specific implementation process is as follows: The process involves obtaining a multi-granularity text unit sequence corresponding to the Lao query text. This sequence includes word-level unit sequences, syllable-level unit sequences, character-level n-gram unit sequences, and sub-word unit sequences. The word-level unit sequence is obtained by segmenting the text at multiple levels of word boundaries and then taking the union of the first, second, and third segmentation sequences to remove duplicates; each element represents a word unit. The syllable-level unit sequence is obtained using a deterministic segmenter based on syllable rules, which is the third segmentation sequence; each element represents a syllable. Additionally, based on the Lao query text, a sliding window is used to extract continuous character segments to determine the character-level n-gram unit sequence. Specifically, a fixed-length character segment is continuously extracted from the Lao query text using a sliding window. Each segment consists of multiple consecutive characters, and the window length takes several preset values, such as 3, 4, 5, and 6. After collecting and removing duplicates from all the segments extracted at all window positions, the character-level n-gram unit sequence is obtained. The Lao query text is input into a pre-trained byte-pair encoding model to determine the sub-word unit sequence. The byte-pair encoding (BPE) model performs statistical learning on a large-scale unsupervised Lao corpus, merging frequently co-occurring character sequences into sub-word units and outputting the sub-word segmentation result of the query text. Specifically, raw Lao corpora from publicly available Lao news websites, social media texts, and government announcements are collected, removing HTML tags, special symbols, and excessively short texts (less than 10 characters). The SentencePiece tool is used for BPE training, with a vocabulary size of 8000 characters, employing a unigram language model mode, achieving a character coverage of 0.9995. During training, the algorithm starts at the character level, iteratively counting the most frequently co-occurring character pairs and merging them into new sub-word units until the vocabulary reaches the preset size. The resulting BPE model can segment any Lao text into a finite sequence of sub-word units.

[0039] BM25 retrieval is performed using a weighted algorithm based on multi-granularity text unit sequences. The dynamic weight of each granularity is determined by a pre-obtained word segmentation ambiguity index, thus identifying the first candidate document set and the corresponding first document score. The specific implementation method for this step is described below: The first step involves performing BM25 searches in the pre-built inverted indexes for each granularity to determine the temporary candidate document set and the initial BM25 score for each granularity. In the offline phase, for each document in the document library, a multi-granularity segmentation method identical to the query method is used to construct inverted indexes for each granularity, including word-level, syllable-level, character-level n-gram, and sub-word-level inverted indexes. Each inverted index stores a list of units to document identifiers for the corresponding granularity, along with their frequency and position information within the document. During the actual retrieval, the text unit sequence for each granularity of the query is input into the corresponding inverted index, and a BM25 search is performed. Within each granularity, each unit in the query unit sequence is traversed, and the document containing that unit is located in the inverted index. The contribution score of each document relative to that query unit is calculated using the BM25 formula. The contribution scores of all query units are summed to obtain the initial BM25 score for each document at that granularity. Then, the top M documents are extracted from high to low scores. Here, M is a large temporary number, such as all possible candidates or a preset upper limit. The temporary candidate document set corresponding to this granularity and the initial BM25 score of each document are determined. Here, the initial BM25 score refers to the initial BM25 score of each document in the temporary candidate document set.

[0040] The second step is to determine the dynamic weight of each granularity using the word segmentation ambiguity index. The dynamic weight of the word-level granularity is negatively correlated with the word segmentation ambiguity index. The higher the word segmentation ambiguity index, the more unreliable the word boundary segmentation is, and the lower the credibility of word-level matching. Therefore, the weight of the word-level granularity is lower. The dynamic weights of the syllable-level granularity, character-level n-gram granularity, and sub-word granularity are positively correlated with the word segmentation ambiguity index. The syllable-level granularity, character-level n-gram granularity, and sub-word granularity do not depend on word boundary decisions. The higher the word segmentation ambiguity index, the higher the weight of these fine-grained units, which can serve as an effective compensation matching method in the event of word segmentation ambiguity.

[0041] In one example, the weight calculation example for determining the dynamic weight of each granularity using the word segmentation ambiguity index is as follows: Let the word segmentation ambiguity index be... d Its value range is [0,1], where 0 indicates that the three segmentation results are completely consistent and unambiguous, and 1 indicates that the three segmentation results are completely inconsistent with the greatest ambiguity. The dynamic weight of the word-level granularity is set to 1-d, that is, the higher the segmentation ambiguity index, the lower the weight of the word-level granularity is considered. When d=0, the word-level weight is 1, and when d=1, the word-level granularity weight is 0. The dynamic weights of the syllable-level granularity, character-level n-gram granularity, and sub-word granularity are all set to d / 3, that is, the weights of the three granularities are equal and the sum is d. At this time, the sum of the weights of all four granularities (1 d) + d / 3 + d / 3 + d / 3 = 1, satisfying the normalization requirement. When word segmentation is unambiguous, it relies entirely on word-level BM25 retrieval; when word segmentation ambiguity reaches its maximum, word-level matching is completely abandoned, and syllable, character n-gram, and sub-word granularities equally bear all the weights to jointly compensate for the recall loss caused by word boundary errors. For cases where the word segmentation ambiguity index is at an intermediate value, the weights transition linearly, ensuring the continuity and smoothness of the retrieval strategy.

[0042] The third step is to perform a weighted summation of the initial BM25 scores at each granularity according to the dynamic weight of each granularity, and determine the BM25 score of each temporary candidate document set. Here, the BM25 score of the temporary candidate document set refers to the score of each temporary candidate document in the temporary candidate document set.

[0043] The fourth step is to determine the first candidate document set and the corresponding first document score for the sparse term recall channel according to the recall limit parameter N1. After completing the weighted summation of all temporary candidate documents in the temporary candidate document set, the top N1 documents with the highest overall BM25 score are selected according to the recall limit parameter N1, i.e., the recall limit parameter N1 corresponding to the sparse term recall channel. These documents and their corresponding overall BM25 scores are determined as the first candidate document set and the corresponding first document score. The first document score refers to the document score of each first candidate document in the first candidate document set.

[0044] By introducing four granularities of text units—word-level, syllable-level, character-level n-gram, and sub-word-level—and constructing an inverted index independently for each granularity, the negative impact of word segmentation ambiguity caused by the absence of spaces in Lao writing on recall rate is fundamentally alleviated. When the word segmentation ambiguity index is high, the weight of word-level granularity is automatically reduced while the weights of syllable-level, character-level n-gram, and sub-word granularity are simultaneously increased. This shifts the search focus from unreliable word boundary matching to fine-grained unit matching that does not rely on word segmentation, effectively compensating for the lexical gap caused by word segmentation errors. Compared to schemes that only use conventional BM25 keyword retrieval without considering word segmentation ambiguity compensation, the technical solution in this specification uses multi-granularity dynamic weighted BM25 based on the word segmentation ambiguity index to achieve adaptive robust retrieval for Lao word segmentation uncertainty. This significantly improves recall coverage in scenarios with fuzzy boundaries, while ensuring no loss of search accuracy in scenarios with consistent word segmentation, providing a rich and reliable source of candidate documents for subsequent fusion and ranking.

[0045] When the target's activation recall channel includes the dense vector recall channel, the specific implementation process is as follows: The pre-stored word-level unit sequence is obtained and input into the query-side encoder of the cross-lingual knowledge distillation alignment model. The cross-lingual knowledge distillation alignment model is a dual-encoder model trained using knowledge distillation techniques. The training process is as follows: Utilizing parallel corpora such as Chinese-Lao and English-Lao, for example, aligned sentence pairs from publicly available government documents and news websites, the semantic knowledge of the pre-trained teacher model in the high-resource language is distilled into the Lao student model. Here, the teacher model can be a Chinese BERT or English BERT trained on a large-scale corpus, enabling the student model to map the input Lao text to the semantic space of the high-resource language where the teacher model resides. That is, the output query vector is spatially proximate to the high-resource language text vector with the same semantics. The teacher model uses a Chinese BERT-base model pre-trained on a large Chinese corpus (such as Chinese Wikipedia and news corpora). This model contains a 12-layer Transformer encoder, a 768-dimensional hidden layer, and 12 attention heads. The student model uses a Lao BERT model with the same architecture, and its parameters are randomly initialized. Parallel corpora consist of sentence pairs from Chinese-Lao bilingual news websites. After cleaning, deduplication, and length filtering (retaining sentence pairs with a source-to-target language length ratio between 0.5 and 2.0), the data is used for training. The distillation loss function comprises three parts: prediction layer distillation loss (mean squared error between the output distribution of the student model and the teacher model); intermediate layer distillation loss (cosine similarity loss between the hidden states of each layer in the student model and the corresponding hidden states in the teacher model); and contrastive learning loss (maximizing vector similarity for positive sample pairs (aligned Chinese-Lao sentences) and minimizing vector similarity for negative sample pairs (random combinations of non-aligned sentences) within the same batch). The three losses are weighted and summed to obtain the total loss. Training uses the AdamW optimizer with a learning rate of 3e-5, a batch size of 64, and five training epochs. After training, the query-side encoder of the student model can map the input Lao text to the Chinese semantic space of the teacher model. The document-side encoder shares the same network structure and parameters as the query-side encoder. The query-side encoder of the dual-encoder model outputs a dense vector of fixed dimensions, which is the query vector mapped to the high-resource language semantic space.

[0046] It is important to clarify that, in the offline phase, each document in the document library is pre-processed using the same Lao word segmentation and multi-level word boundary segmentation methods to extract word-level unit sequences. These sequences are then input into the document-side encoder of the cross-language knowledge distillation alignment model to generate document vectors for each document. Here, the document-side encoder and the query-side encoder have the same structure and share parameters. All document vectors, along with their corresponding document identifiers and content, are stored in a document vector library. This document vector library can be a vector database based on an HNSW graph index to support efficient approximate nearest neighbor search.

[0047] During retrieval, the query vector is used as input, and an approximate nearest neighbor search is performed in a pre-generated document vector database. The vector database uses cosine similarity as a distance metric, calculating the cosine similarity score between the query vector and each document vector in the database. This score ranges from -1 to 1, with higher values ​​indicating greater semantic relevance. The HNSW index structure returns the top M documents with the highest similarity in sublinear time complexity, where M is the number of temporary candidates. Here, M is typically much larger than the upper limit parameter for subsequent recall. After obtaining the identifiers of these documents and their corresponding cosine similarity scores from the vector database, the top N2 documents with the highest similarity scores are selected according to the upper limit parameter N2 for the dense vector recall channel. These documents and their corresponding cosine similarity scores are used as the second candidate document set and their corresponding second document scores. Here, the second document score refers to the cosine similarity score of each second candidate document in the second candidate document set.

[0048] By leveraging the high discriminative power and rich semantic information of the high-resource language vector space, the accuracy of vector representations for low-frequency words, specialized terms, and complex semantic queries is significantly improved, thereby enhancing the recall precision and coverage of semantically relevant documents. Simultaneously, by pre-constructing a document vector library and employing HNSW indexing for approximate nearest neighbor search, high retrieval efficiency is ensured. This allows for the selection of semantically relevant candidate documents from a large-scale document library within milliseconds, providing a high-quality semantic candidate set for downstream fusion and ranking. This approach is particularly suitable for low-resource language scenarios like Lao, which lack large-scale, high-quality annotated corpora.

[0049] When the target activates the recall channel, including the entity-specific recall channel, the specific implementation process is as follows: The process involves obtaining a syllable-level unit sequence, enumerating all consecutive syllable segments within that sequence, with each segment's length ranging from a preset minimum to a maximum length (e.g., 2 to 6 syllables), thus enumerating all possible syllable combinations that could constitute an entity name. Each enumerated candidate entity segment is then converted into a transliteration code. The conversion rule involves extracting the consonant symbols from each syllable in the segment (ignoring vowel and tone marks) and concatenating them sequentially to form the encoded string. It's worth noting that an entity tag inverted index is pre-built offline. This index stores all entity names extracted from the document library and their corresponding standard transliterations. Each entity entry is also associated with a unique node identifier for that entity in the knowledge graph and a list of documents containing that entity.

[0050] The knowledge graph is constructed as follows: Entities (including names of people, places, organizations, events, etc.) are extracted from Lao language encyclopedia websites, public databases, and news corpora. Initial extraction is performed using named entity recognition tools, followed by expert verification to obtain an entity list. Pre-defined relational trigger word patterns (such as located at, belonging to, cooperating with, born in, etc.) are used to extract relationships between entities from the text. Each relation entry includes a head entity, a tail entity, and a relation type. Entity links employ a disambiguation algorithm based on vector similarity and contextual features, linking entity references appearing in the embodiments of this specification to unique nodes in the knowledge graph. The knowledge graph is stored in a graph database. Each node contains an entity identifier, entity name, entity type, and a list of associated document identifiers. Each edge contains a relation type and a confidence score.

[0051] During retrieval, the transliterated encoding of each candidate entity fragment is matched against the entity tag inverted index. First, an exact match is performed, checking if a standard transliterated encoding identical to the candidate encoding exists in the index. If the exact match results are less than a preset threshold, a fuzzy match with an edit distance less than the preset threshold is further performed to cover spelling variations. All successfully matched candidate entity fragments and their associated documents are identified as directly related documents, with each directly related document having a base score set to a first preset value (e.g., 1.0). Then, based on the successfully matched entity nodes, a one-hop neighborhood expansion is performed in a pre-constructed knowledge graph. The knowledge graph is a directed graph storing relationships between entities; each node represents an entity, and edges represent relationships between entities, such as belonging to, cooperating with, or located at. Starting from a successfully matched entity node, all directly connected adjacent nodes (i.e., all nodes within a one-hop range) are traversed, and the documents associated with these adjacent nodes are obtained as expanded documents. Each expanded document has a base score set to a second preset value, lower than the first preset value for directly related documents (e.g., 0.5), to reflect that indirectly related documents are less relevant than directly related documents. The directly related documents and extended documents are merged, sorted in descending order of the base score, and the top N3 documents with the highest scores are selected according to the maximum recall quantity parameter N3 corresponding to the entity-oriented recall channel. These documents and their corresponding base scores are determined as the third candidate document set and the corresponding third document score. In other words, the third document score refers to the score corresponding to each third candidate document in the third candidate document set.

[0052] In the embodiments described above, by enumerating all continuous syllable segments in the syllable-level unit sequence and converting them into transliteration codes, entity segments that may exist in Lao queries can be captured in a rule-based manner without relying on a complete named entity recognition model, effectively avoiding the problem of low recognition accuracy of low-resource Lao NER models. By introducing exact matching and edit distance fuzzy matching of transliteration codes, different Lao spelling variants of the same foreign word can be recalled, significantly improving the recall coverage of transliterated proper nouns. Through one-hop neighborhood expansion of the knowledge graph, not only can documents directly related to the query entity be recalled, but also documents associated with other entities that are related to the query entity (such as belonging to the same organization or located in the same region), thereby expanding the information richness of candidate documents and returning results with more dimensions related to the semantics of the query entity.

[0053] After determining the candidate document set and corresponding document score for each target activation recall channel, if there is only one target activation recall channel, the document scores corresponding to each candidate document in the candidate document set are sorted from high to low, and the top K documents are truncated according to the preset final output quantity parameter to determine the unified candidate set output to the downstream fine ranking module.

[0054] When multiple target activation recall channels exist, a unified candidate set is determined by using multiple candidate document sets and their corresponding document scores. Specifically, this includes: normalizing the document scores for each candidate document set to map each score to a preset numerical range, thus determining the normalized score for each candidate document set; obtaining the word segmentation ambiguity index, mixed language degree index, and low-frequency word coverage from the intent analysis results; determining the channel confidence weight for each recall channel using the word segmentation ambiguity index, the mixed language degree index, and the low-frequency word coverage; determining the fusion score for each candidate document set using the normalized score and the channel confidence weight for the corresponding recall channel; and using this fusion score to determine the unified candidate set.

[0055] When multiple target activation recall channels exist, the specific implementation steps for determining a unified candidate set by using multiple candidate document sets and their corresponding document scores are as follows: First, the document scores corresponding to each candidate document set are normalized to map each document score to a preset numerical range, thereby determining the normalized score of each candidate document set.

[0056] The relevance scores output by different recall channels fall into different dimensions and distribution ranges. The BM25 score of the sparse term recall channel is affected by factors such as document length and term frequency distribution, and its value range is usually between zero and tens. The cosine similarity score of the dense vector recall channel ranges from -1 to 1. The entity matching score of the entity-oriented recall channel is a discrete preset value, such as 1 for directly associated documents and 0.5 for indirectly associated documents. When multiple recall channels are activated, if these scores are directly weighted and summed, the score with the larger dimension will dominate the fusion result, causing high-quality documents from other channels to be incorrectly suppressed due to differences in score scale. Therefore, the document scores corresponding to the candidate documents in each candidate document set are first normalized to map each document score to a preset numerical range, usually [0,1], to determine the normalized score of each candidate document in each candidate document set.

[0057] In one example, a min-max normalization method can be used. For each channel's candidate document set, the minimum and maximum scores of all candidate documents in that channel are found. Then, the original score of each document is subtracted from the minimum score, and the result is divided by the difference between the maximum and minimum scores to obtain the normalized score. If all document scores in that channel are the same (i.e., the maximum score equals the minimum score), then all normalized scores are set to a preset constant, such as 0.5. After normalization, the normalized score for each document in each candidate document set is obtained.

[0058] Second, obtain the word segmentation ambiguity index, mixed language intensity index, and low-frequency word coverage from the intent analysis results. Then, determine the channel confidence weight for each recall channel based on these metrics. The specific implementation method is as follows: Based on the word segmentation ambiguity index, the first confidence weight for the sparse term recall channel is determined. This first confidence weight is negatively correlated with the word segmentation ambiguity index; that is, the higher the word segmentation ambiguity index, the lower the reliability of term matching, and the smaller the first confidence weight. Based on the mixed language degree index, the second confidence weight for the dense vector recall channel is determined. This second confidence weight is positively correlated with the mixed language degree index; that is, the higher the mixed language degree, the more the cross-language query relies on the generalization ability of the semantic vector model, and the larger the second confidence weight. Through low-frequency word coverage, the third confidence weight for the entity-oriented recall channel is determined. This third confidence weight is negatively correlated with low-frequency word coverage; that is, the higher the low-frequency word coverage, the more rare entities are included in the query, the lower the accuracy of named entity recognition and entity linking, and the smaller the third confidence weight.

[0059] In one example, the weight values ​​can be calculated using a linear function. The first confidence weight is the first base weight multiplied by (1 minus the word segmentation ambiguity index), the second confidence weight is the second base weight multiplied by (1 plus the mixed language degree index), and the third confidence weight is the third base weight multiplied by (1 minus the low-frequency word coverage rate). For activated recall channels, the confidence weights of each channel are divided by their sum to obtain the normalized final channel confidence weights for each channel, ensuring that the sum of the final channel confidence weights for all activated channels is 1. For inactive recall channels, the confidence weights of inactive recall channels are set to zero, and then the confidence weights of activated recall channels are normalized to obtain the normalized final channel confidence weights for each channel.

[0060] Third, the fusion score for each document in each candidate document set is calculated by weighting and summing the normalized score of each candidate document set with the channel confidence weight of the corresponding recall channel. It should be noted that if multiple recall channels are activated, there are multiple candidate document sets, and each candidate document set contains at least one candidate document. If a document appears only in a single channel (i.e., belongs only to a single candidate document set), the fusion score is the normalized score of that channel multiplied by the channel confidence weight. If a document appears in multiple channels (i.e., in multiple candidate document sets), the fusion score is the sum of the normalized scores of each channel multiplied by the corresponding channel confidence weight. All documents appearing in at least one candidate document set are sorted from highest to lowest fusion score, and the top K documents are truncated according to a preset final output quantity parameter to determine the unified fusion candidate set for output to the downstream fine-ranking module.

[0061] By performing min-max normalization on the raw scores of each channel, the BM25 score, cosine similarity score, and entity matching score are mapped to a unified numerical range, eliminating the interference of different units on the weighted fusion. This allows the contributions of each channel to be compared and superimposed on the same scale, avoiding the problem of unit dominance. By introducing a word segmentation ambiguity index, a mixed language intensity index, and low-frequency word coverage to drive the confidence weights of the three channels respectively, dynamic adaptive adjustment of the fusion weights is achieved. When the word segmentation ambiguity index is high, the weight of the sparse term recall channel is automatically reduced to decrease noise introduced by word boundary errors. When the mixed language intensity index is high, the weight of the dense vector recall channel is automatically increased to leverage the generalization advantage of the semantic vector model in multilingual mixed queries. When the low-frequency word coverage is high, the weight of the entity-oriented recall channel is automatically reduced to avoid introducing low-quality candidates due to inaccurate identification of rare entities.

[0062] The above-described at least one technical solution used in the embodiments of this specification can achieve the following beneficial effects: By introducing multi-dimensional query intent analysis, the system proactively identifies the semantic characteristics and low-resource features of user queries, avoiding the waste of computational resources caused by indiscriminate full-channel retrieval. Based on the intent analysis results, it dynamically determines the set of recall channels to be activated for the current query from a pre-defined recall channel combination configuration table. This allows for the selection of the most suitable recall path for lexical, semantic, entity, and mixed queries, reducing invalid retrieval overhead while ensuring recall quality. Simultaneously, based on the intent analysis results, a corresponding recall quantity limit parameter is configured for each activated recall channel, enabling dynamic adjustment of the candidate document truncation quantity for each recall channel according to query information. This avoids over-recall in simple queries and ensures sufficient coverage in complex queries. By executing each activated recall channel in parallel to obtain candidate document sets and corresponding document scores, the system leverages the complementary advantages of sparse term matching, dense semantic matching, and entity-oriented retrieval. Finally, a unified candidate set is generated through score fusion, effectively solving the coverage blind spot problem of a single recall strategy in low-resource language scenarios. This significantly improves the coverage rate and stability of the candidate set, while optimizing computational resource utilization.

[0063] This specification also provides an intelligent recall strategy scheduling system for Lao, comprising an access interface, a query preprocessing module, a query intent analysis module, a recall strategy scheduling module, a multi-path recall execution module (including sparse term recall channel, dense vector recall channel, and entity-oriented recall channel), a recall result fusion module, and a data resource layer. The components and their connections are described below: The access interface consists of a query input interface and a candidate set output interface. The query input interface receives the original Lao query submitted by the upper-layer application and passes it to the query preprocessing module. The candidate set output interface passes the unified candidate set (including a document list and fusion score) generated by the recall result fusion module to the downstream fine-ranking module. The query preprocessing module receives the original Lao query and sequentially performs Unicode encoding normalization, Lao word segmentation, keyword extraction, and named entity pre-identification (preliminarily marking proper nouns such as personal names, place names, and organization names that may exist in the query), outputting a normalized query sequence and preliminary features, which is connected to the query intent analysis module. The query intent analysis module performs independent intent dimension analysis on the normalized query, aggregates and outputs an intent analysis result vector, which is connected to the recall strategy scheduling module. The recall strategy scheduling module internally contains a recall channel configuration decision unit. This unit maintains a mapping rule table from intent type to recall channel combination, receives the intent analysis results, and outputs the channel combination to be activated for this query and the TopK parameter configuration for each channel, which are connected to the control interface of each activated recall channel.

[0064] The multi-path recall execution module consists of three parallel recall channels: The sparse term recall channel contains an inverted index retrieval engine that receives a normalized query term list, performs BM25 score calculation and TopK document retrieval in the inverted index storage of the data resource layer, and outputs a document set and corresponding BM25 scores R_s. The dense vector recall channel contains a dual-tower encoder query-side encoder and a document vector database ANN retrieval interface. The query-side encoder receives a normalized query sequence and generates a query vector v_q. The ANN retrieval interface retrieves the TopK documents with the highest cosine similarity in the document vector database of the data resource layer, and outputs a document set and corresponding cosine similarity scores R_d. The entity-oriented recall channel contains a Lao named entity recognizer (NER recognizer) and an entity tag inverted index retrieval interface. The NER recognizer receives a normalized query sequence, identifies named entities, and extracts entity tags. The entity tag inverted index retrieval interface retrieves documents containing the corresponding entity tags in the entity tag inverted index of the data resource layer, and outputs a document set and corresponding entity matching scores R_e. The recall result fusion module receives multiple document sets and corresponding scores from each activated recall channel, and performs the following operations in sequence: score normalization (converting scores from each channel to the [0,1] interval), document deduplication (merging the same document from different channels based on the document's unique identifier), multi-channel score weighted fusion (weighting and summing the normalized scores according to channel weights to generate a fused score), TopK descending order truncation, and outputting a unified candidate set to the candidate set output interface. The data resource layer stores the four types of indexes and configuration resources upon which the system relies for operation: inverted index storage (for sparse term recall channels), document vector database (for dense vector recall channels), entity tag inverted index (for entity-oriented recall channels), and channel configuration rule table (for the recall strategy scheduling module to query).

[0065] This specification also provides an intelligent recall strategy scheduling device for Lao language, such as... Figure 2 As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method.

[0066] This specification also provides a non-volatile computer storage medium storing computer-executable instructions configured to execute the above-described method.

[0067] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0068] The above description is merely one or more embodiments of this specification and is not intended to limit this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of this specification.

Claims

1. A Lao language intelligent recall strategy scheduling method, characterized in that, The method includes: Receive Lao query text input by the user, perform multi-dimensional query intent analysis on the Lao query text, and determine the intent analysis result of the Lao query text; Based on the intent analysis results, in the preset recall channel combination configuration table, at least one target activation recall channel and the corresponding recall quantity limit parameter corresponding to the current query are determined. The recall channel includes any one or more of sparse term recall channels, dense vector recall channels, and entity-oriented recall channels. Based on the target activation recall channel and the corresponding recall quantity limit parameter, a candidate document set and a corresponding document score are determined for each target activation recall channel, so as to determine a unified candidate set by using multiple candidate document sets and corresponding document scores; A multi-dimensional query intent analysis is performed on the Lao query text to determine the intent analysis results, specifically including: The Lao query text is segmented into multi-level word boundaries to obtain a first segmentation sequence, a second segmentation sequence, and a third segmentation sequence. The multi-level word boundary segmentation includes a dictionary-based maximum matching segmenter, a conditional random field-based statistical segmenter, and a syllable rule-based deterministic segmenter. The word-level unit sequence is determined by the union of the first segmentation sequence, the second segmentation sequence, and the third segmentation sequence; Based on the word-level unit sequence, the Lao query text is evaluated for low-resource features, and corresponding low-resource feature indicators are determined. The low-resource feature indicators include word segmentation ambiguity index, mixed language degree index, and low-frequency word coverage. A multidimensional feature vector is determined using the word-level unit sequence and the low-resource feature index. Based on the multidimensional feature vector and a pre-trained intent classification model, the query intent type of the Lao query text is determined. The query intent type includes any one of lexical, semantic, entity, and mixed types. The intent analysis result is determined based on the query intent type and the low resource characteristic index; Based on the target activation and recall channels and the corresponding recall limit parameters, the candidate document set and corresponding document score for each target activation and recall channel are determined, specifically including: When the target activation recall channel includes the sparse term recall channel, the multi-granularity text unit sequence corresponding to the Lao query text is obtained, wherein the multi-granularity text unit sequence includes word-level unit sequence, syllable-level unit sequence, character-level n-gram unit sequence and sub-word unit sequence; Weighted BM25 retrieval is performed based on the multi-granularity text unit sequence, and the dynamic weight of each granularity is determined by the pre-acquired word segmentation ambiguity index to determine the first candidate document set and the corresponding first document score. When the target activation recall channel includes the dense vector recall channel, the pre-acquired word-level unit sequence is mapped to the high-resource language semantic space to determine the query vector. Using the query vector, an approximate nearest neighbor search is performed in a pre-generated document vector library to determine the second candidate document set and the corresponding second document score; When the target activation recall channel includes the entity-oriented recall channel, the continuous syllable segments in the syllable-level unit sequence are enumerated and converted into transliteration codes, so as to perform transliteration code matching in the entity label inverted index and determine the matched candidate entity segments. Based on the candidate entity fragments, a one-hop neighborhood expansion is performed in the pre-constructed knowledge graph to determine the third candidate document set and the corresponding third document score.

2. The intelligent recall strategy scheduling method for Lao language according to claim 1, characterized in that, Based on the word-level unit sequence, the Lao query text is evaluated for low-resource features to determine the corresponding low-resource feature indicators, specifically including: Calculate the Jaccard similarity coefficient between any two segmentation sequences based on the first segmentation sequence, the second segmentation sequence, and the third segmentation sequence; The word segmentation ambiguity index corresponding to the Lao language query text is determined by the arithmetic mean of the multiple Jaccard similarity coefficients. Each word item is extracted from the word-level unit sequence and input into a pre-trained language classifier to determine the language label corresponding to each word item; The number of times the language tag is switched in the word-level unit sequence is counted, and the mixed language degree index corresponding to the Lao query text is determined based on the ratio of the number of switches to the sequence length of the word-level unit sequence; Each word in the word-level unit sequence is compared with a pre-constructed Lao word frequency table, and the number of low-frequency words in the word-level unit sequence that are below a preset threshold is counted. The low-frequency word coverage rate corresponding to the Lao query text is determined based on the ratio of the number of low-frequency words to the sequence length of the word-level unit sequence.

3. The intelligent recall strategy scheduling method for Lao language according to claim 1, characterized in that, Based on the intent analysis results, in the preset recall channel combination configuration table, at least one target activation recall channel and the corresponding recall quantity limit parameter corresponding to the current query are determined, specifically including: Determine the query intent type and low resource characteristic indicators in the intent analysis results; When the query intent type is lexical, the sparse term recall channel is activated, and the target is determined to activate the recall channel. When the query intent type is semantic, the dense vector recall channel is activated, and the target is determined to activate the recall channel. When the query intent type is entity type, the entity-oriented recall channel is activated, and the target is determined to activate the recall channel. When the query intent type is mixed, the sparse term recall channel, the dense vector recall channel, and the entity-oriented recall channel are activated, and the target activation recall channel is determined. Based on the query intent type and the low resource characteristic index, the recall quantity limit parameter corresponding to each target activated recall channel is obtained from the recall channel combination configuration table, wherein the recall quantity limit parameter is used to represent the maximum number of candidate documents output by each recall channel.

4. The intelligent recall strategy scheduling method for Lao language according to claim 1, characterized in that, Weighted BM25 retrieval is performed based on the multi-granularity text unit sequence, and the dynamic weight of each granularity is determined using the word segmentation ambiguity index to determine the first candidate document set and the corresponding first document score, specifically including: Perform BM25 retrieval in the pre-built inverted index at each granularity to determine a temporary candidate document set and an initial BM25 score for each granularity; The dynamic weight of each granularity is determined by the word segmentation ambiguity index, wherein the dynamic weight of the word-level granularity is negatively correlated with the word segmentation ambiguity index, and the dynamic weights of the syllable-level granularity, character-level n-gram granularity, and sub-word granularity are positively correlated with the word segmentation ambiguity index. According to the dynamic weight of each granularity, the initial BM25 score under each granularity is weighted and summed to determine the BM25 score of each temporary candidate document set; Based on the upper limit parameter for the number of recalls, determine the first candidate document set and the corresponding first document score for the sparse term recall channel.

5. The intelligent recall strategy scheduling method for Lao language according to claim 1, characterized in that, By using multiple candidate document sets and their corresponding document scores, a unified candidate set for fusion is determined, specifically including: The document scores corresponding to each candidate document set are normalized to map each document score to a preset numerical range, thereby determining the normalized score of each candidate document set. Obtain the word segmentation ambiguity index, mixed language degree index, and low-frequency word coverage from the intent analysis results; The channel confidence weight corresponding to each recall channel is determined by the word segmentation ambiguity index, the mixed language degree index, and the low-frequency word coverage. The fusion score of each candidate document set is determined by the normalized score of each candidate document set and the channel confidence weight of the recall channel corresponding to the candidate document set. The fusion score is then used to determine the unified fusion candidate set.

6. The intelligent recall strategy scheduling method for Lao language according to claim 5, characterized in that, The confidence weight of each recall channel is determined by the word segmentation ambiguity index, the mixed language degree index, and the low-frequency word coverage rate, specifically including: Based on the word segmentation ambiguity index, the first confidence weight corresponding to the sparse term recall channel is determined, wherein the first confidence weight is negatively correlated with the word segmentation ambiguity index; Based on the mixed language proficiency index, a second confidence weight is determined for the dense vector recall channel, wherein the second confidence weight is positively correlated with the mixed language proficiency index; The third confidence weight corresponding to the entity targeted recall channel is determined based on the low-frequency word coverage rate, wherein the third confidence weight is negatively correlated with the low-frequency word coverage rate.

7. A Lao-language intelligent recall strategy scheduling device, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-6.

8. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured to perform the method as described in any one of claims 1-6.

Images