Vector retrieval method based on language model

Abstract

The application relates to the technical field of cross-language vector retrieval, in particular to a vector retrieval method based on a language model. The application adopts a mixed English query text; identifies Chinese segments and English segments, and eliminates stop words; calls a Chinese large-text vector conversion model to generate a Chinese segment vector after optimization; calls an English large-text vector conversion model to generate an English segment vector after optimization; determines a primary index and a secondary index based on the proportion of Chinese segments; calls the primary index for retrieval to obtain a first candidate vector set; calls the secondary index for verification to obtain a second candidate vector set; calculates a plurality of index correction coefficients corresponding to the candidate vectors to calculate a correction similarity; sorts the plurality of candidate vectors based on the correction similarity to generate a first output result; and determines whether a screening mechanism is triggered based on the number of candidate vectors to generate a second output result. The application significantly improves the semantic precision of vector matching.

Description

Technical Field

[0001] This invention relates to the field of cross-language vector retrieval technology, and in particular to a vector retrieval method based on a language model. Background Technology

[0002] With the deepening of globalized information exchange, the demand for cross-language mixed query text in academic, medical, and business scenarios continues to grow. Simultaneously, the rapid development of pre-trained language models has provided the technological foundation for text semantic vectorization, making vector retrieval a core means of cross-language information acquisition. However, existing cross-language vector retrieval methods are insufficiently adapted to the inherent characteristics of different languages ​​and fail to fully consider the differences in semantic carriers between languages. This results in generated text vectors that struggle to accurately capture the core semantics of specific professional fields, thus affecting the accuracy of semantic matching in cross-language scenarios and failing to adequately meet users' needs for precise cross-language information retrieval.

[0003] Chinese Patent Publication No. CN106372187A discloses a cross-language retrieval method for big data. This invention discloses a cross-language retrieval method for big data that uses Chinese and English Wikipedia entries and the cross-language relationships between entries to construct a bilingual word vector model. This bilingual word vector model is then used to translate the query, and finally, a new query is constructed based on candidate translations to perform the retrieval. The cross-language retrieval model takes the source language query vector as input and outputs the similarity between the target language documents and the query vector. During the query translation process, the results of canonical association analysis are used. This invention, from the perspective of automatic query translation, utilizes the semantic similarity characteristics of documents in different languages ​​to find a shared semantic space between the two languages. The query is then semantically paraphrased within this shared space, thereby achieving automatic query translation.

[0004] Therefore, it is evident that the existing technology has the following problems:

[0005] Traditional cross-language retrieval relies on a single encoding model, which cannot take into account the differences in semantic characteristics between Chinese and English, resulting in a decrease in the accuracy of mixed text vector matching. Summary of the Invention

[0006] To address this, the present invention provides a vector retrieval method based on a language model, which overcomes the limitations of traditional cross-language retrieval in the prior art, which relies on a single encoding model and cannot take into account the differences in semantic characteristics between Chinese and English, resulting in a decrease in the accuracy of mixed text vector matching.

[0007] To achieve the above objectives, this invention provides a vector retrieval method based on a language model, comprising:

[0008] Collect mixed Chinese and English query text input by the target user, and identify Chinese and English segments in the mixed Chinese and English query text;

[0009] Based on the stop word lists for Chinese and English, stop words were removed from the Chinese and English segments respectively;

[0010] Based on the Chinese segment after stop word removal, a text vector conversion is performed to generate a Chinese segment vector with optimized weights for radical features. Based on the English segment after stop word removal, a text vector conversion is performed to generate an English segment vector with optimized weights for root and affix features.

[0011] Based on the comparison results of the proportion of Chinese fragments and the preset threshold, the primary and secondary indexes for hybrid retrieval are determined;

[0012] Based on the Chinese and English fragment vectors and their corresponding attention weights, the main index of the corresponding semantic space is called to retrieve the first candidate vector set.

[0013] Based on the first candidate vector set, the language fragment vector corresponding to the secondary index is used to call the secondary index for verification, and the second candidate vector set is obtained.

[0014] Based on the similarity scores corresponding to the first and second candidate vectors, calculate several index correction coefficients corresponding to the candidate vectors.

[0015] The corresponding corrected similarity is calculated based on several index correction coefficients, and the candidate vectors are sorted to generate the first output result.

[0016] The number of candidate vectors determines whether a preset threshold is exceeded, which in turn determines whether a filtering mechanism should be triggered to determine whether a second output result should be generated.

[0017] Furthermore, the process of generating Chinese segment vectors after optimizing the weights of radical features includes,

[0018] The Chinese segment is processed by word segmentation and character splitting to extract the radicals of the core characters;

[0019] Calculate the semantic similarity between the core character and high-frequency characters with the same radical, and calculate the mean.

[0020] Obtain the basic weight of the radical;

[0021] Optimized weights are calculated based on the mean semantic similarity of the core characters and the basic weights of the corresponding radicals.

[0022] Obtain the radical vectors corresponding to the core characters and several word segmentation vectors;

[0023] Based on several word segmentation vectors, the radical vectors corresponding to core characters, and the optimized weights of the radicals corresponding to core characters, optimized Chinese fragment vectors are generated.

[0024] Furthermore, the process of generating English fragment vectors after optimizing the weights of root and affix features includes,

[0025] The English fragment is segmented and decomposed into word roots to extract core word roots and calculate the semantic contribution of the core word roots.

[0026] Obtain the basic weight of the word root;

[0027] Calculate the optimized weights based on the semantic contribution of the core word roots and the basic weights of the corresponding word roots;

[0028] Obtain the semantic vectors corresponding to the core word roots and several word segmentation vectors;

[0029] Based on several word segmentation vectors, semantic vectors corresponding to core word roots, and optimized weights corresponding to core word roots, optimized English fragment vectors are generated.

[0030] Furthermore, based on the comparison between the proportion of Chinese fragments and preset thresholds, the process of determining the primary and secondary indexes in the hybrid retrieval strategy includes:

[0031] Calculate the proportion of Chinese segments based on the number of Chinese and English segments after stop word removal is completed;

[0032] If the proportion of Chinese characters is greater than or equal to a preset threshold, then the primary index is determined to be the Chinese semantic space and the English semantic space, and the secondary index is the English semantic space.

[0033] If the proportion of Chinese characters is less than a preset threshold, then the primary index is determined to be the English semantic space and the Chinese semantic space, and the secondary index is the Chinese semantic space.

[0034] Furthermore, based on the Chinese and English fragment vectors and their corresponding attention weights, the process of retrieving the first candidate vector set by calling the main index of the corresponding semantic space includes:

[0035] The attention weights for Chinese and English fragment vectors are calculated based on preset language domain adaptability parameters.

[0036] The main index of the Chinese semantic space is invoked to retrieve the Chinese fragment vector, and the first retrieval result and the similarity with the Chinese fragment vector are obtained;

[0037] The main index of the English-specific semantic space is invoked to retrieve the English fragment vector, and a second retrieval result and the similarity with the English fragment vector are obtained;

[0038] Based on the attention weights of Chinese and English fragment vectors, the similarity between the first and second search results is corrected, and several first similarities are determined for the search results.

[0039] Based on the first similarity, the vectors are sorted in descending order to generate the first candidate vector set.

[0040] Furthermore, based on the first candidate vector set, the process of using language fragment vectors with a low proportion to call the secondary index for verification to obtain the second candidate vector set includes:

[0041] For several candidate vectors in the first candidate vector set, the secondary index is called to find the corresponding cross-language vector;

[0042] A second similarity calculation is performed in the secondary index using language fragment vectors with low proportions and the cross-language vectors;

[0043] Candidate vectors with a similarity greater than a preset threshold are retained and sorted in descending order to generate a second candidate vector set.

[0044] Furthermore, the process of calculating several index correction coefficients corresponding to the candidate vectors based on several similarities between the first and second candidate vectors includes:

[0045] The index correction coefficient is determined based on the ratio of the first similarity to the second similarity.

[0046] Furthermore, the corrected similarity is determined by multiplying the first similarity with the corresponding index correction coefficient.

[0047] Furthermore, based on the corrected similarity, the process of sorting several candidate vectors in the second candidate vector set to generate the first output result includes:

[0048] Based on the modified similarity, the candidate vectors are sorted in descending order.

[0049] If the number of candidate vectors is less than or equal to a preset threshold, then the sorted candidate vectors will be selected as the first output result.

[0050] Furthermore, the process of determining whether the number of candidate vectors exceeds a preset threshold to determine whether to trigger a filtering mechanism, and then determining whether to generate a second output result, includes:

[0051] If the number of candidate vectors exceeds a preset threshold, a filtering mechanism is triggered.

[0052] Calculate the corresponding timeliness weight based on the document generation time and the current time of several candidate vectors;

[0053] Based on the corrected similarity and the timeliness weight, a comprehensive score for several candidate vectors is determined;

[0054] Based on the comprehensive score, several candidate vectors are sorted in descending order to extract the number of candidate vectors corresponding to the threshold, which are then determined as the second output result.

[0055] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention provides a vector retrieval method based on a language model. By collecting mixed Chinese and English query text input by target users, a fast text classifier is used to accurately identify Chinese and English segments, achieving structured processing of cross-language retrieval input and laying the foundation for subsequent language-specific optimization. By removing stop words, the interference of meaningless words on semantic extraction is reduced, ensuring that the text participating in vector conversion is a core semantic carrier. Based on the Chinese large text vector conversion model, the weight of radical features is optimized, and the weight of word roots and affixes features is optimized, so that the generated vectors can accurately reflect the rules of Chinese character formation and the logic of English word formation, while also strengthening the core semantics of the domain. The main index and secondary index are dynamically determined by the proportion of Chinese segments, achieving adaptation of the retrieval strategy to the distribution of user input language. The combination of main index retrieval and secondary index verification ensures that the retrieval results cover the main semantic tendencies while avoiding the bias of a single perspective. By calculating the index correction coefficient and the correction similarity, the ranking results are made more consistent with the actual situation of cross-language semantic association. The selection mechanism triggered by the number of candidate vectors takes into account both the comprehensiveness of the results and the presentation efficiency. The entire process forms a closed loop from input processing to result output, adapting to the characteristics of Chinese and English languages ​​and improving the semantic accuracy of vector matching.

[0056] Furthermore, by loading the radical weight optimization module, the Chinese fragments are segmented and decomposed into characters to accurately extract the radicals of core characters, focusing the Chinese vector generation process on the character-forming units that play a key role in semantics. By calculating the average semantic similarity between the core character and high-frequency characters with the same radical, the semantic association strength of the core character in the group of characters with the same radical is quantified, providing data support for weight optimization. The basic weights of radicals are obtained by leveraging the knowledge graph of Chinese character radicals, and the optimized weights are calculated by combining the average semantic similarity, realizing the combination of domain knowledge and data-driven weight adjustment. The radical vectors and segmented word vectors are obtained based on the Chinese large text vector conversion model, and the final Chinese fragment vectors are generated by combining the optimized weights, so that the vectors retain the overall semantics at the word segmentation level while strengthening the core features at the radical level. The process of optimizing the weights of radical features fully utilizes the "form-meaning combination" characteristic of Chinese language. Especially in professional fields, it can make the radical features of core characters more closely related to the semantics of the domain, improve the semantic representation accuracy of Chinese fragment vectors for professional terms and domain-specific expressions, and provide a more accurate semantic carrier for subsequent Chinese semantic matching and cross-language retrieval.

[0057] Furthermore, by loading a root semantic optimization module, the English fragments are segmented and decomposed into roots to accurately locate core roots, grasp the core units of English semantic expression, and provide clear targets for semantic enhancement. By calculating the semantic contribution of core roots, the semantic determinism of roots in corresponding words is quantified, making weight adjustments more aligned with English word formation logic. Basic root weights are obtained using an English root domain knowledge graph, and optimized weights are calculated based on semantic contribution, achieving an objective assessment of root importance. Root semantic vectors and segmented vectors are obtained based on an English large text vector conversion model, and the final English fragment vector is generated by combining optimized weights. This ensures that the vector contains both the semantic information of the complete vocabulary and highlights the core semantics of the roots. The process of optimizing the weights of root and affix features fully utilizes the English linguistic characteristic that "roots determine core semantics." In professional fields, this makes the semantic association between core roots and domain terms clearer, improves the semantic representation accuracy of English fragment vectors for professional vocabulary and terminology variants, and provides a more accurate semantic carrier for English semantic matching and cross-language retrieval, especially suitable for professional scenarios with a large number of derived words and compound words.

[0058] Furthermore, by calculating the proportion of Chinese fragments based on the number of Chinese and English fragments after stop word removal, the distribution ratio of the two languages ​​in user input is objectively quantified, providing data basis for adjusting the retrieval strategy. By comparing the proportion of Chinese fragments with a preset threshold, the primary and secondary indexes are dynamically determined: when the proportion of Chinese is high, the Chinese semantic space is used as the primary index and English as the secondary index; conversely, when the proportion is low, the opposite is true, achieving targeted allocation of retrieval resources. This mechanism avoids the limitations of a single index mode in adapting to multilingual input, ensuring that the retrieval system prioritizes index resources that better match the distribution of user input languages, reducing the drag on retrieval efficiency from non-dominant language indexes. Simultaneously, the secondary index provides necessary retrieval channels for language fragments with low proportions, avoiding semantic omissions caused by the primary index's emphasis. The process of determining the primary and secondary indexes dynamically adapts the retrieval strategy to the user's language usage habits, improving the retrieval response efficiency of dominant language fragments while ensuring semantic coverage of non-dominant language fragments, thus enhancing the adaptability of the cross-language retrieval system to diverse inputs.

[0059] Furthermore, attention weights for Chinese and English fragment vectors are calculated based on preset language domain adaptability parameters, ensuring that the contribution of the two language vectors matches the domain adaptability characteristics and avoiding semantic bias caused by simply distributing weights equally. By calling the main indexes of the Chinese and English semantic spaces for separate searches, the semantic features of the two languages ​​are fully matched in their respective optimized spaces, improving the accuracy of intra-language searches. Attention weights are used to correct the similarity between the first and second search results, ensuring that the final first similarity reflects both the degree of vector matching and the semantic importance of the language in the domain, avoiding the bias caused by ranking solely based on the original similarity. A first candidate vector set is generated based on descending order of the first similarity, ensuring a closer correlation between the ranking results and semantic importance. This process organically combines language domain adaptability and vector matching degree, enabling the main index search results to cover the core semantics of both Chinese and English and to be reasonably ranked according to actual semantic importance, providing a high-quality initial candidate set for subsequent verification and correction.

[0060] Furthermore, by calling the secondary index on candidate vectors in the first candidate vector set to find corresponding cross-lingual vectors, a connection channel between the main index retrieval results and cross-lingual semantics is established, allowing single-language retrieval results to be validated in another language space. A second similarity is calculated using language fragment vectors with a low proportion and cross-lingual vectors to quantify the consistency of candidate vectors in cross-lingual semantics, providing an objective basis for selecting effective results. Candidate vectors with a second similarity greater than a preset threshold are retained and sorted, achieving precise filtering of the first candidate set and eliminating results that only match in the main index language space but are inconsistent in cross-lingual semantics. The verification process using the secondary index compensates for the limitations of the main index's single-language perspective. Through cross-lingual consistency verification, it ensures that the final candidate vectors are reasonable in both Chinese and English semantics, reducing retrieval bias caused by language differences, improving the overall quality of the candidate set, and laying a more reliable foundation for subsequent correction and sorting.

[0061] Furthermore, by determining the index correction coefficient based on the ratio of the first similarity to the second similarity, the evaluation difference between the primary and secondary indexes for the same candidate vector is quantified into a calculable coefficient, objectively representing the consistency of the two retrieval perspectives. When the ratio of the first similarity to the second similarity is close to 1, it indicates that the evaluation results of the two indexes are highly consistent, and the correction coefficient tends to stabilize, avoiding over-adjustment of results with high confidence. When the ratio deviates from 1, it indicates that there is a difference in evaluation, and the correction coefficient is adjusted accordingly, providing a clear direction for subsequent similarity correction. The process of determining the index correction coefficient enables the retrieval system to identify the degree of consistency in the evaluation of candidate vectors, providing a quantitative basis for balancing the retrieval results of the primary and secondary indexes, avoiding the one-sidedness that may be caused by relying solely on the evaluation of the primary or secondary index, enhancing the objectivity of the confidence assessment of the retrieval results, and laying the foundation for generating more reasonable corrected similarities.

[0062] Furthermore, by multiplying the first similarity score by the corresponding index correction coefficient to determine the corrected similarity score, dynamic adjustments to the main index retrieval results are achieved. This allows the similarity score to retain the basic matching information from the main index retrieval while incorporating the consistency assessment from the secondary index verification. When the main and secondary index assessments are consistent, the corrected similarity score maintains a reasonable correlation with the first similarity score, avoiding meaningless adjustments. When there are discrepancies in the assessments, the corrected similarity score reflects this inconsistency through coefficient adjustments, making the score more closely reflect the actual semantic association of the candidate vectors. This process organically integrates information from two retrieval perspectives, overcoming the limitations of a single similarity assessment. The corrected similarity score reflects both the dominance of the main index and incorporates the verification results from the secondary index, improving the accuracy of the similarity score in representing cross-linguistic semantic associations and providing a more reliable quantitative basis for the reasonable ranking of candidate vectors.

[0063] Furthermore, by sorting the candidate vectors in the second candidate vector set in descending order based on modified similarity, the ranking results directly correspond to the cross-linguistic semantic correlation strength of the candidate vectors, prioritizing vectors that better match the user's search intent. When the number of candidate vectors is less than or equal to a preset threshold, the ranked vectors are directly determined as the first output result, avoiding unnecessary filtering steps and reducing redundancy in the search process. With a moderate number of candidates, using modified similarity as the core ranking criterion ensures both the relevance of the results and improves search response efficiency, enabling users to quickly obtain results matching their search intent. Simultaneously, the clear ranking logic makes the results interpretable; users can intuitively perceive the degree of correlation between the candidate vectors and their search needs through the ranking order, enhancing the usability of the search results.

[0064] Furthermore, by triggering a filtering mechanism when the number of candidate vectors exceeds a preset threshold, information overload caused by too many results is avoided, ensuring that the output results are within the acceptable range for users. Timeliness weights are calculated based on the document generation time and the current time of the candidate vectors, ensuring that the search results consider not only semantic relevance but also the timeliness of information, meeting users' needs for the latest information. By adjusting similarity and timeliness weights to determine the comprehensive score, a combination of semantic relevance and timeliness is achieved, making the ranking results both consistent with the search intent and reflect the time value of the information. The comprehensive score is used to rank the results and extract a threshold number of candidate vectors as the second output, ensuring that the final output maintains a controllable quantity while optimizing the overall quality. Dynamically adjusting the processing strategy based on the number of results balances the comprehensiveness and usability of the results, improving the efficiency of users obtaining effective information, especially suitable for fields with frequent information updates, and enhancing the practical application value of the search results. Attached Figure Description

[0065] Figure 1 This is a flowchart illustrating the steps of the vector retrieval method based on a language model according to an embodiment of the present invention.

[0066] Figure 2 Flowchart of the steps in this embodiment of the invention for generating Chinese segment vectors after weighting the features of radicals;

[0067] Figure 3 Flowchart of the steps in generating English fragment vectors after optimizing the feature weights of word roots and affixes according to an embodiment of the present invention;

[0068] Figure 4 The flowchart of the steps for determining the primary and secondary indexes in the hybrid retrieval strategy in this embodiment of the invention is shown in the figure. Detailed Implementation

[0069] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0070] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0071] Please see Figure 1 The diagram shows the steps of a language model-based vector retrieval method according to an embodiment of the present invention. The present invention provides a language model-based vector retrieval method, including:

[0072] Step S1: Collect the mixed Chinese and English query text input by the target user, and identify the Chinese and English segments in the mixed Chinese and English query text;

[0073] Step S2: Based on the stop word lists for Chinese and English, remove stop words from the Chinese and English segments respectively;

[0074] Step S3: Based on the Chinese segment after stop word removal, call the Chinese large text vector conversion model to generate a Chinese segment vector with optimized weights for radical features; based on the English segment after stop word removal, call the English large text vector conversion model to generate an English segment vector with optimized weights for root and affix features.

[0075] Step S4: Based on the comparison between the proportion of Chinese fragments and the preset threshold, determine the primary index and secondary index in the hybrid retrieval strategy;

[0076] Step S5: Based on the Chinese fragment vectors, English fragment vectors, and corresponding attention weights, the main index of the corresponding semantic space is called to perform a search to obtain the first candidate vector set;

[0077] Step S6: Based on the first candidate vector set, use the language fragment vectors with a low proportion to call the secondary index verification to obtain the second candidate vector set;

[0078] Step S7: Based on several similarities between the first candidate vector and the second candidate vector, calculate several index correction coefficients corresponding to the candidate vectors, and calculate the corresponding corrected similarity based on each index correction coefficient.

[0079] Step S8: Based on the corrected similarity, sort the candidate vectors to generate the first output result;

[0080] Step S9: Based on the comparison result between the number of candidate vectors and the preset vector threshold, determine whether to trigger the filtering mechanism to determine whether to generate a second output result.

[0081] Specifically, in step S1, a fast text classifier can be used to identify Chinese and English segments in the text.

[0082] Understandably, the fast text classifier is a lightweight language recognition model in the existing technology. It can quickly distinguish the language of each character or word in the text and output the "zh" sequence, which is a continuous sequence of Chinese text fragments, and the "en" sequence, which is a continuous sequence of English text fragments.

[0083] Specifically, in step S2, a Chinese stop word list is used to remove stop words from the Chinese segment, and an English stop word list is used to remove stop words from the English segment.

[0084] It can be understood that the stop words in Chinese or English refer to words that frequently appear in sentences and have no actual retrieval significance, such as "的", "是", "is", "the", etc. The stop word lists for Chinese and English in the prior art can be used, which will not be elaborated here.

[0085] Specifically, in step S3, the Chinese large text vector conversion model is called based on the Chinese fragment after stop word removal to generate a Chinese fragment vector with optimized weights for radical and character component features.

[0086] The English large text vector conversion model is called based on the English fragment after stop word removal to generate an English fragment vector with optimized weights for root and affix features.

[0087] It can be understood that the Chinese large text vector conversion model is a pre-trained language model optimized for Chinese language characteristics. It converts the Chinese fragment after stop word removal into a vector, enabling the vector to accurately reflect the semantics, grammar, and character formation features of Chinese. The English large text vector conversion model is a pre-trained language model optimized for English language characteristics. It converts the English fragment after stop word removal into a vector, enabling the vector to reflect the roots, word meanings, and grammar features of English. The converted vector quantifies the text semantics into a form that can be understood by a computer and is an ordered sequence of numbers, where each number represents the feature value of the text in a semantic dimension.

[0088] In this embodiment, the Chinese large text vector conversion model can adopt, but is not limited to, "text2vec-large-chinese" developed by GanymedeNil; the English large text vector conversion model can adopt "bert-base-uncased" developed by Google; the Chinese fragment and the English fragment after stop word removal are respectively converted into 512-dimensional low-dimensional dense vectors, that is, 512 numerical dimensions cover all the semantic information of the text, which not only solves the problem that the computer cannot directly understand the text but also ensures the accuracy of semantics and the efficiency of calculation, providing a computable semantic carrier for subsequent cross-language matching, hybrid vector generation, and other operations.

[0089] Please refer to Figure 2 as shown in the flowchart of the steps for generating a Chinese fragment vector with optimized weights for radical and character component features in the embodiment of the present invention.

[0090] Specifically, in step S3, the process of calling the Chinese large text vector conversion model based on the Chinese fragment after stop word removal to generate a Chinese fragment vector with optimized weights for radical and character component features includes

[0091] Step S311, performing word segmentation and character decomposition on the Chinese fragment to extract the radicals of the core single characters.

[0092] Step S312: Calculate the semantic similarity between the core single character and the high-frequency characters with the same radical, and calculate the average value.

[0093] Step S313: Obtain the basic weight of the radical.

[0094] Step S314: Calculate the optimized weight based on the average value of the semantic similarity of the core single character and the basic weight of the corresponding radical.

[0095] Step S315: Obtain the radical vector corresponding to the core single character and several segmented word vectors based on the Chinese large text vector conversion model.

[0096] Step S316: Generate an optimized Chinese fragment vector based on several segmented word vectors, the radical vector corresponding to the core single character, and the optimized weight of the radical corresponding to the core single character.

[0097] In this embodiment, the radical weight optimization module can be used to perform operations such as grading, character splitting, and calling the basic weight in the above process. It is a post-processing component independent of the basic model, used to strengthen the radical features based on the output vector of the model. The built-in core resources of the module include a Chinese character radical mapping table constructed based on the "Table of Chinese Character Radicals", a radical basic weight library constructed based on the knowledge graph of the Chinese character radical field, and a Chinese character library classified by radicals. The basic weight of the radical can also be determined by the proportion of the occurrence frequency in the Chinese character radical library.

[0098] In this embodiment, the vector optimization process of the Chinese fragment "vaccine research and development" is as follows:

[0099] Load the radical weight optimization module, segment "vaccine research and development" to obtain the vocabulary sequence: ["vaccine", "research and development"]. Based on the corresponding medical field dictionary, the character "yi" in "vaccine" is marked as the domain core single character.

[0100] Extract the radical of the core single character through the Chinese character radical mapping table of the optimization module to obtain the radical "疒" of "yi".

[0101] Calculate the average value of the semantic similarity between the core single character and the high-frequency characters with the same radical. Taking "yi" as an example, select 5 high-frequency sample characters from the Chinese character library with the radical "疒": ["disease", "illness", "symptom", "treatment", "pain"]; calculate the vector similarity between "yi" and each sample character through the cosine similarity formula, and calculate the average value of the similarity; it can be understood that according to the need to select the number of high-frequency sample characters, the Chinese characters in the Chinese character library are sorted in descending order according to the occurrence frequency of the characters with the corresponding radical, and the first corresponding number of Chinese characters are extracted as high-frequency sample characters.

[0102] Obtain the basic weight of the radical corresponding to the single character based on the radical basic weight library, and use the formula: optimized weight = basic weight + (average semantic similarity - benchmark threshold) * influence amplitude control factor, where the benchmark threshold is set to 0.5 and the influence amplitude control factor is set to 0.4, to calculate the optimized weight of the radical corresponding to the single character;

[0103] Input the segmented lexical sequence: ["vaccine", "research and development"] and the radical "疒" corresponding to the core single character into the Chinese large text vector conversion model, and output the 512-dimensional "vaccine" vector, "research and development" vector, and "疒" vector respectively;

[0104] Calculate the "vaccine" optimized vector = "vaccine" vector * (1 - optimized weight) + "疒" vector * optimized weight;

[0105] Generate the "vaccine research and development" optimized vector = "vaccine" optimized vector + "research and development" vector.

[0106] By focusing on the core single character "疫" and its radical "疒", and strengthening the contribution of the "疒" radical vector to the "vaccine" vector based on the optimized weight, making the vector of "vaccine" more accurately anchored to the medical attribute, and retaining the original vector for the non-core segmentation "research and development" to ensure the integrity of its "research and development" action attribute.

[0107] Please refer to Figure 3 as shown, the step flowchart of generating the optimized English fragment vector for the root and affix features in the embodiment of the present invention.

[0108] Specifically, in step S3, the process of calling the English large text vector conversion model based on the English fragment after stop word removal to generate the optimized English fragment vector for the root and affix features includes,

[0109] Step S321, perform word segmentation and root decomposition processing on the English fragment to extract the core root and calculate the semantic contribution degree of the core root;

[0110] Step S322, obtain the basic weight of the root;

[0111] Step S323, calculate the optimized weight based on the semantic contribution degree of the core root and the basic weight of the corresponding root;

[0112] Step S324, obtain the semantic vector corresponding to the core root and several segmented vectors based on the English large text vector conversion model;

[0113] Step S325, generate the optimized English fragment vector based on several segmented vectors, the semantic vector corresponding to the core root, and the optimized weight corresponding to the core root.

[0114] In this embodiment, the root and affix weight optimization module is a post-processing component independent of the base model. It is used to enhance root features based on the model's output vector. The module's built-in core resources include a root-vocabulary mapping library built based on the English etymology dictionary to determine the semantic contribution of roots in target words, and a root basic weight library built based on a medical corpus. The root basic weights can also be determined by the frequency proportion of occurrence in the English root library.

[0115] In this embodiment, the vector optimization process for providing the English phrase "influenza vaccine trials" includes:

[0116] Load the root semantic optimization module to segment “influenza vaccine trials” and obtain the word sequence: ["influenza","vaccine","trials"]. Based on the corresponding medical domain dictionary, and based on the frequency of occurrence of “vaccine”, mark it as the domain core word.

[0117] The core word root is extracted through the word root-vocabulary mapping table of the optimization module to obtain the core word root "vaccin-" of "vaccine" and its semantic contribution. The semantic contribution is determined by the proportion of the word root meaning in the corresponding word definition. For example, the semantic contribution of "vaccin-" in "vaccine" is 1.0, which is the highest value, indicating that the word root can completely determine the core semantics of the target word.

[0118] The basic weight of "vaccin-" as a core word root in the field of vaccines was obtained through a basic weight library of medical word roots;

[0119] Calculate the optimization weight = base weight * (1 + semantic contribution * influence amplitude control factor), where the influence amplitude control factor is set to 0.2;

[0120] The word segmentation sequence and core word roots are input into the English large text vector conversion model, which outputs 512-dimensional vectors for "influenza" (preserving the disease attribute of "influenza"), "vaccine", "trials" (preserving the action attribute of "trials"), and "vaccin-".

[0121] Calculate the "vaccine" optimization vector = "vaccine" vector * (1 - optimization weight) + "vaccin-" semantic vector * optimization weight;

[0122] Generate the optimized vector for "influenza vaccine trials" = "influenza" vector + "vaccine" optimized vector + "trials" vector.

[0123] Understandably, quantifying the semantic determinism of word roots on target words by their semantic contribution within the vocabulary allows for more targeted weight adjustments. In English, word roots determine core semantics, while morphology fulfills grammatical functions; therefore, strengthening only the semantic vector preserves the grammatical features of the original word form. The optimized target word vector, obtained by weighting word roots through optimized weighting, strengthens the most relevant dimensions, enabling the optimized vector to encode more core semantic information.

[0124] Please see Figure 4 The flowchart shown illustrates the steps for determining the primary and secondary indexes in a hybrid retrieval strategy according to an embodiment of the present invention.

[0125] Specifically, in step S4, the process of determining the primary and secondary indexes in the hybrid retrieval strategy based on the comparison between the proportion of Chinese fragments and a preset threshold includes the following:

[0126] Step S41: Calculate the proportion of Chinese segments based on the number of characters in the Chinese and English segments after stop word removal is completed;

[0127] Step S42: If the proportion of Chinese characters is greater than or equal to a preset threshold, then the primary index is determined to be the Chinese semantic space and the English semantic space, and the secondary index is the English semantic space.

[0128] Step S43: If the proportion of Chinese characters is less than a preset threshold, then the primary index is determined to be the English semantic space and the Chinese semantic space, and the secondary index is the Chinese semantic space.

[0129] Understandably, based on the comparison between the proportion of Chinese fragments and the preset threshold, the primary and secondary indexes in the hybrid retrieval strategy are determined to allow the parts with clearer language and greater information content to dominate the main retrieval process, so as to improve the hit rate of the first round of retrieval. It is recommended that the preset threshold be taken in the range of [0.55, 0.6].

[0130] In this embodiment, the primary index can be, but is not limited to, an HNSW (Hierarchical Navigable Small World) index. HNSW provides an optimal balance between query speed and recall in Approximate Nearest Neighbor (ANN) search, making it suitable as a first-round coarse screening. The secondary index can be, but is not limited to, an IVF_FLAT (Inverted File with Flat storage) index. IVF indexes are based on clustering and can perform fast range queries or precise similarity calculations on a specific set of vectors. While it is fast at precise comparisons of specific targets, it is slow for full database scans, making it suitable for scenarios where a small number of candidates need to be verified.

[0131] Specifically, in step S5, the process of retrieving the first candidate vector set by calling the main index of the corresponding semantic space based on the Chinese and English fragment vectors and their corresponding attention weights includes:

[0132] Step S51: Calculate the attention weights of Chinese and English segment vectors based on preset language domain adaptability parameters;

[0133] Step S52: Call the main index of the Chinese-specific semantic space to search the Chinese fragment vector, and obtain the first search result and the similarity with the Chinese fragment vector;

[0134] Step S53: Call the main index of the English-specific semantic space to search the English fragment vector, and obtain the second search result and the similarity with the English fragment vector;

[0135] Step S54: Based on the attention weights of the Chinese fragment vector and the English fragment vector, correct the similarity between the first search result and the second search result, and determine several first similarities corresponding to the search results;

[0136] Step S55: Sort the vectors in descending order based on the first similarity to generate the first candidate vector set.

[0137] Understandably, attention weights are used to quantify the contribution of Chinese and English vectors when mixed. Higher semantic similarity and a language vector that more accurately expresses the core concept result in a higher weight, while ensuring that both language vectors retain their base weights to avoid semantic loss. Preset language domain adaptability parameters are used to annotate the accuracy of Chinese and English expressions of the core concept using a domain dictionary, taking values ​​in the range [0,1], with higher values ​​indicating greater accuracy. The base weight threshold is set to 0.25 to ensure that the weight of any language is not lower than 0.25, preventing semantic loss. When the semantic similarity is greater than 0.7, the preset language domain adaptability parameters for Chinese and English are normalized. The calculation formula is: Chinese attention weight = Chinese adaptability parameter / (Chinese adaptability parameter + English adaptability parameter), English attention weight = English adaptability parameter / (Chinese adaptability parameter + English adaptability parameter). If any attention weight is less than the base weight threshold, 0.25 is determined as the corresponding attention weight.

[0138] In this embodiment, the optimized Chinese and English fragment vectors are input into the Chinese HNSW index and the English HNSW index, respectively. Cosine similarity is used as the distance metric to retrieve several results and their corresponding original similarity scores. Weighted fusion: For the results from the Chinese index, the final score S_ch_final = S_ch_raw * W_ch. The results are weighted and fused according to attention weights. Then, all results are merged into a list and sorted in descending order as the first candidate set.

[0139] Understandably, weighted merging ensures that results from more important language indexes have an advantage in the merged ranking, which is more reasonable than simply merging the two result lists.

[0140] Specifically, in step S6, the process of obtaining the second candidate vector set by using language fragment vectors with a low proportion to call the secondary index for verification based on the first candidate vector set includes the following steps:

[0141] Step S61: For several candidate vectors in the first candidate vector set, call the secondary index to find the corresponding cross-language vector;

[0142] Step S62: Use the language segment vector with a low proportion and the cross-language vector to perform a second similarity calculation in the secondary index;

[0143] Step S63: Retain the candidate vectors whose similarity is greater than the preset threshold, and sort them in descending order to generate a second candidate vector set.

[0144] Understandably, primary index retrieval may produce results that are not entirely accurate; for example, a document might closely match the English part of the query but be completely unrelated to the Chinese part. Cross-language consistency checks can filter out these inconsistent candidates, significantly improving the accuracy of the final results.

[0145] In this embodiment, the process of finding cross-language vectors includes: when constructing the vector library, each document stores both Chinese name vectors and English name vectors, and they are associated with a unique ID, meaning that each vector in the first candidate set contains this unique ID; using this ID, the corresponding cross-language vector can be quickly retrieved from the vector table where the secondary index (IVF_FLAT) is located. For example, if the primary index is an English index, and each vector in the first candidate set is matched using English vectors, then the corresponding Chinese vectors for these entries can be retrieved using the ID.

[0146] In this embodiment, cosine similarity is calculated between language segment vectors with low proportions and the cross-language vectors, and compared with a preset threshold. Candidate vectors with similarity higher than the preset threshold are retained, and after being sorted in descending order, a second candidate vector set is generated. The threshold is recommended to be within the range of [0.55, 0.65].

[0147] Specifically, in step S7, the process of calculating several index correction coefficients corresponding to the candidate vectors based on several similarities between the first candidate vector and the second candidate vector includes:

[0148] Step S71: Determine the index correction coefficient based on the ratio of the first similarity to the second similarity.

[0149] In this embodiment, based on the second similarity obtained in step S62 and the first similarity obtained in step S54 for each candidate vector in the second candidate vector set, an index correction coefficient is calculated for each candidate vector. It can be understood that the closer the ratio of the first similarity to the second similarity is to 1, the more consistent the scores of the two indices are. The index correction coefficient is equal to the base coefficient plus the adjustment coefficient multiplied by 1 minus the absolute value of the difference between the ratio of the first similarity to the second similarity. The base coefficient is set to 0.8 to ensure that the correction is not too drastic, and the adjustment coefficient is set to 0.2 to control the degree of influence of inconsistency on the similarity. The closer the ratio of the first similarity to the second similarity is to 1, the closer the index correction coefficient is to the base coefficient; the further the ratio of the first similarity to the second similarity deviates from 1, the greater the inconsistency, and the smaller the index correction coefficient.

[0150] Understandably, the index correction coefficient is used to adjust the confidence level of the primary index retrieval results. When the calculation results of the primary and secondary indexes are highly consistent, the score of the primary index is slightly suppressed to avoid overconfidence; when the differences are large, the correction magnitude needs to be increased. The alignment coefficient comes from a pre-trained Chinese and English word vector alignment matrix. It can use FastText cross-language vectors to calculate the cosine similarity of the Chinese and English names of candidate entities, which is used to improve the ranking of candidate vectors with high alignment between Chinese and English expressions, thereby improving the accuracy of cross-language retrieval.

[0151] Specifically, in step S7, the corrected similarity is determined by multiplying the first similarity with the corresponding index correction coefficient.

[0152] Specifically, in step S8, the process of sorting several candidate vectors in the second candidate vector set based on the corrected similarity to generate the first output result includes:

[0153] Step S81: Sort the candidate vectors in descending order based on the corrected similarity.

[0154] Step S82: If the number of candidate vectors is less than or equal to a preset threshold, then the sorted candidate vectors are determined as the first output result.

[0155] Specifically, in step S9, the process of determining whether the number of candidate vectors exceeds a preset threshold to determine whether to trigger a filtering mechanism, and then determining whether to generate a second output result, includes the following steps:

[0156] Step S91: If the number of candidate vectors is greater than a preset threshold, the filtering mechanism is triggered.

[0157] Step S92: Calculate the corresponding timeliness weight based on the document generation time and the current time of several candidate vectors;

[0158] Step S93: Based on the corrected similarity and the timeliness weight, determine the comprehensive score of several candidate vectors;

[0159] Step S94: Sort several candidate vectors in descending order according to the comprehensive score to extract the number of candidate vectors corresponding to the threshold and determine them as the second output result.

[0160] If the number of candidate vectors exceeds a preset threshold, a filtering mechanism is triggered. This threshold is typically the number of vectors a user can view on one screen, usually set between 10 and 50. For example, web search typically uses 10 vectors, while mobile apps or internal retrieval systems may allow 20 or 50. A comprehensive score is calculated for several candidate vectors using a timeliness weight. These candidate vectors are then sorted in descending order based on their comprehensive scores to extract the number of candidate vectors corresponding to the threshold, which is determined as the second output result. This second output result is the final output result.

[0161] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A vector retrieval method based on a language model, characterized in that, include: Collect mixed Chinese and English query text input by the target user, and identify Chinese and English segments in the mixed Chinese and English query text; Based on the stop word lists for Chinese and English, stop words were removed from the Chinese and English segments respectively; Based on the Chinese segment after stop word removal, a text vector conversion is performed to generate a Chinese segment vector with optimized weights for radical features. Based on the English segment after stop word removal, a text vector conversion is performed to generate an English segment vector with optimized weights for root and affix features. Based on the comparison results of the proportion of Chinese fragments and the preset threshold, the primary and secondary indexes for hybrid retrieval are determined; Based on the Chinese and English fragment vectors and their corresponding attention weights, the main index of the corresponding semantic space is called to retrieve the first candidate vector set. Based on the first candidate vector set, the language fragment vector corresponding to the secondary index is used to call the secondary index for verification, and the second candidate vector set is obtained. Based on the similarity scores corresponding to the first and second candidate vectors, calculate several index correction coefficients corresponding to the candidate vectors. The corresponding corrected similarity is calculated based on several index correction coefficients, and the candidate vectors are sorted to generate the first output result. The number of candidate vectors determines whether a preset threshold is exceeded, which in turn determines whether a filtering mechanism should be triggered to determine whether a second output result should be generated.

2. The vector retrieval method based on a language model according to claim 1, characterized in that, The process of generating Chinese segment vectors after optimizing the weights of radical features includes: The Chinese segment is processed by word segmentation and character splitting to extract the radicals of the core characters; Calculate the semantic similarity between the core character and high-frequency characters with the same radical, and calculate the mean. Obtain the basic weight of the radical; Optimized weights are calculated based on the mean semantic similarity of the core characters and the basic weights of the corresponding radicals. Obtain the radical vectors corresponding to the core characters and several word segmentation vectors; Based on several word segmentation vectors, the radical vectors corresponding to core characters, and the optimized weights of the radicals corresponding to core characters, optimized Chinese fragment vectors are generated.

3. The vector retrieval method based on a language model according to claim 1, characterized in that, The process of generating English fragment vectors after optimizing the feature weights of root words and affixes includes, The English fragment is segmented and decomposed into word roots to extract core word roots and calculate the semantic contribution of the core word roots. Obtain the basic weight of the word root; Calculate the optimized weights based on the semantic contribution of the core word roots and the basic weights of the corresponding word roots; Obtain the semantic vectors corresponding to the core word roots and several word segmentation vectors; Based on several word segmentation vectors, semantic vectors corresponding to core word roots, and optimized weights corresponding to core word roots, optimized English fragment vectors are generated.

4. The vector retrieval method based on a language model according to claim 1, characterized in that, The process of determining the primary and secondary indexes in a hybrid retrieval strategy based on a comparison of the proportion of Chinese fragments with a preset threshold includes: Calculate the proportion of Chinese segments based on the number of Chinese and English segments after stop word removal is completed; If the proportion of Chinese characters is greater than or equal to a preset threshold, then the primary index is determined to be the Chinese semantic space and the English semantic space, and the secondary index is the English semantic space. If the proportion of Chinese characters is less than a preset threshold, then the primary index is determined to be the English semantic space and the Chinese semantic space, and the secondary index is the Chinese semantic space.

5. The vector retrieval method based on a language model according to claim 4, characterized in that, The process of obtaining the first candidate vector set by retrieving the first vector set by calling the main index of the corresponding semantic space based on the Chinese and English fragment vectors and their corresponding attention weights includes: The attention weights for Chinese and English fragment vectors are calculated based on preset language domain adaptability parameters. The main index of the Chinese semantic space is invoked to retrieve the Chinese fragment vector, and the first retrieval result and the similarity with the Chinese fragment vector are obtained; The main index of the English-specific semantic space is invoked to retrieve the English fragment vector, and a second retrieval result and the similarity with the English fragment vector are obtained; Based on the attention weights of Chinese and English fragment vectors, the similarity between the first and second search results is corrected, and several first similarities are determined for the search results. Based on the first similarity, the vectors are sorted in descending order to generate the first candidate vector set.

6. The vector retrieval method based on a language model according to claim 5, characterized in that, Based on the first candidate vector set, the process of using language fragment vectors with a low proportion to call the secondary index for verification to obtain the second candidate vector set includes: For several candidate vectors in the first candidate vector set, the secondary index is called to find the corresponding cross-language vector; A second similarity calculation is performed in the secondary index using language fragment vectors with low proportions and the cross-language vectors; Candidate vectors with a similarity greater than a preset threshold are retained and sorted in descending order to generate a second candidate vector set.

7. The vector retrieval method based on a language model according to claim 6, characterized in that, The process of calculating several index correction coefficients for candidate vectors based on several similarities between the first and second candidate vectors includes: The index correction coefficient is determined based on the ratio of the first similarity to the second similarity.

8. The vector retrieval method based on a language model according to claim 7, characterized in that, The corrected similarity is determined by multiplying the first similarity with the corresponding index correction coefficient.

9. The vector retrieval method based on a language model according to claim 1, characterized in that, The process of sorting several candidate vectors in the second candidate vector set based on the corrected similarity and generating the first output result includes: Based on the modified similarity, the candidate vectors are sorted in descending order. If the number of candidate vectors is less than or equal to a preset threshold, then the sorted candidate vectors will be selected as the first output result.

10. The vector retrieval method based on a language model according to claim 1, characterized in that, The process of determining whether the number of candidate vectors exceeds a preset threshold, to determine whether to trigger a filtering mechanism, and to determine whether to generate a second output result includes... If the number of candidate vectors exceeds a preset threshold, a filtering mechanism is triggered. Calculate the corresponding timeliness weight based on the document generation time and the current time of several candidate vectors; Based on the corrected similarity and the timeliness weight, a comprehensive score for several candidate vectors is determined; Based on the comprehensive score, several candidate vectors are sorted in descending order to extract the number of candidate vectors corresponding to the threshold, which are then determined as the second output result.

Images