Short text recall method, system, device and medium based on business standard words

Abstract

The application belongs to the field of data mining, and discloses a short text recall method, system, device and medium based on business standard words. The recall method comprises: through a standard word table containing business standard words constructed in advance, a candidate word set is generated by sequentially performing synonym matching and pinyin correction on user query words, and then a two-dimensional matching score of character similarity and vector similarity is combined to finally determine the target standard word. The scheme combines the advantages of rule matching and semantic matching: synonym matching and pinyin correction solve the problems of homophonic and different forms, near-synonymous confusion and other fuzzy input problems, character similarity guarantees accurate filtering at the literal level, and vector similarity captures deep semantic association, so that the accuracy and robustness of short text recall are effectively improved; meanwhile, the layered calculation strategy controls the calculation overhead under the premise of ensuring the recall quality, meets the real-time retrieval demand, and supports high-precision standard word positioning in multiple scenes.

Description

Technical Field

[0001] This application belongs to the field of data mining, and specifically relates to a short text retrieval method, system, device and medium based on business standard words. Background Technology

[0002] In Natural Language Processing and Structured Query Transformation (NL2SQL) scenarios, users often describe their business query needs using fuzzy, colloquial natural language. The system must first accurately map such input to standardized business terms before it can generate valid SQL statements to complete data retrieval. The core of this process lies in the accuracy of recalling standardized business terms in short texts, which directly affects the reliability of subsequent NL2SQL transformation and the effectiveness of business scenario implementation.

[0003] Current short text standard word retrieval technology faces multiple challenges: First, the flexibility of Chinese expression leads to prominent semantic drift issues, with user input often containing homophones, near-synonyms, regional abbreviations, and other variations, which traditional methods struggle to effectively identify; second, there is a disconnect between business rules and semantic understanding, with existing retrieval systems relying heavily on general text matching and failing to deeply integrate industry-specific thesaurus standards, resulting in retrieval results that are out of sync with actual business scenarios; third, insufficient utilization of multi-dimensional features, with mainstream solutions primarily based on single text similarity calculations, neglecting complementary information such as pinyin correction, character shape similarity, and word frequency distribution, making them less robust to input noise (such as misspellings and chaotic sentence breaks). Summary of the Invention

[0004] To address the aforementioned issues, this application provides a method, system, device, and medium for short text retrieval based on business standard terms. This solution integrates rule matching and semantic matching, and through the collaborative efforts of three layers—pinyin error correction, character similarity, and vector similarity—it balances precise filtering with semantic generalization, achieving high-precision, low-latency short text retrieval.

[0005] The first aspect of this disclosure proposes a short text recall method based on business standard terms, the recall method comprising: Based on a pre-built standard vocabulary list containing standard business terms, synonym matching and pinyin correction are performed on the query terms input by the user to obtain candidate terms corresponding to the query terms, forming a candidate term set; Calculate the character similarity between the query term and each candidate word in the candidate word set; convert the query term and each candidate word in the candidate word set into word vectors respectively, and calculate the vector similarity based on the word vectors; The matching scores between the query term and each candidate term are obtained through the character similarity and the vector similarity; based on the matching scores, the target standard term corresponding to the query term is determined from the candidate terms.

[0006] According to a preferred embodiment of this disclosure, the recall method further includes: Collect user query logs in real time and extract valid query texts to form a short text set; The query short texts in the short text set are matched with the standard words in the standard word list to determine the standard words that have a preset matching degree with the short texts, and these standard words are used as potential synonyms. Count the number of times the query text and the potential synonyms co-occur, and determine candidate synonyms from the potential synonyms; Calculate the candidate confidence scores between the query text and the candidate synonyms, and select candidate synonyms to be added to the standard thesaurus based on the candidate confidence scores.

[0007] According to a preferred embodiment of this disclosure, the step of calculating the candidate confidence scores between the query short text and the candidate synonyms, and selecting candidate synonyms to be added to the standard thesaurus based on the candidate confidence scores, includes: Calculate the candidate similarity between the query short text and the candidate synonyms, and use it as the candidate confidence score; When the candidate similarity meets the first preset threshold range, the candidate synonym is added to the standard vocabulary. When the candidate similarity meets the second preset threshold range, the candidate synonym is added to the manual review queue, and the review results are used to determine whether to add it to the standard word list; the second preset threshold range is smaller than the first preset threshold range.

[0008] According to a preferred embodiment of this disclosure, the step of performing synonym matching and pinyin correction on the user-input query word based on a pre-constructed standard vocabulary containing business standard words, to obtain candidate words corresponding to the query word, forming a candidate word set, including: The query term is matched with the standard terms in the standard term list to determine the standard terms that match the query term as the matching synonyms; The query term and each of the matching synonyms are identified as having multiple pronunciations. Based on the business domain, the query term and the matching synonyms are converted into their corresponding pinyin. Calculate the semantic similarity between the pinyin of the matching synonym and the pinyin of the query word, perform pinyin correction based on the semantic similarity, and select matching synonyms whose semantic similarity meets the preset semantic similarity threshold as candidate words for the corresponding query word; The candidate words are combined to form a candidate word set.

[0009] According to a preferred embodiment of this disclosure, calculating the character similarity between the query term and each candidate term in the candidate term set includes: Add positional weights to each word in the query term and the candidate terms according to their word order; Based on the position weight, the intersection ratio of the word sets of the query word and each word in the candidate words is calculated as the first character similarity; Calculate the semantic similarity between the query term and the candidate term, and correct the first character similarity using the semantic similarity to obtain the corrected character similarity; Convert the query term and the candidate term into pinyin, calculate the intersection ratio of the word sets of the pinyin of the query term and the candidate term, and use it as the second character similarity. By combining the corrected character similarity and the second character similarity, the character similarity between the query term and the candidate term is obtained; The character similarity between the query term and each candidate term in the candidate term set is calculated.

[0010] According to a preferred embodiment of this disclosure, the query term and each candidate term in the candidate term set are converted into word vectors, and vector similarity is calculated based on the word vectors, including: The query terms and candidate terms are respectively converted into query term vectors and candidate term vectors of specific dimensions; The query word vector and candidate word vector are reduced in dimensionality using principal component analysis. Calculate the cosine similarity between the query word vector and the candidate word vector, and use it as the vector similarity. The vector similarity between the query term and each candidate term in the candidate term set is calculated respectively.

[0011] According to a preferred embodiment of this disclosure, determining the target standard word corresponding to the query word from the candidate words based on the matching score includes: When the matching score of any candidate word is greater than or equal to a preset confidence threshold, the candidate word is used as the target standard word. When the matching scores of the candidate words are all less than a preset confidence threshold, a preset number of the candidate words are selected as the target standard words in descending order of matching scores.

[0012] To address the aforementioned technical problems, a second aspect of this disclosure proposes a short text retrieval system based on business standard terms, the retrieval system comprising: The word filtering module is used to perform synonym matching and pinyin correction on the query words input by the user based on a pre-built standard word list containing business standard words, and to obtain candidate words corresponding to the query words, forming a candidate word set; A semantic coarse screening module is used to calculate the character similarity between the query term and each candidate word in the candidate word set; convert the query term and each candidate word in the candidate word set into word vectors respectively, and calculate the vector similarity based on the word vectors; The fine-ranking output module is used to obtain the matching score between the query term and each of the candidate terms through the character similarity and the vector similarity; and to determine the target standard term corresponding to the query term among the candidate terms based on the matching score.

[0013] To address the aforementioned technical problems, a third aspect of this disclosure provides an electronic device, comprising: Processor; and A memory storing computer-executable instructions, which, when executed, cause the processor to perform the method described in any of the above embodiments.

[0014] To address the aforementioned technical problems, a fourth aspect of this disclosure provides a computer storage medium that stores one or more programs, which, when executed by a processor, implement the method described in any of the above embodiments.

[0015] Compared with existing technologies, this application has the following advantages: By pre-constructing a standard vocabulary containing standard business terms, the user query terms are sequentially matched with synonyms and corrected with pinyin to generate a candidate word set. Then, the target standard word is finally determined by combining the dual-dimensional matching scores of character similarity and vector similarity. This solution integrates the advantages of rule matching and semantic matching: synonym matching and pinyin correction solve fuzzy input problems such as homophones and near-synonyms; character similarity ensures accurate filtering at the literal level; and vector similarity captures deep semantic relationships. The three work together to effectively improve the accuracy and robustness of short text retrieval. At the same time, the hierarchical calculation strategy controls the computational cost while ensuring the quality of retrieval, meets the needs of real-time retrieval, and supports high-precision standard word positioning in multiple scenarios.

[0016] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures pointed out in the description, claims and drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A schematic diagram of one of the short text recall methods based on business standard words according to an embodiment of the present disclosure is shown; Figure 2 A second schematic diagram of a short text recall method based on business standard words according to an embodiment of the present disclosure is shown; Figure 3 A schematic diagram of a short text recall method based on business standard words according to an embodiment of the present disclosure is shown in part three. Figure 4 A schematic diagram of a short text recall method based on business standard words according to an embodiment of this disclosure is shown in Figure 4. Figure 5 A schematic diagram of a short text recall method based on business standard words according to an embodiment of the present disclosure is shown in Figure 5. Figure 6 A schematic diagram of a short text recall system based on business standard terms according to an embodiment of the present disclosure is shown; Figure 7 A schematic diagram of an electronic device structure according to an embodiment of the present disclosure is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The same reference numerals in the accompanying drawings denote the same or similar elements, components, or parts, and therefore repeated descriptions of the same or similar elements, components, or parts may be omitted below. It should also be understood that although qualifiers such as first, second, third, etc., indicating numbers may be used herein to describe various devices, elements, components, or parts, these devices, elements, components, or parts should not be limited by these qualifiers. That is, these qualifiers are only used to distinguish one from another. For example, a first device may also be referred to as a second device, without departing from the essence of the technical solution of this disclosure. Furthermore, the terms "and / or" and "and / or" refer to all combinations including the first or more of the listed items.

[0021] Please see Figure 1 , Figure 1 This is one of the flowcharts of a short text retrieval method based on business standard terms provided in this publication, such as... Figure 1 As shown, the recall methods include: S11. Based on a pre-built standard vocabulary list containing standard business terms, perform synonym matching and pinyin correction on the query terms entered by the user to obtain candidate terms for the corresponding query terms and form a candidate term set.

[0022] In this embodiment, a standard vocabulary containing standard business terms is pre-constructed. The user-input query terms are then subjected to two layers of rule-based processing: synonym matching and pinyin correction. The matched candidate words are grouped into a candidate word set, which serves as the basis for subsequent fine-grained ranking. Synonym matching utilizes the mapping relationship between standard words and synonyms in the business vocabulary, effectively solving semantic drift problems such as industry abbreviations and the mixing of near-synonyms. Pinyin correction corrects homophones at the pronunciation level, significantly reducing the risk of missed recalls due to typos. The combination of these two methods forms a lightweight pre-filtering layer, quickly narrowing the candidate range before vector calculation. This improves the accuracy and robustness of recall while reducing subsequent computational overhead, laying the foundation for millisecond-level real-time response.

[0023] In this embodiment, the solution performs word filtering on the user-input query terms, including: Synonym expansion: By adaptively updating the word list, the user input is expanded into a candidate set containing synonyms (e.g., "Smart Energy Department" is expanded to "Smart Energy Business Unit" and "Energy Business Unit"), filtering out standard words with no matching. Pinyin correction: Based on semantically enhanced pinyin matching, obvious homophone errors are corrected, generating a corrected candidate set. This layer can accurately find standard words through lightweight rules and dictionary operations.

[0024] S12. Calculate the character similarity between the query term and each candidate word in the candidate word set; convert the query term and each candidate word in the candidate word set into word vectors respectively, and calculate the vector similarity based on the word vectors.

[0025] In this embodiment, the matching degree between query terms and candidate terms is evaluated using both character similarity and vector similarity, achieving multi-level accurate scoring from literal to semantic perspectives. Character similarity, based on word set overlap and word order weights, effectively filters out noisy candidates with similar shapes but irrelevant semantics, ensuring accuracy at the literal level. Vector similarity, on the other hand, uses pre-trained models (such as BGE) to map short texts to a semantic space, capturing deep semantic relationships and solving semantic drift problems such as synonyms, abbreviations, and full names that traditional character matching cannot identify. The complementary integration of these two approaches avoids the superficiality of single character matching and compensates for the insufficient discriminative power of pure vector matching in short text scenarios, significantly improving recall accuracy and robustness, while providing a reliable multi-dimensional scoring basis for subsequent hierarchical ranking.

[0026] S13. Obtain the matching score between the query term and each candidate term through character similarity and vector similarity; based on the matching score, determine the target standard term corresponding to the query term among the candidate terms.

[0027] In this embodiment, a comprehensive matching score between the query term and each candidate term is calculated by fusing character similarity and vector similarity, and the target standard term with the highest score is selected from the candidate set accordingly. The dual-dimensional scoring mechanism takes into account both literal accuracy and semantic relevance, effectively avoiding misjudgments or missed recalls caused by single similarity calculation; the ranking mechanism of matching scores provides quantifiable priority criteria for candidate terms, and combined with a dynamic truncation strategy, it significantly shortens the decision path while ensuring recall accuracy, achieving millisecond-level real-time response; at the same time, the results can be labeled with the matching method, improving the interpretability of the recall process, making it easier for users to understand the ranking logic, and providing high-precision input assurance for downstream NL2SQL and other services.

[0028] In this embodiment, when the matching score of any candidate word is greater than or equal to a preset reliability threshold, the candidate word is used as the target standard word; when the matching scores of all candidate words are less than the preset reliability threshold, a preset number of candidate words are selected as target standard words in descending order of matching score value. If the corresponding standard word is obtained based on the word list filtering, the matching result is directly output. Otherwise, based on semantic initial screening, the remaining candidates are used, and the matching score is calculated using different weighting methods (such as 0.4×simvec+0.6×J′) as the final ranking score, and the Top N (such as N=5) standard words are output.

[0029] Furthermore, a dynamic truncation scheme can be used to determine the target standard words: if the score of the first result is ≥0.95 (high confidence), return the first result directly; or if the standard word is accurately matched directly through the thesaurus, return the matching result directly. Otherwise, return the Top 5, balancing efficiency and recall quality.

[0030] In this embodiment, a standard vocabulary containing standard business terms is pre-constructed. The user's query terms are then sequentially subjected to synonym matching and pinyin correction to generate a candidate word set. Finally, a dual-dimensional matching score combining character similarity and vector similarity is used to determine the target standard word. This solution integrates the advantages of rule-based matching and semantic matching: synonym matching and pinyin correction solve fuzzy input problems such as homophones and near-synonyms; character similarity ensures accurate filtering at the literal level; and vector similarity captures deep semantic relationships. The three work together to effectively improve the accuracy and robustness of short text retrieval. Simultaneously, the hierarchical computation strategy controls computational overhead while ensuring retrieval quality, meeting real-time retrieval needs and supporting high-precision standard word localization across multiple scenarios.

[0031] Please see Figure 2 , Figure 2 This is the second illustration of a short text retrieval process based on business standard terms provided in this public disclosure, such as... Figure 2 As shown, the recall methods include: S21. Collect user query logs in real time and extract valid query short texts to form a short text set.

[0032] In this embodiment, user query logs from the business system are collected in real time (such as natural language input in NL2SQL scenarios), invalid input is filtered (such as garbled characters or duplicate queries), and a set of valid short texts Q={ , ,..., }

[0033] In this embodiment, user query logs from the business system are collected in real time. After cleaning to remove invalid data such as garbled characters and duplicate queries, valid short query texts are extracted and compiled into a short text set, which serves as the data foundation for subsequent thesaurus mining and vocabulary expansion. The log data directly reflects users' real query habits and can naturally capture emerging terms, industry abbreviations, and vague expressions (such as "Smart Energy Department") that frequently appear in business scenarios but are not included in the thesaurus. This provides a high-value data source for the adaptive updating of the thesaurus. The real-time collection mechanism ensures that the thesaurus keeps up with the dynamic changes in business terms (such as the addition of new branches and the iteration of business names). New words can be detected without manual intervention, effectively solving the problem of "missed new word recall" caused by the lag in manual maintenance of traditional static thesauruses. At the same time, the cleaning operation ensures the quality of input data and avoids noise interference with the accuracy of subsequent co-occurrence analysis.

[0034] S22. Match the query short texts in the short text set with the standard words in the standard word list, and determine the standard words that match the short texts with a preset matching degree as potential synonyms.

[0035] In this embodiment, the user query With standard lexicon S={ , ,..., Perform preliminary matching (such as coarse screening based on an edit distance threshold of 0.7) to filter out "fuzzy queries" that partially match multiple standard words (such as a user inputting "Smart Energy Department" may match multiple standard words such as "Smart Energy Business Unit" and "Energy Business Unit").

[0036] In this embodiment, the cleaned set of query short texts is matched one by one with the standard vocabulary. Based on a preset matching degree threshold (e.g., edit distance 0.7), fuzzy queries that partially match multiple standard words are filtered out, and their corresponding standard words are marked as potential synonyms. The preset matching degree threshold achieves coarse filtering, avoiding the computational redundancy caused by full matching, and quickly locks the fuzzy mapping relationship of "one word with multiple meanings" or "one meaning with multiple words" (e.g., "Smart Energy Department" matches "Smart Energy Business Unit" and "Energy Business Unit"), providing a high-quality candidate foundation for subsequent precise synonym mining based on co-occurrence frequency. At the same time, the threshold control effectively filters out completely irrelevant standard words, reduces noise interference in subsequent Apriori co-occurrence analysis, and improves the accuracy and efficiency of potential synonym identification.

[0037] S23. Count the number of co-occurrences between the query short text and potential synonyms, and identify candidate synonyms among the potential synonyms.

[0038] In this embodiment, the number of co-occurrences of fuzzy queries and standard words (such as the frequency of "Smart Energy Department" and "Smart Energy Business Unit" appearing simultaneously in the log) is counted, and the Apriori algorithm is used to mine frequent co-occurrence pairs (support ≥ 5%, confidence ≥ 80%) to identify potential synonyms (such as "Smart Energy Department" and "Smart Energy Business Unit").

[0039] In this embodiment, the co-occurrence frequency of user query texts and potential synonyms in logs is statistically analyzed. The Apriori algorithm is then used to mine frequent co-occurrence pairs that meet conditions such as support ≥ 5% and confidence ≥ 80%, from which candidate synonyms are selected. The number of co-occurrences directly reflects the actual correlation strength between the ambiguous expression and the standard word in the user's real query. Compared to manual pre-definition, it can more objectively identify the true synonym mapping relationship (such as "Smart Energy Department" and "Smart Energy Business Unit"). The introduction of the Apriori algorithm ensures statistical significance in synonym mining, effectively filtering out noise interference from occasional co-occurrences, significantly improving the accuracy and reliability of candidate synonym identification, and providing a high-confidence candidate basis for the automated updating of the synonym list.

[0040] S24. Calculate the candidate confidence scores of the query short text and candidate synonyms, and select candidate synonyms to be added to the standard thesaurus based on the candidate confidence scores.

[0041] In this embodiment, the candidate similarity between the query short text and the candidate synonyms can be calculated as the candidate confidence score; when the candidate similarity meets the first preset threshold range, the candidate synonyms are added to the standard vocabulary; when the candidate similarity meets the second preset threshold range, the candidate synonyms are added to the manual review queue, and the addition to the standard vocabulary is confirmed based on the review results; the second preset threshold range is smaller than the first preset threshold range.

[0042] In this embodiment, vector similarity and pinyin / edit distance can be used as candidate similarities. If the candidate confidence score is greater than the automatic threshold (e.g., 0.95), it is automatically added to the vocabulary; if it is in the middle confidence interval, it enters the manual review queue to ensure the accuracy of the vocabulary. The updated thesaurus supports real-time synchronization to subsequent matching modules without manual intervention.

[0043] In this embodiment, the candidate confidence score (combining vector similarity, pinyin / edit distance, and other multi-dimensional indicators) between the query text and candidate synonyms is calculated. A preset automatic threshold (e.g., 0.95) is used to determine whether to directly add the synonym to the standard vocabulary. Synonyms with intermediate confidence scores are then added to a manual review queue, thus achieving automated screening and database entry for synonyms. The fusion of multi-dimensional indicators ensures the accuracy of candidate synonym determination and avoids low-quality words polluting the vocabulary. The automatic threshold mechanism enables real-time database entry of high-confidence synonyms with zero manual intervention, significantly improving vocabulary update efficiency and effectively solving the "new word omission" problem caused by the lag in manual maintenance of traditional static vocabulary. The manual review process balances accuracy and security, ensuring the reliability of vocabulary expansion and providing a continuously updated, high-quality synonym foundation for subsequent recall. Please see Figure 3 , Figure 3 This is the third illustration of a short text retrieval process based on business standard terms, as provided in this publication. Figure 3 As shown, the recall methods include: S31. Perform synonym matching between the query term and the standard terms in the standard term list, and determine the standard terms that match the query term as the matching synonyms.

[0044] In this embodiment, by performing synonym matching between a user query word and a labeled and classified business standard vocabulary (i.e., the standard vocabulary), the successfully matched standard words are used as matching synonyms and output to a candidate word set. The standard vocabulary achieves domain adaptation through labeled classification (such as customer information, territorial information, etc.), so that synonym matching is no longer limited to general dictionaries, but is deeply combined with industry specifications of business scenarios (such as hierarchical relationship of enterprise names, territorial attribution, etc.), which effectively solves the problem that the recall results of traditional synonym matching are disconnected from actual scenarios due to the lack of combination with business rules; meanwhile, as the first-layer lightweight filtering in the recall process, it quickly locates relevant standard words from a million-level vocabulary, greatly narrows the candidate range for subsequent character similarity and vector similarity calculation, improves recall accuracy while significantly reducing computational overhead, and lays a foundation for millisecond-level real-time response.

[0045] S32, performing polyphonic character discrimination on the query word and each matching synonym, and converting the query word and the matching synonyms into corresponding pinyin according to the business domain.

[0046] In this embodiment, domain-aware disambiguation is performed on polyphonic characters in the query word and each matching synonym, and the correct pronunciation of the polyphonic character is determined in combination with business scenario context (such as finance, energy, etc.), and then the query word and the matching synonyms are uniformly converted into corresponding pinyin. Polyphonic character discrimination effectively solves the pinyin mapping ambiguity problems caused by polyphonic characters such as "重行" (chóng / zhòng háng / xíng) and "乐清" (yuè / lè qīng); the introduction of business domain enables pinyin conversion to have scenario adaptive capability, ensuring that the same word is correctly transcribed under different services (for example, "行长" is pronounced as háng zhǎng instead of xíng cháng in the financial field), which provides a high-precision pronunciation benchmark for subsequent pinyin error correction, greatly reduces the risk of missing recall of homophonic heterographs caused by misjudgment of polyphonic characters, and significantly improves the robustness and accuracy of short text recall in fuzzy input scenarios.

[0047] In this embodiment, a polyphonic character dictionary for industry domains is constructed (for example, "行长" is pronounced "háng zhǎng" in the financial field, and "长城" is pronounced "cháng chéng" in the geographical field), and the correct pronunciation is selected according to the potential domain of the user's query during pinyin conversion. For example, when a user inputs "长城", the system detects an industry keyword of "finance" and determines that it refers to "Great Wall Securities" instead of a geographical name.

[0048] S33, calculating the semantic similarity between the pinyin of the matching synonyms and the pinyin of the query word, performing pinyin error correction through the semantic similarity, and using the matching synonyms whose semantic similarity meets a preset semantic similarity threshold as candidate words corresponding to the query word.

[0049] In this embodiment, for candidate words converted from Pinyin (e.g., "gāo gāng" corresponds to "高刚" and "高岗"), a pre-trained model (e.g., BERT) is used to calculate the semantic similarity between the candidate word and the user's original input (for example, the semantic similarity between "高建钢" and "高刚" is 0.92, and that between "高建钢" and "高岗" is 0.21). Only candidates with high similarity are retained for subsequent processes, reducing the false recall of homophones with different meanings.

[0050] S34, forming a candidate word set from each candidate word.

[0051] In this embodiment, by calculating the semantic similarity between the Pinyin of matched synonyms and the Pinyin of the query word (e.g., cosine distance based on Pinyin vectors), the similarity measure is used to perform error correction matching at the Pinyin level, and matched synonyms that meet the conditions are screened out as candidate words through a preset semantic similarity threshold. Compared with the limitation that traditional exact Pinyin matching can only handle complete homophones, semantic similarity calculation maps Pinyin to a vector space, which can effectively capture fuzzy inputs including homophones with different characters (e.g., "高建钢" and "高刚" are both pronounced gāo jiàn gāng / gāng), near-phonetic substitution (e.g., "部署" is incorrectly input as "步署"), and even fuzzy inputs with partial consistency of initials and finals, greatly expanding the coverage of Pinyin error correction; meanwhile, the introduction of the semantic similarity threshold effectively filters out mismatches with large pronunciation differences, avoiding the "over-correction" problem caused by pure Pinyin matching, and significantly improving the robustness and recall coverage of short texts in fuzzy scenarios such as voice input and manual typo input on the premise of ensuring recall accuracy.

[0052] Please refer to Figure 4 , Figure 4 is the fourth schematic diagram of the short text recall process based on business standard words provided by the present disclosure, for example, as shown in Figure 4 , the recall method includes: S41, adding position weights to each word in the query word and candidate words respectively according to the word order.

[0053] In this embodiment, a position-sensitive weighting mechanism is introduced into character similarity calculation by assigning differentiated position weights (e.g., an exponential decreasing strategy of 0.4 for the first word, 0.3 for the second word, and 0.1 for the last word) to each word in the query term and candidate terms according to their position in the word sequence (e.g., first, second, and last). The position of a word in a short text is strongly correlated with its semantic importance. The first word usually carries core business entity information (e.g., "national" is more distinctive than "company" in "State Grid Corporation of China"). Position weights allow similarity calculation to focus on key semantic segments rather than treating all characters evenly, effectively distinguishing pseudo-matches with different word orders but the same word set (e.g., "China Telecom Beijing Branch" and "Beijing China Telecom Branch"). This significantly improves the distinguishing ability and accuracy of character similarity in multi-word combination scenarios. At the same time, position weights complement character similarity and vector similarity, making the final matching score more closely match the user's true query intent, significantly reducing the risk of false recalls caused by word order disturbances, and providing fine-grained position-aware assurance for accurate matching of short texts.

[0054] In this embodiment, the traditional Jaccard coefficient only calculates the overlap ratio of word sets (e.g., the overlap between the word sets of "Shanghai Financial Technology Company" and "Shanghai Technology Company Financial Department" is 2 / 3), ignoring word order and semantics.

[0055] S42. Based on position weight, calculate the intersection ratio of the word sets of the query term and each word in the candidate terms, and use it as the first character similarity.

[0056] In this embodiment, a positional weight is assigned to each word (e.g., the first word has a weight of 1.5, the middle word 1.0, and the last word 1.2), and the weighted intersection ratio of the word sets is calculated. For example, the word order weight vector of "Shanghai Financial Technology Company" is [1.5, 1.0, 1.0], and that of "Shanghai Technology Company Financial Department" is [1.5, 1.0, 1.2], increasing the weighted intersection ratio to 0.85 (the original Jaccard value was 0.67).

[0057] S43. Calculate the semantic similarity between the query term and the candidate term, and correct the first character similarity using the semantic similarity to obtain the corrected character similarity.

[0058] In this embodiment, the semantic similarity between the query term and the candidate term (e.g., cosine similarity based on word vectors) is calculated, and used as a adjustment factor to weight and correct the first character similarity obtained previously based on the character level, thereby obtaining a corrected character similarity that fuses semantic information. Pure character similarity has inherent blind spots when facing scenarios where literal forms are similar but semantics are completely different (e.g., "Apple phone" and "apple fruit") or literal forms are greatly different but semantics are highly consistent (e.g., "mobile communication" and "mobile phone signal"). The introduction of semantic similarity provides high-level semantic calibration for character matching: when semantics are highly consistent, the weight of character similarity can be increased to enhance recall confidence; when semantics are obviously conflicting, character similarity can be suppressed to avoid misjudgment. This effectively compensates for the deficiency of single character matching in scenarios with semantic ambiguity, so that the corrected character similarity has both literal accuracy and semantic consistency, and significantly improves the discrimination accuracy and robustness of short text matching in complex scenarios such as polysemous words, synonyms and cross-domain expressions.

[0059] In this embodiment, a short text optimized pre-trained model is used to calculate the semantic similarity simsem between the user input and the standard term, and the final Jaccard score is adjusted to J′=J×(1+simsem), where J is the original Jaccard coefficient. For example, the semantic similarity of the two above-mentioned examples is 0.6 (with large difference in service attributes), after correction, J′=0.67×(1+0.6)=1.07, and the actual score after normalization is 0.82, which reduces misjudgment.

[0060] S44, converting the query term and the candidate term into pinyin, and calculating the intersection ratio of the pinyin word sets of the query term and the candidate term, which is taken as the second character similarity.

[0061] In this embodiment, by uniformly converting the query term and the candidate term into pinyin, splitting the pinyin into word sets according to words, and calculating the intersection ratio between the two word sets (e.g., number of intersecting words / number of union words) as the second character similarity, which provides an independent similarity measurement for character matching from the pronunciation level. The intersection ratio of pinyin word sets is inherently insensitive to word order, and can effectively cope with common scenarios such as reversed word order and partial missing in user input. Compared with the strict mode of bit-by-bit comparison of the first character similarity, the second character similarity captures the association at the pronunciation level in a looser word set overlapping manner, greatly improving the tolerance for fuzzy input such as homophones with different forms, near-phonetic substitution and word order disturbance; meanwhile, as an independent measurement dimension complementary to the first character similarity, the combination of the two can effectively distinguish ambiguous scenarios of "similar literal form but different pronunciation" (e.g., "xíng" and "háng") and "different literal form but same pronunciation" (e.g., "bù shǔ" written as "bù shǔ" with different Chinese characters), which significantly enhances the recall coverage and discrimination robustness of short text matching in noisy environments such as voice input and manual typo input.

[0062] In this embodiment, a set of pinyin words is generated for the user input and the standard words respectively. The pinyin Jaccard coefficient Jpinyin is calculated and averaged with the corrected text Jaccard coefficient J′ to obtain the final character set matching score, which further enhances the fault tolerance.

[0063] S45. Combine the corrected character similarity and the second character similarity to obtain the character similarity between the query term and the candidate term.

[0064] S46. Calculate the character similarity between the query term and each candidate term in the candidate term set.

[0065] In this embodiment, a weighted synthesis of modified character similarity (emphasizing precise word-by-word matching) that integrates semantic calibration and positional weights is obtained, along with a second character similarity based on the intersection of pinyin word sets (emphasizing loose word set overlap at the pronunciation level). This yields a final character similarity that takes into account multi-dimensional information. The two similarity measures complement each other efficiently from different perspectives. Modified character similarity provides high discriminative judgment in scenarios with precise word order and semantic consistency, while second character similarity provides strong tolerance and recall in ambiguous scenarios such as word order reversal, homophones, and partial omissions. The fusion of the two effectively overcomes the inherent blind spots of a single similarity measure (such as pure character matching being unable to handle word order perturbations and pure pinyin word sets being unable to distinguish semantic conflicts). This allows the final character similarity to possess both literal-level accuracy and pronunciation-level robustness, significantly improving the matching accuracy and recall coverage of short texts in various noisy scenarios such as typos, speech transcription, and colloquial expressions.

[0066] Please see Figure 5 , Figure 5 This is the fifth illustration of a short text retrieval process based on business standard terms provided in this public disclosure, such as... Figure 5 As shown, the recall methods include: S51. Convert the query terms and candidate terms into query term vectors and candidate term vectors of specific dimensions, respectively.

[0067] In this embodiment, a pre-trained word vector model (such as Word2Vec, BERT, FastText, etc.) maps the query word and candidate words to a dense vector space of a unified specific dimension (such as 128-dimensional, 256-dimensional, or 768-dimensional), generating query word vectors and candidate word vectors, thereby transforming discrete text symbols into continuous semantic vector representations. Vectorization breaks through the rigid dependence of traditional character matching on complete literal consistency, enabling the capture of word pairs with significant literal differences but high semantic relevance in a continuous vector space. This effectively solves the semantic blind spot problem in scenarios such as synonyms, near-synonyms, and cross-domain expressions. Simultaneously, the unified dimension and fixed-length vectors provide standardized input for subsequent vector calculations such as cosine similarity, ensuring that words of different lengths and granularities can be directly compared in the same space, significantly improving computational efficiency and consistency. Furthermore, the pre-trained vectors naturally aggregate contextual semantic information from large-scale corpora, giving them a certain degree of polysemy disambiguation capability, providing a deep semantic understanding foundation for short text matching, and significantly enhancing the robustness and accuracy of recall in scenarios with fuzzy expressions and colloquial input.

[0068] In this embodiment, a pre-trained model optimized for short texts is selected (such as adding positional encoding to BERT to enhance word order information). After inputting short text, it outputs semantic vectors vq (user input) and vs (standard words) with fixed dimensions (such as 768 dimensions).

[0069] S52. Dimensionality reduction is performed on the query word vector and candidate word vector using principal component analysis.

[0070] In this embodiment, PCA (Principal Component Analysis) is used to reduce the dimensionality of high-dimensional vectors to 256 dimensions, reducing computational redundancy; simultaneously, the dimensionality-reduced vectors are stored in a vector library. Principal Component Analysis (PCA) is an unsupervised linear dimensionality reduction technique. After centering the original high-dimensional data matrix, the covariance matrix is ​​calculated. By solving for the eigenvalues ​​and eigenvectors of this covariance matrix, the top k principal components whose cumulative variance contribution rate reaches a preset threshold (e.g., 95%) after sorting the eigenvalues ​​from largest to smallest are selected as new orthogonal bases. The original high-dimensional data is then linearly projected onto the k-dimensional principal component space, thereby achieving data dimensionality reduction.

[0071] S53. Calculate the cosine similarity between the query word vector and the candidate word vectors, and use it as the vector similarity.

[0072] S54. Calculate the vector similarity of the query term and each candidate term in the candidate term set.

[0073] In this embodiment, cosine similarity is used to calculate the vector similarity between the user input and the standard words, with the output value ranging from [0,1]. A higher value indicates a greater semantic relevance. By mapping the query word vector generated in the previous steps to the vectors of each candidate word in the candidate word set to the same high-dimensional semantic space, the cosine value of the angle between the query word vector and each candidate word vector is calculated one by one using vector distance measurement methods such as cosine similarity, thereby obtaining the vector similarity score corresponding to each candidate word. Vector similarity measures the degree of association between query words and candidate words at a deep semantic level. It can effectively capture matching pairs that have large literal differences but high semantic consistency, making up for the inherent shortcomings of character similarity and pinyin similarity in scenarios such as cross-word expression and abbreviation matching. At the same time, as the final semantic verification layer of the recall link, vector similarity has a natural ability to distinguish pseudo-matches that are literally similar but semantically conflicting. It effectively suppresses the semantic noise that may be introduced by the preceding character-level and pinyin-level similarity, so that the final recall results have strong semantic consistency while ensuring high coverage. It significantly improves the discrimination accuracy and intent alignment ability of short text matching in complex scenarios such as abbreviation input, alias query, and colloquial expression.

[0074] Please see Figure 6 , Figure 6 This disclosure provides a short text retrieval system based on business standard terms. The retrieval system includes: a word list filtering module 11, a semantic coarse screening module 12, and a fine sorting output module 13.

[0075] In this embodiment, the word filtering module 11 is used to perform synonym matching and pinyin correction on the query words input by the user based on a pre-built standard word list containing business standard words, to obtain candidate words for the corresponding query words and form a candidate word set.

[0076] In this embodiment, the semantic coarse screening module 12 is used to calculate the character similarity between the query term and each candidate word in the candidate word set; convert the query term and each candidate word in the candidate word set into word vectors respectively, and calculate the vector similarity based on the word vectors.

[0077] In this embodiment, the fine-ranking output module 13 is used to obtain the matching score between the query term and each candidate term through character similarity and vector similarity; and to determine the target standard term corresponding to the query term among the candidate terms based on the matching score.

[0078] In this embodiment, the recall system further includes: a standard vocabulary adaptive expansion module, used to collect user query logs in real time, extract valid query short texts to form a short text set; match the query short texts in the short text set with standard words in the standard vocabulary, determine the standard words with a matching degree of a preset matching degree as potential synonyms; count the number of co-occurrences of query short texts and potential synonyms, determine candidate synonyms among potential synonyms; calculate the candidate confidence of query short texts and candidate synonyms, and select candidate synonyms to be added to the standard vocabulary based on the candidate confidence.

[0079] In this embodiment, the standard vocabulary adaptive expansion module is specifically used to calculate the candidate similarity between the query short text and the candidate synonyms as the candidate confidence score; when the candidate similarity meets the first preset threshold range, the candidate synonyms are added to the standard vocabulary; when the candidate similarity meets the second preset threshold range, the candidate synonyms are added to the manual review queue, and the review results are used to confirm whether to add them to the standard vocabulary; the second preset threshold range is smaller than the first preset threshold range.

[0080] In this embodiment, the word filtering module 11 is specifically used to perform synonym matching between the query word and the standard words in the standard word list, and determine the standard words that match the query word as matching synonyms; to perform polyphonic character identification on the query word and each matching synonym, and convert the query word and matching synonyms into corresponding pinyin according to the business domain; to calculate the semantic similarity between the pinyin of the matching synonym and the pinyin of the query word, to perform pinyin error correction through semantic similarity, and to select matching synonyms whose semantic similarity meets the preset semantic similarity threshold as candidate words for the corresponding query word; and to form a candidate word set through each candidate word.

[0081] In this embodiment, the semantic coarse screening module 12 is specifically used to add positional weights to each word in the query term and candidate terms according to word order; based on the positional weights, calculate the intersection ratio of the word sets of the query term and each word in the candidate terms as the first character similarity; calculate the semantic similarity between the query term and the candidate terms, and correct the first character similarity through semantic similarity to obtain the corrected character similarity; convert the query term and candidate terms into pinyin, calculate the intersection ratio of the word sets of the pinyin of the query term and the candidate terms as the second character similarity; combine the corrected character similarity and the second character similarity to obtain the character similarity between the query term and the candidate terms; and calculate the character similarity between the query term and each candidate term in the candidate term set.

[0082] In this embodiment, the semantic coarse screening module 12 is specifically used to convert the query term and candidate term into query term vectors and candidate term vectors of specific dimensions, respectively; perform dimensionality reduction processing on the query term vectors and candidate term vectors by principal component analysis; calculate the cosine similarity between the query term vectors and candidate term vectors as vector similarity; and calculate the vector similarity of each candidate term in the query term and candidate term set, respectively.

[0083] In this embodiment, the fine-ranking output module 13 is specifically used to select a candidate word as the target standard word when the matching score of any candidate word is greater than or equal to a preset confidence threshold; and to select a preset number of candidate words as the target standard words in descending order of matching score values ​​when the matching scores of all candidate words are less than the preset confidence threshold.

[0084] like Figure 7 As shown, this embodiment of the present disclosure provides an electronic device, including a processor 1110, a communication interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140. Memory 1130 is used to store computer programs; When the processor 1110 executes the program stored in the memory 1130, it implements any of the above methods.

[0085] The electronic device provided in this embodiment of the present disclosure includes a processor 1110 that executes a program stored in a memory 1130 to perform synonym matching and pinyin correction on a user-input query word based on a pre-built standard word list containing business standard words, thereby obtaining candidate words for the corresponding query word and forming a candidate word set; calculating the character similarity between the query word and each candidate word in the candidate word set; converting the query word and each candidate word in the candidate word set into word vectors respectively, and calculating the vector similarity based on the word vectors; obtaining the matching score between the query word and each candidate word through character similarity and vector similarity; and determining the target standard word corresponding to the query word from the candidate words based on the matching score.

[0086] The communication bus 1140 mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 1140 can be divided into an address bus, a data bus, and a component bus, etc. For ease of illustration, it is represented by only one thick line in the figure, but this does not indicate that there is only one bus or one type of bus.

[0087] The communication interface 1120 is used for communication between the above-mentioned electronic device and other devices.

[0088] The memory 1130 may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 1130 may also be at least one storage device located remotely from the aforementioned processor 1110.

[0089] The processor 1110 mentioned above can be a general-purpose processor 1110, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0090] This disclosure provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors 1110 to implement the methods of any of the above embodiments.

[0091] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this disclosure is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0092] Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A short text retrieval method based on business standard terms, characterized in that, The recall method includes: Based on a pre-built standard vocabulary list containing standard business terms, synonym matching and pinyin correction are performed on the query terms input by the user to obtain candidate terms corresponding to the query terms, forming a candidate term set; Calculate the character similarity between the query term and each candidate word in the candidate word set; convert the query term and each candidate word in the candidate word set into word vectors respectively, and calculate the vector similarity based on the word vectors; The matching scores between the query term and each candidate term are obtained through the character similarity and the vector similarity; based on the matching scores, the target standard term corresponding to the query term is determined from the candidate terms.

2. The recall method according to claim 1, characterized in that, The recall method also includes: Collect user query logs in real time and extract valid query texts to form a short text set; The query short texts in the short text set are matched with the standard words in the standard word list to determine the standard words that have a preset matching degree with the short texts, and these standard words are used as potential synonyms. Count the number of times the query text and the potential synonyms co-occur, and determine candidate synonyms from the potential synonyms; Calculate the candidate confidence scores between the query text and the candidate synonyms, and select candidate synonyms to be added to the standard thesaurus based on the candidate confidence scores.

3. The recall method according to claim 2, characterized in that, The calculation of the candidate confidence scores between the query text and the candidate synonyms, and the selection of candidate synonyms to be added to the standard thesaurus based on the candidate confidence scores, includes: Calculate the candidate similarity between the query short text and the candidate synonyms, and use it as the candidate confidence score; When the candidate similarity meets the first preset threshold range, the candidate synonym is added to the standard vocabulary. When the candidate similarity meets the second preset threshold range, the candidate synonym is added to the manual review queue, and the review results are used to determine whether to add it to the standard word list; the second preset threshold range is smaller than the first preset threshold range.

4. The recall method according to claim 1, characterized in that, The process involves using a pre-built standard vocabulary list containing standard business terms to perform synonym matching and pinyin correction on the user-input query terms, resulting in candidate words corresponding to the query terms, forming a candidate word set, including: The query term is matched with the standard terms in the standard term list to determine the standard terms that match the query term as the matching synonyms; The query term and each of the matching synonyms are identified as having multiple pronunciations. Based on the business domain, the query term and the matching synonyms are converted into their corresponding pinyin. Calculate the semantic similarity between the pinyin of the matching synonym and the pinyin of the query word, perform pinyin correction based on the semantic similarity, and select matching synonyms whose semantic similarity meets the preset semantic similarity threshold as candidate words for the corresponding query word; The candidate words are combined to form a candidate word set.

5. The recall method according to claim 1, characterized in that, The calculation of the character similarity between the query term and each candidate term in the candidate term set includes: Add positional weights to each word in the query term and the candidate terms according to their word order; Based on the position weight, the intersection ratio of the word sets of the query word and each word in the candidate words is calculated as the first character similarity; Calculate the semantic similarity between the query term and the candidate term, and correct the first character similarity using the semantic similarity to obtain the corrected character similarity; Convert the query term and the candidate term into pinyin, calculate the intersection ratio of the word sets of the pinyin of the query term and the candidate term, and use it as the second character similarity. By combining the corrected character similarity and the second character similarity, the character similarity between the query term and the candidate term is obtained; The character similarity between the query term and each candidate term in the candidate term set is calculated.

6. The recall method according to claim 1, characterized in that, The query term and each candidate term in the candidate term set are converted into word vectors, and the vector similarity is calculated based on the word vectors, including: The query terms and candidate terms are respectively converted into query term vectors and candidate term vectors of specific dimensions; The query word vector and candidate word vector are reduced in dimensionality using principal component analysis. Calculate the cosine similarity between the query word vector and the candidate word vector, and use it as the vector similarity. The vector similarity between the query term and each candidate term in the candidate term set is calculated respectively.

7. The recall method according to any one of claims 1 to 6, characterized in that, The step of determining the target standard word corresponding to the query word from the candidate words based on the matching score includes: When the matching score of any candidate word is greater than or equal to a preset confidence threshold, the candidate word is used as the target standard word. When the matching scores of the candidate words are all less than a preset confidence threshold, a preset number of the candidate words are selected as the target standard words in descending order of matching scores.

8. A short text retrieval system based on business standard terms, characterized in that, The recall system includes: The word filtering module is used to perform synonym matching and pinyin correction on the query words input by the user based on a pre-built standard word list containing business standard words, and to obtain candidate words corresponding to the query words, forming a candidate word set; A semantic coarse screening module is used to calculate the character similarity between the query term and each candidate word in the candidate word set; convert the query term and each candidate word in the candidate word set into word vectors respectively, and calculate the vector similarity based on the word vectors; The fine-ranking output module is used to obtain the matching score between the query term and each of the candidate terms through the character similarity and the vector similarity; and to determine the target standard term corresponding to the query term among the candidate terms based on the matching score.

9. An electronic device, characterized in that, include: processor; as well as A memory storing computer-executable instructions, which, when executed, cause the processor to perform the method according to any one of claims 1-7.

10. A computer storage medium, characterized in that, in, The computer storage medium stores one or more programs that, when executed by a processor, implement the method of any one of claims 1-7.

Images