Chinese semantic enhanced fuzzy retrieval method and system based on adaptive weight

Abstract

The invention relates to the technical field of information retrieval, in particular to a Chinese semantic enhanced fuzzy retrieval method and system based on adaptive weight, and aims to solve the problems that an existing fuzzy retrieval method cannot balance the recall rate and the accuracy rate, semantic weight quantization is missing, and sorting is inaccurate. The method comprises the following steps: performing semantic analysis on a query condition; calculating a part-of-speech weight value of each query segmented word in combination with a predefined part-of-speech weight rule and a part-of-speech combination rule; determining a domain weight value associated with the query condition based on preset domain or context information; applying a correlation scoring algorithm, combining the part-of-speech weight value and the domain weight value as weighting factors, and calculating correlation scores of the candidate documents; and sorting and outputting the candidate documents according to the scoring result. According to the method, double dynamic weights of part-of-speech combination and the industry field are introduced, the understanding ability of Chinese semantics is enhanced through a dynamic weight learning mechanism, and the correlation and accuracy of fuzzy retrieval results are effectively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of information retrieval, and particularly relates to a Chinese semantic enhanced fuzzy retrieval method and system based on adaptive weight. BACKGROUND

[0002] With the explosive growth of digital information, the demand for fuzzy retrieval of long text information (such as video names, document titles, and commodity detail titles) by users is increasingly urgent. For example, in a video platform, a user inputs a long text query target video such as "music art in mathematics", which requires both "character-level fuzzy matching" and "semantic-level intent accurate capture", but the existing retrieval technology still has significant defects in this scenario.

[0003] Traditional fuzzy queries based on relational databases usually use wildcard matching, which is simple to implement but has obvious limitations. Right fuzzy matching with "keyword%" uses B-tree indexing, but only matches the beginning text of the keyword. Left fuzzy query with "%keyword" cannot effectively use database indexing, resulting in low query performance. Full fuzzy matching with "%keyword%" is completely based on character matching, has very low recall rate, and does not consider the importance of vocabulary semantics. Traditional full-text retrieval query technology (such as Elasticsearch) has significantly improved query efficiency and recall rate through mechanisms such as inverted indexing, but its core is still based on statistical features such as term frequency (TF) and inverse document frequency (IDF), which lack the ability to distinguish the semantic importance of query words, resulting in deviations between the sorting results and the user's true intent.

[0004] To make up for the above defects, the industry has proposed various optimization schemes, but there are still some problems: query expansion (such as adding synonyms) can introduce noise and reduce accuracy; query rewriting has high complexity, is easy to deviate from the user's intent, and is only suitable for correcting input errors; sorting learning relies on a large amount of data and user privacy, the model is black-box and cannot optimize the recall stage; vector retrieval confuses character and semantic fuzziness (such as finding "artificial intelligence" by searching for "artificial intelligence"), has poor domain adaptability and vectorization delay, and is difficult to support real-time queries.

[0005] In summary, existing technologies still face challenges in balancing recall rate and accuracy, and have limitations in quantifying the semantic importance of query words, which may lead to deviations between the sorting results and the user's true intent. Therefore, there is an urgent need for a fuzzy retrieval method that quantifies semantic importance, optimizes multiple indicators, and unifies high adaptability to meet the actual needs of long text fuzzy queries. SUMMARY

[0006] ​In order to solve the above problems in the prior art, i.e., the existing fuzzy retrieval method cannot balance the recall rate and the accuracy rate, the missing semantic weight quantization, and the inaccurate sorting, the first aspect of the present application provides a Chinese semantic enhanced fuzzy retrieval method based on adaptive weight, which comprises the following steps: obtaining a query condition input by a user, performing semantic analysis on the query condition, and generating a query word sequence composed of at least one query word; determining a part-of-speech weight value corresponding to each query word based on the part-of-speech feature and / or the part-of-speech combination feature of the query word sequence; determining a domain weight value associated with the query word based on the preset domain attribution or the context domain information of the query condition; applying a relevance score algorithm to any document in a candidate document set, combining the part-of-speech weight value and the domain weight value as weighting factors, and calculating the relevance score of the document and the query condition; sorting the candidate document set based on the relevance score, and outputting the sorting result as a retrieval result.

[0007] In some preferred embodiments, the method for determining the part-of-speech weight value corresponding to each query word comprises: performing part-of-speech tagging on each query word, and obtaining an initial part-of-speech weight; identifying whether one or more preset part-of-speech combination features exist in the query word sequence; when any query word and its adjacent query word constitute a preset part-of-speech combination feature, determining the center word of the part-of-speech combination feature based on linguistic features, and increasing the initial part-of-speech weight of the center word as the part-of-speech weight value; the part-of-speech combination feature comprises at least one of the following: continuous noun combination, continuous time noun combination, adjacent relationship between special noun and ordinary noun, and modification relationship between adverb / adjunct and verb.

[0008] In some preferred embodiments, the method for determining the center word of the part-of-speech combination feature based on linguistic features comprises: when the part-of-speech combination feature is continuous noun combination, determining the postposed noun as the center word based on the syntax structure of the centering phrase; when the part-of-speech combination feature is continuous time noun combination, determining both time nouns as the center word based on the recursive nesting property of the time adverbial; when the part-of-speech combination feature is the adjacent relationship between special noun and ordinary noun, and the special noun plays a semantic limiting role in the noun phrase, determining the special noun as the center word according to the centripetal principle of semantic focus; When the word combination feature is a modifying relationship between an adverb / adjective and a verb, the verb is determined as the center word based on a state-independence relationship.

[0009] In some preferred embodiments, the preset field attribution or context field information of the query condition is used to determine a field weight value associated with the query tokenization, and the method is as follows: The query condition is input into a pre-trained field classification model to automatically identify the main field and the secondary field of the industry to which it belongs, and the field weight value is determined based on the identification result. The field classification model is trained based on a field training corpus with labeled field information.

[0010] In some preferred embodiments, the field training corpus is constructed through crowdsourcing labeling, and the method is as follows: Through a graphical interactive interface, the query condition and its tokenization result in the training data to be labeled are pushed to the labeling user as a query to be labeled, and one or more core tokens that can reflect the query intent are selected and marked by the user. The user labeling result is subjected to credibility evaluation, and the credibility evaluation adopts at least one of a multi-user cross-validation, a known answer adversarial test, and a user historical accuracy evaluation mechanism. Based on the credibility evaluation result, the user labeling result that meets a preset credibility threshold in terms of credibility is selected to constitute the field training corpus.

[0011] In some preferred embodiments, after the field weight value associated with the query tokenization is determined, the method further includes: The historical query records before the current query operation of the user are obtained, and the historical query records are subjected to field analysis to extract the historical field preference of the user; at least two query records in the historical query records have common field information, and the historical field preference includes historical main field information and historical secondary field information. When multiple candidate fields are identified in the current query, the weight of the candidate field that matches the historical preference field is increased according to the historical field preference to correct the field weight value.

[0012] In some preferred embodiments, when the semantics of the query condition is analyzed, if a number token is identified, the number token and the next query token are merged into an indivisible semantic unit; if the next query token is a quantifier, the number token, the quantifier, and the query token after the quantifier are jointly merged into an indivisible semantic unit.

[0013] In some preferred embodiments, the relevance score algorithm is a standard BM25 score function, and the relevance score is calculated based on the standard BM25 score function in combination with the part-of-speech weight value and the field weight value as weighting factors, and the method is as follows: ; wherein IDF(q i ) is the inverse document frequency of the term q i , which measures the distinguishing ability thereof; a domain (q i ) is the field weight value, and the importance of the segmented words in different fields in the query is different; b pos (q i ) is the part-of-speech weight value, and the importance of the segmented words in different parts of speech in the query is different; f(q i , D) is the word frequency of the term q i in the document D; k1 and b are the classic parameters of BM25, which control the word frequency saturation and the document length normalization, is the length of the document D; and avgdl is the average document length of the document set.

[0014] In some preferred embodiments, the candidate document set is obtained from a document index library pre-constructed by a retrieval system or screened from full-text documents through preliminary matching by keywords, and the document index library is constructed based on indexing and contains segmented words and index information of Chinese texts.

[0015] According to a second aspect of the present application, a Chinese semantic enhanced fuzzy retrieval system based on adaptive weights is provided, and the system comprises: a semantic analysis module configured to acquire a query condition input by a user, perform semantic analysis on the query condition, generate a query segmented word sequence, and perform part-of-speech tagging on each query segmented word, wherein the query segmented word sequence is composed of at least one query segmented word; a weight generation module configured to determine a part-of-speech weight value corresponding to each query segmented word based on part-of-speech features and / or part-of-speech combination features of the query segmented word sequence, and determine a field weight value associated with the query segmented word based on a preset field attribution or context field information of the query condition; a relevance score module configured to calculate a relevance score of any document in a candidate document set with the query condition by extending a BM25 score function; the extended BM25 score function is based on a standard BM25 score function in combination with the part-of-speech weight value and the field weight value as core weighting factors; a result processing module configured to sort the candidate document set according to the relevance score and output the sorting result as a retrieval result.

[0016] The present application has the following beneficial effects: (1) The method of the application fuses word semantic analysis and domain perception modeling, introduces part-of-speech weight (part-of-speech weight value) reflecting the importance of words and domain weight (domain weight value) reflecting the context background on the basis of benchmark relevance scoring, and weights and modulates the benchmark score; it is suitable for Chinese unstructured text, so that the scoring process is not only simple word frequency statistics, but also incorporates the understanding of word grammatical function and domain background, improves the semantic understanding depth in the query process, and makes the result ranking closer to the user's intention; (2) Through the double-channel domain recognition and weight learning mechanism, higher weight is given to key words (such as nouns and domain terms), which can place the documents that really hit the theme in front, significantly improving the recall rate, accuracy and relevance ranking quality of the search results; (3) The ranking result is more consistent with the user's cognition and intention, reducing the user's time to filter irrelevant information, so that the user can find the required content faster and more accurately. BRIEF DESCRIPTION OF DRAWINGS

[0017] Other features, objects and advantages of the application will become more apparent after reading the detailed description of the non-limiting embodiments made with reference to the following drawings: Figure 1 is a flowchart of the Chinese semantic enhanced fuzzy retrieval method based on adaptive weight in the embodiment of the application; Figure 2 is a flowchart of part-of-speech weight model training in the embodiment of the application; Figure 3 is a flowchart of domain classification model training in the embodiment of the application; Figure 4 is a structural diagram of the Chinese semantic enhanced fuzzy retrieval system based on adaptive weight in the embodiment of the application. DETAILED DESCRIPTION

[0018] The application will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related application, and not to limit the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for ease of description.

[0019] It should be noted that the embodiments in the application and the features in the embodiments can be combined with each other without conflict. The application will be described in detail below with reference to the drawings and embodiments.

[0020] The present application aims at the problems of the fuzzy search method based on a relational database in the prior art in terms of Chinese semantic understanding, query performance and result relevance sorting, and proposes a Chinese semantic enhanced fuzzy retrieval method based on adaptive weight, constructs a high-precision and high-performance retrieval system suitable for Chinese unstructured text by fusing part-of-speech semantic analysis, domain perception modeling, combination rule optimization and machine learning dynamic weight adjustment, and improves semantic understanding depth and realizes result sorting close to user intent.

[0021] The Chinese semantic enhanced fuzzy retrieval method based on adaptive weight of the present application comprises the following steps: obtaining a query condition input by a user, performing semantic analysis on the query condition, and generating a query word sequence, wherein the query word sequence is composed of at least one query word; determining a part-of-speech weight value corresponding to each query word based on part-of-speech features and / or part-of-speech combination features of the query word sequence; determining a domain weight value associated with the query word based on a preset domain attribution or context domain information of the query condition; applying a relevance scoring algorithm to any document in a candidate document set, combining the part-of-speech weight value and the domain weight value as weighting factors, and calculating a relevance score of the document and the query condition; sorting the candidate document set based on the relevance score, and outputting a sorting result as a retrieval result.

[0022] In order to more clearly illustrate the Chinese semantic enhanced fuzzy retrieval method based on adaptive weight of the present application, the following will be described in combination with Figures 1-4 The steps in the embodiments of the present application will be described in detail.

[0023] The Chinese semantic enhanced fuzzy retrieval method based on adaptive weight of the first embodiment of the present application comprises steps S100-S500, and each step will be described in detail as follows: Step S100: obtaining a query condition input by a user, performing semantic analysis on the query condition, and generating a query word sequence, wherein the query word sequence is composed of at least one query word.

[0024] In an embodiment of the present application, obtaining a query condition input by a user and performing semantic analysis on the query condition comprises: Real-time query: obtaining user input content (including fuzzy input such as spelling errors, abbreviations and colloquial expressions) and corresponding click / conversion results; historical search logs, manually annotated data sets and vertical domain corpora can also be used as data sources to enrich the analysis model; Data cleaning: standardizing the query string, including but not limited to: removing invalid or meaningless text, unifying character case, filtering special symbols, and correcting obvious spelling errors based on dictionary or edit distance algorithm; Word segmentation processing: using a Chinese word segmentation tool (in this embodiment, HanLP is preferred) to segment the cleaned query condition to obtain a query segmentation sequence; for example, the query "music art in mathematics" is processed as [music, art, in, of, mathematics].

[0025] When analyzing the semantics of the query condition, if the query segmentation is a number, the number and the next query segmentation are combined into an indivisible semantic unit to deal with special cases such as "double first-class"; if the next query segmentation is a quantifier, the number, quantifier and the next query segmentation are combined into an indivisible semantic unit to deal with special cases such as "three squirrels".

[0026] Step S200: based on the part-of-speech features and / or part-of-speech combination features of the query segmentation sequence, determine the part-of-speech weight value corresponding to each query segmentation, the method is: Use HanLP to perform part-of-speech tagging on each query segmentation, and according to the general importance of part-of-speech in Chinese semantics, assign an initial part-of-speech weight value to different part-of-speech; When any query segmentation and its adjacent query segmentation form a preset part-of-speech combination feature, determine the center word of the part-of-speech combination feature based on linguistic features, and increase the initial part-of-speech weight of the center word as the part-of-speech weight value.

[0027] In an embodiment of the present application, the part-of-speech combination feature includes at least one of the following: Continuous noun combination, continuous time noun combination, adjacency relationship between special noun and ordinary noun, and modification relationship between adverb / adjective and verb.

[0028] Part-of-speech refers to the function or category of a word in the grammatical structure. Common part-of-speeches include nouns, verbs, adjectives, adverbs, prepositions, conjunctions, pronouns, etc. Part-of-speech is crucial for understanding sentence structure and grammatical function. Specific examples are shown in Table 1.

[0029] Table 1

[0030] In the table, the specific meaning of the part-of-speech used can refer to the content in the existing technology webpage "https: / / blog.csdn.net / u014258362 / article / details / 81044286".

[0031] In some embodiments, after the part-of-speech tagging of each query token, an initial part-of-speech weight value is assigned to each token according to the general importance of each part-of-speech in Chinese grammar. For example: Content words, such as nouns (e.g. common nouns n, proper noun series g / j / ns / nt, etc.), verbs (v), nominal verbs (vn), adjectives (a), etc. carry core semantics and are assigned a higher initial weight; function words, such as adverbs (ad), conjunctions (c), prepositions, etc. mainly play a grammatical connecting role and are assigned a lower or zero weight.

[0032] In the embodiments of the present disclosure, an initial weight value is assigned to different parts-of-speech as shown in Table 2.

[0033] Table 2

[0034] Preferably, to capture the semantic focus in a specific grammatical structure, the central word of the part-of-speech combination feature is determined based on linguistic features, specifically including: When the part-of-speech combination feature is a continuous noun combination, the postposed noun is determined as the central word based on the grammatical structure of the centering phrase; When the part-of-speech combination feature is a continuous time noun combination, both time nouns are determined as the central word based on the recursive nesting property of the time adverbial; When the part-of-speech combination feature is the adjacency relationship between a special noun and a common noun, and the special noun plays a semantic limiting role in the noun phrase (when the special noun is in front and the part-of-speech weight value of the special noun is lower than a preset weight threshold), the special noun is determined as the central word according to the centripetal principle of semantic focus; in this embodiment, the preset weight threshold is 1.4; When the part-of-speech combination feature is the modification relationship between an adverb / adjective and a verb, the verb is determined as the central word based on the adverbial dependency relationship.

[0035] Determining the part-of-speech weight value corresponding to each query token further includes: When the query token is the first in a continuous noun combination but not the first in the search condition, the weight value of the determinative part of the noun phrase is increased based on the grammatical status of the noun phrase as a complex component in the sentence, and the part-of-speech weight of the query token is +0.1; When the query token is the first token in the search condition and is a noun, and there is no noun category after it, it plays the role of query core subject or theme in the Chinese grammatical structure, and the part-of-speech weight of the query token is +1.0.

[0036] As an example, in the embodiments of the present disclosure, in the initial part-of-speech weight gain rules of the center word, the part-of-speech weight of the noun after two consecutive nouns is +0.8, the part-of-speech weight of the time noun combination is +0.8; the part-of-speech weight of the special noun in the adjacent relationship between the special noun and the common noun is +1.0; the part-of-speech weight of the verb in the modification relationship between the adverb / adjective and the verb is +0.5; to reflect the key information in different semantic structures. At the same time, the weight and the context can also be set according to the position of the segmented words in the query statement; for example, a higher weight can be set on the core word, and a lower weight can be set on other segmented words.

[0037] Further preferably, when setting the weight, the logical relationship in the query statement is considered. For example, in "search engine optimization", there is a modification relationship between "search engine" and "optimization", and the corresponding weight can be set according to this relationship.

[0038] In some embodiments, to make the weight more adaptive, the present embodiment further adopts machine learning to train and optimize the model, and the trained learning model is used as the part-of-speech weight prediction model, and the detailed process is as follows: 1) Feature engineering: multi-dimensional features are extracted from the query segmented words and their context, including part-of-speech categories, the position of the segmented words in the query, context part-of-speech combination features, and pre-set semantic importance levels.

[0039] Part-of-speech category features: the part-of-speech labels of the query segmented words (such as noun "n", verb "v", adjective "a", etc.) as basic classification features; part-of-speech position features: the absolute position (such as the first, second word) or relative position (such as the beginning, middle, end of the sentence) of the query segmented words in the query statement; part-of-speech combination features: the part-of-speech combination relationship of the query segmented words and its adjacent segmented words (for example, "noun + noun", "adverb + verb", etc.); semantic importance features: the pre-set part-of-speech relevance level of the query segmented words to the search target.

[0040] Preferably, to facilitate model processing, the extracted features are standardized; for example, the numerical value type features (such as position index) are normalized to scale them to the [0, 1] interval. The category type features (such as part-of-speech labels) are One-Hot encoded to convert them into binary vectors that can be recognized by the model.

[0041] 2) Training labeled data acquisition: generate labels based on explicit manual annotation or implicit feedback mined from user historical click behavior; manual annotation includes: initial high-quality weight annotation of corpus by domain experts to form a seed data set; to expand the data set, design a graphical verification code task and push it to ordinary users. For example, display the query word segmentation in the form of a picture and ask the user to drag 1-2 words that they think are the most important to the designated area; the system evaluates and weights the user's annotation results through mechanisms such as multi-user cross-validation, known answer adversarial testing, etc. to ensure data quality.

[0042] Preferably, during the process of training data acquisition, the credibility of the user's annotation results is evaluated, and the credibility evaluation adopts at least one of the mechanisms of multi-user cross-validation, known answer adversarial testing, and user historical accuracy rate evaluation; based on the credibility evaluation results, the user's annotation results whose credibility meets the preset credibility threshold are selected to constitute the part-of-speech training corpus.

[0043] In some embodiments, the domain experts perform high-quality initial weight annotation on the corpus, and the method is as follows: Using annotation assistance tools, experts initialize the weight annotation of existing corpus, develop machine learning, enable automatic pre-annotation of part-of-speech weight initialization through model, then review the pre-annotated weight through different experts, re-annotate the unqualified ones, and then integrate the annotation results of different experts through algorithm to form an initial high-quality labeled data set.

[0044] In some embodiments, a graphical verification code task is designed and pushed to ordinary users, and the processing is performed in a crowdsourcing annotation manner, and the method is as follows: Provide multiple word segmentation option pictures, let the user select 1-2 most important words and drag them to the corresponding graphics; let the user sort the words by importance (such as from left to right or from top to bottom); for example, the short sentence in the data set: "philosophical thinking in great power strategy", the background word segmentation result is "great country ns strategy n in f of uj philosophy n thinking n", in the picture pushed to the user, "in" and "of" are fixed in position, and "great country", "strategy", "philosophy", and "thinking" are cut into pictures and can be moved. The user can drag the two most important ones to the candidate box to complete this annotation.

[0045] Preferably, the verification code picture pushed each time needs to be randomized in terms of word segmentation layout and font style to avoid pattern cracking. In the crowd annotation mode, the reliability of users is evaluated by cross-verification of the same sentence segmentation weight annotation results and known answer pictures (such as expert annotated data) in an adversarial manner. If the model quality tends to be stable, the content of the picture can be predicted by a deep learning model (CNN), and then compared with the annotation results of the user. The annotation data submitted by the user with low confidence rate is submitted for manual review to continue to ensure the quality of the annotation data.

[0046] In the crowd annotation process, the reliability of users is adjusted using the Bayesian inference principle. The system can dynamically adjust the weight of the user based on the historical accuracy rate of the user. For example, the answers of high-accuracy users are given higher reliability.

[0047] 3) Model training and iteration: The annotation data of the part-of-speech training corpus constructed above (features can include part-of-speech, position, combination features, etc.) are used to train a part-of-speech weight prediction model (such as XGBoost, neural network, etc.).

[0048] The part-of-speech weight prediction model trains a mapping function that can receive a feature vector of a word segmentation as input and output the optimal weight value (for example, a floating point number in the range of 0.5 to 1.5) in the current query scenario. The weight value is the part-of-speech weight value. Because the combination of word sequences also affects the weight of the part-of-speech, the word sequence information is also input into the part-of-speech weight model for training.

[0049] The trained weight model is encapsulated into an independent API service. In the query analysis module of the search engine, when the word segmentation and part-of-speech annotation are completed, the API is called in real time to obtain the dynamic weight value for each word segmentation.

[0050] Preferably, in the training data process, the data is divided into training set, validation set and test set in the ratio of 7:2:1. Mean squared error (MSE) or mean absolute error (MAE) is used as the loss function to measure the difference between the predicted weight of the model and the true label. Adam or SGD optimizer is selected. The internal parameters of the model are continuously adjusted through the back propagation algorithm to minimize the loss function.

[0051] Preferably, to ensure the performance of the model, an evaluation and iteration mechanism needs to be established: 1) Evaluation index: Offline evaluation: Evaluate the prediction accuracy of the model (such as MAE / MSE) on the test set, and analyze the feature importance ranking of the model to verify whether it meets the business logic (for example, whether the weight of the core noun is generally higher than that of the auxiliary word). Online evaluation: Deploy the model online for A / B testing, monitor core business indicators such as NDCG (Normalized Discounted Cumulative Gain) of search results, and evaluate the improvement of the model on actual search relevance.

[0052] 2) Iterative optimization: Feature adjustment: If systematic bias is found in the model's weight prediction for a certain part of speech, the weight of the relevant feature can be adjusted or increased to correct it; Model tuning: Use grid search and other methods to optimize the model's hyperparameters (such as learning rate, tree depth, etc.), or introduce regularization terms to prevent overfitting; Scenario adaptation: Regularly use newly added labeled data to incrementally train or retrain the model to adapt to emerging network language, specific scenario abbreviations, and other language changes.

[0053] Step S300: Based on the preset field attribution or context field information of the query condition, determine the field weight value associated with the query word segmentation.

[0054] Preferably, the method for determining the field weight value associated with the query word segmentation based on the preset field attribution or context field information of the query condition is: Input the query condition into a pre-trained field classification model to automatically identify the main field and secondary field of the industry to which it belongs, and determine the field weight value based on the identification result; The field classification model is trained based on a field training corpus with labeled field information, and is used to automatically identify the main and secondary fields of a new query and output the field weight value.

[0055] In some embodiments, to make the weight more adaptive, the embodiment further uses a pre-trained field classification model to identify one or more target fields (such as the main field and the secondary field) to which the query condition belongs. For example, for the query "agricultural big data", the model should be able to identify that the main field is "agriculture" and the secondary field is "computer".

[0056] The pre-trained field classification model is constructed as follows: 1) Field training corpus construction: Establish a phrase corpus containing a large amount of labeled main field and secondary field information of the industry. The labeling work can also use a combination of expert labeling and crowdsourcing labeling, and the construction process of the above part-of-speech training corpus can be referred to.

[0057] 2) Field classification model training: Based on the corpus, train a deep learning classification model (such as TextCNN, BERT). The model can analyze new input queries and automatically identify one or more main and secondary fields to which they belong.

[0058] Further preferably, after determining the domain weight value associated with the query term, the method further comprises a personalized weight enhancement based on user historical behavior, which is: obtaining historical query records of the user in the current session with continuity; wherein at least two query records in the historical query records have common domain information, the historical domain preference includes historical primary domain information and historical secondary domain information, and the continuity refers to the intersection of multiple queries in the domain; performing domain analysis on the historical query records to extract the historical domain preference of the user; When the current query identifies multiple candidate domains, then according to the historical domain preference, the weight of the candidate domain matching the historical preference domain is enhanced to correct the domain weight value.

[0059] For example, the user has successively queried "outdoor tent", "camping sleeping bag", and the system identifies that the current intention of the user is highly concentrated in the "outdoor supplies" domain. Therefore, when processing the current query "outdoor folding chair art design", the system will enhance the weight of the target domain "outdoor supplies", so that the "folding chair" document related to outdoor obtains a higher ranking priority.

[0060] For another example, the user has successively queried "graduation art exhibition" and "color control", and the system identifies that the current intention of the user is highly concentrated in the "art" domain. Therefore, when processing the current query "outdoor folding chair art design", the system will enhance the weight of the target domain "art", so that the "art design" document related to art obtains a higher ranking priority.

[0061] Step S400: for any document in the candidate document set, apply a relevance score algorithm, combine the part-of-speech weight value and the domain weight value as weighting factors, and calculate the relevance score of the document to the query condition.

[0062] Preferably, for each document in the candidate document set, the final relevance score is obtained by aggregating the "weighted contribution values" of all query terms. The specific calculation process is as follows: For each query term in the query term sequence, based on the scoring calculation logic for a single word in the standard BM25 algorithm, calculate its term contribution value to the current document; the scoring calculation logic includes the term frequency (TF) of the word in the document, the inverse document frequency (IDF) in the entire document set, and the document length and other statistical characteristics; Using the part-of-speech weight (β) corresponding to the query term and the domain weight (α) of the target domain to which the query condition belongs, the "term contribution value" is modulated (Modulate) to generate the "weighted contribution value" of the query term; The "weighted contribution value" of each query word is aggregated (e.g., summed) to obtain a final relevance score of the document relative to the entire query condition.

[0063] In the present embodiment, the relevance score algorithm is a standard BM25 score function. Based on the standard BM25 score function, the part-of-speech weight value and the field weight value are combined as weight factors to calculate the relevance score, and the method is as follows: ; wherein IDF(q i ) is the inverse document frequency of the word q i , which measures its distinguishing ability; a domain (q i ) is the field weight value, and the importance of the segmented word in the query is different in different fields; b pos (q i ) is the part-of-speech weight value, and the importance of the segmented word in the query is different in different parts of speech; f(q i , D) is the word frequency of the word q i in the document D; k1 and b are the classic parameters of BM25, which control the word frequency saturation and the document length normalization, is the length of the document D; and avgdl is the average document length of the document set.

[0064] The candidate document set is obtained from a document index library pre-constructed by a retrieval system or screened from full-text documents through preliminary matching by keywords. The document index library is constructed based on indexing and contains segmented words and index information of Chinese text.

[0065] Step S500: The candidate document set is sorted based on the relevance score, and the sorting result is output as a retrieval result and returned to the user.

[0066] Preferably, in order to more clearly illustrate the beneficial effects of the present application, the query condition "outdoor folding chair art design" is taken as input, and the sorting differences between the traditional BM25 and the method of the present application are compared. The candidate document set contains 8 documents of different themes, wherein document D8 is the most relevant "folding chair design", D4 is the field mismatch "folding design of car seats and in-vehicle space", and D3 is the relevant "outdoor tent and folding chair matching skills". The specific details are shown in Table 3.

[0067] Table 3

[0068] It can be seen that the df value (the frequency of a word in all documents, global data) is as follows: outdoor: 127, folding chair: 23, art: 75, and design: 104.

[0069] Traditional BM25: only rely on statistical features. Total number of documents N = 12000, average document length avgL = 6, word frequency saturation coefficient k1 = 1.2, document length adjustment coefficient b = 0.75; The scores of each document are as follows: D6 (score 14.93, ranked first): contains three query-related words "outdoor", "art", and "design", and the document length is shorter than the average length, and the document length adjustment coefficient b is added, and the statistical characteristics of multiple words are superimposed, so that the total score is the highest. But it didn't identify the domain conflict between "outdoor" and "art / design", and ranked the cross-domain document first; D3 (score 11.21, ranked second): contains "outdoor" and "folding chair", and "folding chair" has a very low document frequency (df=23) and high rarity, so the IDF value is large, which contributes significantly to the score, so it is ranked high; Similarly, D8 score = 11.55, D1 score = 10.76, D2 score = 10.21, D7 score = 4.84, D5 score = 4.76, D4 score is very low.

[0070] In the calculation process, only the contribution of "term frequency (TF), document frequency (DF), and document length" to the score is calculated, and semantics (part of speech) and domain attributes are not considered. D6 contains multiple query words "outdoor", "art", and "design", and the document length (5) is shorter than the average length (6), so the document length adjustment coefficient b is added, and the statistical characteristics of multiple words are superimposed, so that the final score is the highest. The ranking result is: D6>D3>D8>D1>D2>D7>D5>D4.

[0071] The method of the application comprises the following steps: Determine the part-of-speech weight β pos : For the query "outdoor folding chair art design", the part-of-speech tagging identifies "outdoor" as a directional noun, assigns a weight β = 0.7; "folding chair" is the core noun, assigns a high weight β = 1.6; "art" is an ordinary noun, assigns a weight β = 0.6; "design" is an ordinary noun, assigns a weight β = 1.6; "in" and other auxiliary words and conjunctions assign a weight β = 0; Determine the domain weight α domain: The system dynamically learns the field preference by analyzing the user's historical query records. Scene 1: When it is detected that the user's coherent historical records before the current query are "tent recommendation" and "which brand of camping sleeping bag is good", the system identifies that the user's current core preference is the "outdoor products" field, and therefore assigns a high weight α = 1.8 to the "outdoor products" field to which "outdoor" and "folding chair" belong, and sets the weight of non-target fields such as "art" and "design" to α = 1.0. Scene 2: When it is detected that the user's historical records are "graduation art exhibition" and "color control", the system identifies that the user's preference is the "art / design" field, and therefore assigns a high weight α = 1.8 to the "art / design" field to which "art" and "design" belong, and sets the weight of non-target fields such as "outdoor" and "folding chair" to α = 1.0.

[0072] On the basis of traditional statistical features, the "part-of-speech weight β pos " and "field weight α domain " are superimposed on each document to correct the deviation of non-semantic and non-field matching: Scene 1 (Preference "Outdoor Products"): D8, D1, and D3 all contain "outdoor" and "folding chair" tokens, which belong to the target field, and a high field weight α = 1.8 is used when scoring. For example, D8, the BM25 base score of its "folding chair" is multiplied by α = 1.8 and β = 1.6 before accumulation, and the final score is 31.75. D1 combines the weighted scores of "outdoor" and "folding chair", and the final score is 25.90. D2 and D6 contain "design" and other words, but because the "art / design" field weight is reduced to α = 1.0, their total scores are effectively suppressed, and D2 scores 11.05; Similarly, the scores of each document can be obtained, and the final ranking is D8 (31.75) > D1 (25.90) > D3 (24.38) > D6 (16.42) > D2 (11.05) > D7 (7.77) > D5 (6.63) > D4 (lower), and the document that best fits the "outdoor products" preference is precisely ranked at the top.

[0073] Scene 2 (Preference "Art / Design"): D8, D6, and D2 all contain "art" or "design" tokens, which belong to the target field, and a high field weight α = 1.8 is used when scoring. For example, D8, the BM25 base score of its "design" part is multiplied by α = 1.8 and β = 1.6 before accumulation, and the final score is 24.90. D6 gets high field weighting for both "art" and "design", and the final score is 23.24. D1 and D3 contain "outdoor" and other words, but because the "outdoor products" field weight is reduced to α = 1.0, their total scores are effectively suppressed, and D1 scores 14.38; Similarly, each document score can be obtained, and the final ranking is D8 (24.90) > D6 (23.24) > D2 (19.90) > D1 (14.38) > D3 (13.90) > D7 (13.99) > D5 (3.70) > D4 (lower), and the document most consistent with the "art / design" preference is precisely ranked at the top.

[0074] The ideal ranking of scenario 1 is (D8>D1>D3>D4>D6>D5>D2>D7), and the ideal ranking of scenario 2 is (D8>D2>D6>D4>D1>D3>D7>D5). The NDCG (normalized discounted cumulative gain) score of the traditional BM25 algorithm result of scenario 1 is (0.894); the NDCG (normalized discounted cumulative gain) score of the traditional BM25 algorithm result of scenario 2 is (0.852). The calculation result of the method of the application is that the ranking results of the two scenarios are highly consistent with the user's intention inferred based on the historical query; the NDCG scores are improved to 0.982 (scenario 1) and 0.972 (scenario 2), respectively, and both of them place the file D8 at the first sequence, which is consistent with the ideal ranking. Compared with the result of the traditional BM25 scoring function, there is a great improvement, the effect is obvious, the ranking result is closer to the user's intention, and the effectiveness of the application in realizing personalized and precise Chinese semantic enhanced fuzzy retrieval through dynamic weight learning is fully proved.

[0075] This exposes the defect that the BM25 algorithm itself cannot understand semantics and fields, resulting in a certain deviation of the ranking result from the user's intention and general relevance.

[0076] The method of the application innovatively fuses the dynamically learned part-of-speech weight and field weight, constructs an extended BM25 scoring function, effectively solves the defects of "field generalization and semantic insensitivity" in the prior art, makes the ranking result more consistent with the real intention of the user's fuzzy query, significantly improves the ranking accuracy and relevance of the Chinese long text fuzzy query, and has very high application value.

[0077] Although the steps in the above embodiments are described in the above order, those skilled in the art can understand that, in order to achieve the effect of the embodiments, the different steps do not have to be executed in such an order, but can be executed simultaneously (in parallel) or in a reversed order, and these simple changes are within the protection scope of the application.

[0078] The Chinese semantic enhanced fuzzy retrieval system based on adaptive weight of the second embodiment of the application comprises: The semantic analysis module is configured to obtain a query condition input by a user, perform semantic analysis on the query condition, generate a query word sequence, and perform part-of-speech tagging on each query word, wherein the query word sequence is composed of at least one query word. a weight generation module configured to determine a part-of-speech weight value corresponding to each query token based on part-of-speech features and / or part-of-speech combination features of the query token sequence, and determine a domain weight value associated with the query token based on a preset domain attribution or context domain information of the query condition; a relevance score module configured to calculate a relevance score of any document in the candidate document set with the query condition by extending a BM25 score function; the extended BM25 score function takes the standard BM25 score function as a basis and combines the part-of-speech weight value and the domain weight value as core weighting factors; a result processing module configured to sort the candidate document set according to the relevance score and output the sorting result as a retrieval result.

[0079] Preferably, the weight generation module includes a part-of-speech weight identification module for determining the part-of-speech weight value corresponding to each query token, and a domain weight identification module for determining the domain weight value associated with the query token.

[0080] Preferably, it further includes a data collection and cleaning module for collecting training data and labeled data and training the model.

[0081] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process and related description of the above-described system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0082] It should be noted that the above-described Chinese semantic enhanced fuzzy retrieval system based on adaptive weight provided by the embodiments, only the division of the above-mentioned functional modules is exemplified, in actual application, the above-mentioned functions can be completed by different functional modules according to needs, that is, the modules or steps in the embodiments of the present application are further decomposed or combined, for example, the modules of the above-mentioned embodiments can be combined into one module, or can be further split into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present application are only for distinguishing the modules and steps, and should not be considered as an improper limitation of the present application.

[0083] The third embodiment of the electronic device of the present application comprises: at least one processor; and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the processor, and the instructions are used to be executed by the processor to implement the above-mentioned Chinese semantic enhanced fuzzy retrieval method based on adaptive weight.

[0084] A computer readable storage medium of a fourth embodiment of the present application, the computer readable storage medium stores computer instructions, the computer instructions are used to be executed by the computer to implement the above-mentioned adaptive weight-based Chinese semantic enhanced fuzzy retrieval method.

[0085] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes and related descriptions of the above-mentioned electronic device and computer readable storage medium can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0086] Those skilled in the art should be aware that the modules and method steps of each example described in connection with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. The software modules and method steps corresponding to the program can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. In order to clearly illustrate the interchangeability of electronic hardware and software, the components and steps of each example have been generally described in the above description. Whether the functions are performed by electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0087] Computer program code for carrying out operations of the present application can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0088] The computer program product of the present application can be a computer program implemented on one or more computers in one or more locations. The program instructions can be stored on a computer readable medium, such as a hard disk, CD-ROM, optical storage, or any other tangible medium. The program instructions can be downloaded over a network from a remote computer (e.g., a server computer) or to a remote computer (e.g., a client computer) with the assistance of one or more of the various above-mentioned network devices. From the network or remote computer, the program instructions can be loaded into memory and executed by one or more computer processors.

[0089] The terms "first", "second", etc. are used to distinguish similar objects, rather than to describe or indicate a particular order or sequence.

[0090] The term "comprising" or any other similar term is intended to encompass the inclusion of one or more stated elements, steps, or components, but not to the exclusion of any other elements, steps, or components. It is noted that not all of the activities or elements referenced throughout this application are required, that a variety of permutations can be made of the dependent claims depending on the desired sought to be secured, and that one skilled in the art will be able to understand from the

[0091] So far, the technical solutions of the present application have been described in combination with the preferred embodiments shown in the drawings, but it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to the related technical features without departing from the principles of the present application, and the technical solutions after these changes or replacements will all fall within the protection scope of the present application.

Claims

1. A Chinese semantically enhanced fuzzy retrieval method based on adaptive weights, characterized in that, Includes the following steps: Obtain the query conditions input by the user, perform semantic parsing on the query conditions, and generate a query word segmentation sequence, wherein the query word segmentation sequence consists of at least one query word; Based on the part-of-speech features and / or part-of-speech combination features of the query word segmentation sequence, determine the part-of-speech weight value corresponding to each query word segmentation; Based on the preset domain affiliation or contextual domain information of the query conditions, determine the domain weight value associated with the query word segmentation; For any document in the candidate document set, a relevance scoring algorithm is applied, and the part-of-speech weight value and the domain weight value are used as weighting factors to calculate the relevance score between the document and the query conditions. The candidate document set is sorted based on the relevance score, and the sorting result is output as the retrieval result.

2. The Chinese semantically enhanced fuzzy retrieval method based on adaptive weights according to claim 1, characterized in that, The method for determining the part-of-speech weight value corresponding to each query word segment is as follows: Each query word segment is labeled with part-of-speech tags, and the initial part-of-speech weights are obtained; Identify whether one or more preset part-of-speech combination features exist in the query word segmentation sequence; When any query word is identified to form a preset part-of-speech combination feature with its adjacent query words, the central word of the part-of-speech combination feature is determined based on linguistic features, and the initial part-of-speech weight of the central word is increased to obtain the part-of-speech weight value. The part-of-speech combination features include at least one of the following: consecutive noun combinations, consecutive time noun combinations, adjacency relationship between special nouns and common nouns, and modification relationship between adverbs / adjectives and verbs.

3. The Chinese semantically enhanced fuzzy retrieval method based on adaptive weights according to claim 2, characterized in that, The central word of the part-of-speech combination feature is determined based on linguistic features, specifically including: When the part-of-speech combination feature is a continuous noun combination, the postpositional noun is determined as the head word based on the grammatical structure of the attributive phrase; When the part-of-speech combination feature is a combination of consecutive time nouns, based on the recursive nesting characteristics of time adverbs, it is determined that both time nouns are the head words; When the part-of-speech combination feature is the adjacency relationship between a special noun and a common noun, and the special noun plays a semantic limiting role in the noun phrase, the special noun is determined as the central word based on the centripetal principle of semantic focus. When the part-of-speech combination feature is a modifying relationship between an adverb / adjective and a verb, the verb is determined as the head word based on the adverbial-head dependency relationship.

4. The Chinese semantically enhanced fuzzy retrieval method based on adaptive weights according to claim 1, characterized in that, The method for determining the domain weight value associated with the query word segmentation based on the preset domain affiliation or contextual domain information of the query conditions is as follows: The query conditions are input into a pre-trained domain classification model to automatically identify the main and secondary domains of the industry to which it belongs, and the domain weight value is determined based on the identification results. The domain classification model is trained on a domain training corpus with labeled domain information.

5. The Chinese semantically enhanced fuzzy retrieval method based on adaptive weights according to claim 4, characterized in that, The domain training corpus is constructed through crowdsourced annotation, and the method is as follows: Through a graphical interactive interface, the query conditions to be labeled and their word segmentation results in the training data are pushed to the labeling users, who can then select and mark one or more core words that reflect the query intent. The credibility of user annotation results is evaluated, and the credibility evaluation adopts at least one of the following mechanisms: multi-user cross-validation, known answer adversarial testing, and user historical accuracy evaluation. Based on the credibility assessment results, user-annotated results that meet the preset credibility threshold are selected to form a domain training corpus.

6. The Chinese semantically enhanced fuzzy retrieval method based on adaptive weights according to claim 4, characterized in that, After determining the domain weight values ​​associated with the query word segmentation, the method further includes: Obtain the user's historical query records prior to the current query operation, and perform domain analysis on the historical query records to extract the user's historical domain preferences; at least two query records in the historical query records have common domain information, and the historical domain preferences include historical primary domain information and historical secondary domain information; When multiple candidate domains are identified in this query, the weights of candidate domains that match the historical domain preferences are increased based on the historical domain preferences, so as to correct the domain weight values.

7. The Chinese semantically enhanced fuzzy retrieval method based on adaptive weights according to claim 1, characterized in that, When parsing the semantics of the query conditions, if the query segment is identified as a numeral, the numeral and the next query segment are merged into an indivisible semantic unit; if the next query segment is a quantifier, the numeral, the quantifier, and the next query segment are merged into an indivisible semantic unit.

8. The Chinese semantically enhanced fuzzy retrieval method based on adaptive weights according to claim 1, characterized in that, The relevance scoring algorithm is based on the standard BM25 scoring function. It combines the part-of-speech weight value and the domain weight value as weighting factors to calculate the relevance score. The method is as follows: ； Among them, IDF(q) i ) is the term q i Inverse document frequency is used to measure its discriminative power; α domain (q i ) represents the domain weight value; the importance of word segmentation in queries varies across different domains; β pos (q i f(q) represents the part-of-speech weight value. Different parts of speech have different importance in the query. i D) is the term q i The term frequencies in document D; k1 and b are classic BM25 parameters, controlling term frequency saturation and document length normalization. is the length of document D; avgdl is the average document length of the document collection.

9. The Chinese semantically enhanced fuzzy retrieval method based on adaptive weights according to claim 1, characterized in that, The candidate document set is obtained from a document index library pre-built by the retrieval system or from the full set of documents through preliminary keyword matching. The document index library is built based on an index and contains word segmentation and index information of Chinese text.

10. A Chinese semantically enhanced fuzzy retrieval system based on adaptive weights, characterized in that: The system includes: The semantic parsing module is configured to obtain the query conditions input by the user, perform semantic parsing on the query conditions, generate a query word segmentation sequence, and perform part-of-speech tagging on each query word segmentation. The query word segmentation sequence consists of at least one query word segmentation. The weight generation module is configured to determine the part-of-speech weight value corresponding to each query segment based on the part-of-speech features and / or part-of-speech combination features of the query segment sequence; and to determine the domain weight value associated with the query segment based on the preset domain affiliation or contextual domain information of the query conditions. The relevance scoring module is configured to calculate the relevance score between any document in the candidate document set and the query conditions by using an extended BM25 scoring function; the extended BM25 scoring function is based on the standard BM25 scoring function and combines the part-of-speech weight value and the domain weight value as core weighting factors. The results processing module is configured to sort the candidate document set according to the relevance score and output the sorting result as the retrieval result.

Images