A method and device for determining word weights in a query

Abstract

The application provides a method and device for determining word weight in a query sentence, the method comprising: determining a target vocabulary in the query sentence, wherein the target vocabulary comprises a target segmented word or an entity vocabulary, and the target segmented word and the entity vocabulary are not necessarily the same; determining a part-of-speech score and an IDF score of the target segmented word according to a preset corresponding relationship between segmented words and scores; determining an entity score of the entity vocabulary according to an entity type of the entity vocabulary; and determining a vocabulary weight of the target vocabulary by weighted summation according to at least one of the part-of-speech score, the IDF score and the entity score. The application improves the applicability of the scenario.

Description

Technical Field

[0001] This application relates to the field of information retrieval technology, and in particular to a method and apparatus for determining the weight of words in a query phrase. Background Technology

[0002] In the field of information retrieval, query intent understanding is a crucial algorithmic task, encompassing query preprocessing, query segmentation, query word weight analysis, query error correction, query expansion, and query normalization. Among these, query word weight analysis, building upon word segmentation, analyzes the relative importance of different words, and its results significantly influence subsequent ranking, making it a vital step in query intent understanding.

[0003] In existing technologies, there are several approaches to query word weight analysis. One representative approach transforms word weight calculation into a word segmentation classification problem. For example, by dividing query segments into four categories—Super important, Required, Important, and Unimportant—word weight analysis is treated as a machine learning classification task. Information such as part-of-speech tagging, position, and left / right entropy is incorporated, and algorithms like Logistic Regression (LR) and GBDT are used for classification to obtain word weights for different words. Then, corresponding scores are assigned to different categories. Another representative approach uses search click logs to statistically analyze the vocabulary information in the logs to reflect the weight of query segmentation. For example, by statistically analyzing the word segments in the search query set corresponding to the same resource, word weights are calculated based on the frequency of word occurrences. However, this method requires a large amount of click data, and the result is a weight dictionary, which cannot effectively support different weights for the same word in different scenarios or for words not yet logged in.

[0004] Whether using labeled corpora for classification in the first approach or relying on a large number of click logs in the second approach, both are overly dependent on the use case, making them unlikely to be ported across scenarios or applied across domains, and their applicability to a single scenario is limited. Summary of the Invention

[0005] The purpose of this application is to provide a method and apparatus for determining the weight of words in a query sentence, so as to solve the problem of limited applicability in a single scenario. The specific technical solution is as follows:

[0006] Firstly, a method for determining the weight of words in a query sentence is provided, the method comprising:

[0007] Determine the target words in the query phrase, wherein the target words include target word segments or entity words, and the target word segments and entity words are not necessarily the same;

[0008] Based on the preset correspondence between word segmentation and score, the part-of-speech score and IDF score of the target word are determined;

[0009] The entity score of the entity word is determined based on its entity type.

[0010] The lexical weight of the target word is determined by weighted summation based on at least one of the part-of-speech score, the IDF score, and the entity score.

[0011] Optionally, determining the lexical weight of the target word by weighted summation based on at least one of the part-of-speech score, the IDF score, and the entity score includes:

[0012] If the entity word exists in a preset word segmentation list, it is determined that the target word segment and the entity word are the same, wherein the word segmentation list contains each word segment;

[0013] Obtain the preset part-of-speech weights, IDF weights, and entity weights;

[0014] The part-of-speech score, part-of-speech weight, IDF score, IDF weight, entity score, and entity weight are weighted and summed, and the weighted sum is used as the lexical weight of the target word.

[0015] Optionally, determining the lexical weight of the target word by weighted summation based on at least one of the part-of-speech score, the IDF score, and the entity score includes:

[0016] When the entity word does not exist in the preset word segmentation list and the target word is the target word segmentation, the preset part-of-speech weight and IDF weight are obtained;

[0017] The part-of-speech score, the part-of-speech weight, the IDF score, and the IDF weight are weighted and summed, and the weighted sum is used as the lexical weight of the target word.

[0018] Optionally, determining the lexical weight of the target word by weighted summation based on at least one of the part-of-speech score, the IDF score, and the entity score includes:

[0019] If the entity word does not exist in the preset word segmentation list and the target word is the entity word, obtain the preset entity weight;

[0020] The entity scores and entity weights are weighted and summed, and the sum is used as the word weight of the target word.

[0021] Optionally, determining the target words in the query phrase includes:

[0022] The query phrase is segmented using a word segmentation tool, wherein each segmented word exists in a preset word segmentation list;

[0023] Entity recognition schemes are used to extract entities from the query terms;

[0024] If the entity word exists in the preset word segmentation list, then the target word segmentation and the entity word are determined to be the same, and the target word segmentation includes either the target word segmentation or the entity word.

[0025] If the entity word does not exist in the preset word segmentation list, then it is determined that the target word segmentation and the entity word are different, and the target word includes the target word segmentation or the entity word.

[0026] Optionally, determining the part-of-speech score of the target word based on a preset correspondence between word segmentation and score includes:

[0027] The word segments in the query sentence are tagged with parts of speech using a preset part-of-speech tagging method.

[0028] Based on the preset correspondence between word segmentation part of speech and part of speech score, the part of speech score corresponding to the part of speech of the target word is determined.

[0029] Optionally, the IDF score of the target word is determined based on a preset correspondence between word segmentation and score, including:

[0030] If the target word is determined to be in a preset IDF dictionary, the IDF score of the target word is determined according to the correspondence between word segments and IDF scores in the IDF dictionary;

[0031] If it is determined that the target word is not in the preset IDF dictionary, the preset IDF mean value is used as the IDF score of the target word.

[0032] Optionally, after determining the word weights of the target words through weighted summation, the method further includes:

[0033] By using a preset normalization operation method, the word weights of each word in the query phrase are normalized to a preset range.

[0034] Secondly, a device for determining the weight of words in a query phrase is provided, the device comprising:

[0035] The first determining module is used to determine the target words in the query phrase, wherein the target words include target word segments or entity words, and the target word segments and entity words are not necessarily the same;

[0036] The second determining module is used to determine the part-of-speech score and IDF score of the target word based on the preset correspondence between word segmentation and score;

[0037] The third determining module is used to determine the entity score of the entity word based on the entity type of the entity word;

[0038] The fourth determining module is used to determine the lexical weight of the target word by weighted summation based on at least one of the part-of-speech score, the IDF score, and the entity score.

[0039] Thirdly, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0040] Memory, used to store computer programs;

[0041] The processor, when executing a program stored in memory, implements the steps of the method for determining the word weights in any of the aforementioned query phrases.

[0042] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of the method for determining the word weight in any of the aforementioned query phrases.

[0043] Beneficial effects of the embodiments in this application:

[0044] This application is applied to query optimization in the field of information retrieval technology. In this application, the server determines the part-of-speech score and IDF score of the target word by the correspondence between word segmentation and score, determines the entity score of the entity word by entity type, and then determines the word weight of the target word based on at least one of the part-of-speech score, IDF score and entity score.

[0045] In the process of determining word weights, the correspondence between word segmentation and score is pre-set and has nothing to do with the scenario. The entity score of the same entity type is the same in different scenarios. Therefore, the word weights of this application are not restricted by the business scenario and can be ported and used across scenarios and domains, thereby improving the applicability of the scenario.

[0046] Of course, implementing any product or method of this application does not necessarily require achieving all of the above advantages at the same time. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 A schematic diagram of the hardware environment for a method of determining word weights in a query sentence provided in an embodiment of this application;

[0049] Figure 2 A flowchart illustrating a method for determining word weights in a query sentence, provided in an embodiment of this application;

[0050] Figure 3 A flowchart illustrating the processing of word weights in a query sentence, provided in an embodiment of this application.

[0051] Figure 4 A schematic diagram of the structure of a device for determining the word weight in a query sentence, provided in an embodiment of this application;

[0052] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustration and has no specific meaning in itself. Therefore, "module" and "part" may be used interchangeably.

[0055] To address the problems mentioned in the background art, according to one aspect of the embodiments of this application, an embodiment of a method for determining the weight of words in a query sentence is provided.

[0056] Optionally, in the embodiments of this application, the method for determining the word weights in the query phrase described above can be applied to, for example... Figure 1 The hardware environment shown consists of terminal 101 and server 103. Figure 1As shown, server 103 is connected to terminal 101 via a network and can be used to provide services to the terminal or clients installed on the terminal. Database 105 can be set up on the server or independently of the server to provide data storage services for server 103. The network mentioned above includes, but is not limited to, wide area network, metropolitan area network or local area network. Terminal 101 includes, but is not limited to, PC, mobile phone, tablet computer, etc.

[0057] The method for determining the word weight in a query phrase according to one embodiment of this application can be executed by server 103 or by terminal 101, and is used to determine the word weight in the query phrase.

[0058] The following will describe in detail, using a specific implementation method and taking a server as an example, a method for determining word weights in a query phrase provided in this application embodiment. Figure 2 As shown, the specific steps are as follows:

[0059] Step 201: Identify the target words in the query phrase.

[0060] The target vocabulary includes target segmentation or entity vocabulary, and the target segmentation and entity vocabulary are not necessarily the same.

[0061] In this embodiment of the application, the server obtains the query from the terminal record, and then performs word segmentation and entity recognition on the query to obtain each word segmentation and entity word in the query.

[0062] The server takes any word as the target word. If the word is both a segmented word and an entity word, the entity word may be a single segmented word, a collection of multiple segmented words, or a part of a segmented word.

[0063] If the target word segment and the entity word are the same, the target word is either the target word segment or the entity word; if the word is a word segment and not an entity word, then the target word segment and the entity word are different, and the target word is the target word segment; if the word is an entity word and not a word segment, then the target word segment and the entity word are different, and the target word is the entity word.

[0064] For example, the query: Where is Peking University Library? The entity words are: [Peking University Library], and the word segments are: [Beijing, University, Library, At, Where]. Therefore, the entity words and word segments are not the same.

[0065] For example, the query: Where is the library? The entity word is: [library], and the word segmentation is: [library, in, where]. Therefore, [library] is both an entity word and a word segmentation.

[0066] Step 202: Determine the part-of-speech score and IDF score of the target word based on the preset correspondence between word segmentation and score.

[0067] In this embodiment of the application, the database pre-stores the correspondence between word segmentation parts of speech and part-of-speech scores, and the server can determine the part-of-speech score of the target word segment based on the correspondence; the database pre-stores the correspondence between word segmentation and IDF scores in the IDF dictionary, and the server can determine the IDF score of the target word segment based on the correspondence.

[0068] Step 203: Determine the entity score of the entity word based on its entity type.

[0069] In this embodiment, the server can detect the entity type of entity words. Entity types include, but are not limited to, names of people, place names, email addresses, and phone numbers. Different scenarios can have different entity types, and the entity score for the same entity type is the same in different scenarios. The database also pre-stores the correspondence between entity types and entity scores, and the server can determine the entity score based on the detected entity type.

[0070] Step 204: Determine the lexical weight of the target word by weighted summation based on at least one of the part-of-speech score, IDF score, and entity score.

[0071] In this embodiment, if the target word is only the target segmented word, the server determines the word weight of the target word by weighted summation based on the part-of-speech score and the IDF score. If the target word is only the entity word, the server determines the word weight of the target word by weighted summation based on the entity score. If the target word is either the target segmented word or the entity word, the server determines the word weight of the target word by weighted summation based on the part-of-speech score, the IDF score, and the entity score.

[0072] In this application, the server determines the part-of-speech (POS) score and IDF score of the target word by establishing the correspondence between word segmentation and scores, and determines the entity score of the entity word by entity type. Then, based on at least one of the POS score, IDF score, and entity score, the lexical weight of the target word is determined. In determining the lexical weight, the correspondence between word segmentation and scores is pre-set and independent of the scenario. The entity score of the same entity type is the same in different scenarios. Therefore, the lexical weight of this application is not constrained by the business scenario, enabling cross-scenario and cross-domain portability and improving scenario applicability.

[0073] As an optional implementation, the lexical weight of the target vocabulary is determined by weighted summation based on at least one of part-of-speech score, IDF score, and entity score. This includes three implementation methods:

[0074] In one embodiment, if the entity words exist in a preset word segmentation list, it is determined that the target word and the entity words are the same, wherein the word segmentation list contains each word; preset part-of-speech weights, IDF weights, and entity weights are obtained; a weighted sum is performed based on the part-of-speech score, part-of-speech weight, IDF score, IDF weight, entity score, and entity weight, and the weighted sum is used as the word weight of the target word.

[0075] In this embodiment, since the preset word segmentation list contains each word, if the server determines that the entity word exists in the word segmentation list, it means that the target word is both the target word and the entity word. The target word and the entity word are the same. Then the server needs to calculate the word weight together for the target word as the target word and the target word as the entity word.

[0076] The server retrieves preset part-of-speech weights, preset IDF weights, and preset entity weights from the database, and then calculates the word weight of the target word using a weighted summation method.

[0077] The formula for calculating the lexical weight of target words is as follows:

[0078] Weight w =x*Weight pos +y*Weight ner +z*Weight idf

[0079] Among them, Weight w Weight represents the lexical weight of the target word. pos Weight represents the part-of-speech score. ner Weight represents the entity score. idf denoted by IDF score, x represents part-of-speech weight, y represents entity weight, and z represents IDF weight.

[0080] In another embodiment, determining the lexical weight of the target word by weighted summation based on at least one of the part-of-speech score, IDF score, and entity score includes: obtaining the preset part-of-speech weight and IDF weight when the entity word does not exist in the preset word segmentation list and the target word is the target word; performing a weighted summation based on the part-of-speech score, part-of-speech weight, IDF score, and IDF weight, and using the weighted summation result as the lexical weight of the target word.

[0081] In this embodiment, since the preset word segmentation list contains each word, if the server determines that an entity word does not exist in the word segmentation list, it indicates that the target word is either the target word or an entity word. If the server determines that the target word is the target word, then the server obtains the preset part-of-speech weights and preset IDF weights from the database, and then calculates the word weight of the target word according to the weighted summation method.

[0082] The formula for calculating the lexical weight of target words is as follows:

[0083] Weight w =x*Weight pos +z*Weight idf

[0084] Among them, Weight w Weight represents the lexical weight of the target word. pos Weight represents the part-of-speech score. idf denoted by IDF score, x represents part-of-speech weight, and z represents IDF weight.

[0085] In another embodiment, determining the word weight of the target word by weighted summation based on at least one of part-of-speech score, IDF score, and entity score includes: obtaining a preset entity weight when the entity word does not exist in the preset word segmentation list and the target word is an entity word; performing a weighted summation based on the entity score and entity weight, and using the weighted summation result as the word weight of the target word.

[0086] In this embodiment, since the preset word segmentation list contains each word, if the server determines that an entity word does not exist in the word segmentation list, it indicates that the target word is either the target word or an entity word. If the server determines that the target word is an entity word, then the server retrieves the preset entity weights from the database and calculates the word weight of the target word using a weighted summation method.

[0087] The formula for calculating the lexical weight of target words is as follows:

[0088] Weight w =y*Weight ner

[0089] Among them, Weight w Weight represents the lexical weight of the target word. ner y represents the entity score, and y represents the entity weight.

[0090] As an optional implementation, determining the target vocabulary in the query phrase includes: segmenting the query phrase using a word segmentation tool, wherein each word segment exists in a preset word segmentation list; extracting entities from the query phrase using an entity recognition scheme; if the entity vocabulary exists in the preset word segmentation list, then the target word segment and the entity vocabulary are determined to be the same, and the target vocabulary includes either the target word segment or the entity vocabulary; if the entity vocabulary does not exist in the preset word segmentation list, then the target word segment and the entity vocabulary are determined to be different, and the target vocabulary includes either the target word segment or the entity vocabulary.

[0091] In this embodiment, the server uses a word segmentation tool to segment the query terms and extracts entities from the query terms using an entity recognition scheme. The server then determines whether the entity words exist in a preset word segmentation list.

[0092] If it exists, it means that entity words also belong to word segmentation. Then the target word belongs to both entity words and target word segmentation. Target word segmentation and entity words are the same. Target word can be either target word segmentation or entity words.

[0093] If it does not exist, it means that the entity word does not belong to the word segmentation, the target word segmentation and the entity word are different, and the target word is either the target word segmentation or the entity word.

[0094] The word segmentation tool can be jieba, IK, Hanlp, etc., and this application does not impose specific limitations on the word segmentation tool. The entity recognition scheme can be an HMM (Hidden Markov Model), a CRF (Conditional Random Field) model, BILSTM+CRF, or a method based on a pre-trained model, and this application does not impose specific limitations on the entity recognition scheme.

[0095] As an optional implementation method, determining the part-of-speech score of the target word based on the preset correspondence between word segmentation and score includes: performing part-of-speech tagging on the word segmentation in the query sentence using a preset part-of-speech tagging method; and determining the part-of-speech score corresponding to the part-of-speech of the target word based on the preset correspondence between word segmentation and part-of-speech score.

[0096] In this embodiment, the server uses a preset part-of-speech tagging method to tag the parts of speech of the query words; the parts of speech include nouns, verbs, adjectives, adverbs, etc. The database stores the correspondence between the parts of speech of the segmented words and their scores, and the server determines the part-of-speech score corresponding to the target segmented word based on this correspondence. The preset part-of-speech tagging method can be various methods such as HMM models and CRF models, or it can be performed using a part-of-speech dictionary.

[0097] For example, nouns are scored 1.0, verbs are scored 0.5, adjectives and adverbs are scored 0.4, and other parts of speech are scored 0.1, thus obtaining scores based on parts of speech.

[0098] As an optional implementation, determining the IDF score of the target word based on the preset correspondence between word segmentation and score includes: if the target word is located in the preset IDF dictionary, determining the IDF score of the target word based on the correspondence between word segmentation and IDF score in the IDF dictionary; if the target word is not located in the preset IDF dictionary, using the preset IDF average as the IDF score of the target word.

[0099] In this embodiment, the IDF dictionary stores the correspondence between word segmentation and IDF scores, as well as the IDF mean. The IDF mean is calculated using the following formula:

[0100] in, Let w be the IDF mean, w be the w-th target word segment, and N be the total number of documents in the document library. w Let w be the IDF score of the w-th target word.

[0101] If the server determines that the target word is in the preset IDF dictionary, it determines the IDF score of the target word based on the correspondence; if the server determines that the target word is not in the preset IDF dictionary, it uses the preset IDF mean as the IDF score of the target word.

[0102] Optionally, this application determines the IDF dictionary based on formulas. Calculate the inverse document frequency (IDF) of each word. First, obtain the total number of documents N. Second, count how many documents each word appears in, i.e., M in the formula. Then, the IDF of each word can be calculated. w Then, the IDF value is normalized to the range [0, 1] using min-max normalization, which is the IDF value of the word segment w. IDF max and IDF min It consists of the maximum and minimum values ​​of IDF obtained from statistics, and finally an IDF dictionary is constructed based on each word w and the IDF value of word w.

[0103] The server periodically retrieves the query, then calculates the IDF value of the token w, thereby updating the IDF dictionary.

[0104] As an optional implementation, after determining the word weights of the target words by weighted summation, the method further includes: normalizing the word weights of each word in the query sentence to a preset range by using a preset normalization operation.

[0105] The server obtains the word weights for each word segment, and then uses SumNormalizer normalization to normalize the weights of each word to the range [0, 1]. The sum of the word weights is 1. The SumNormalizer normalization formula is as follows: Weight w ^ represents the normalized word weight, and m represents the number of words in the query.

[0106] Optionally, before determining the word weight of the target word by weighted summation, the server also filters out stop words by setting a stop word list and sets the weight of the stop words in the query to 0, thus removing the word score of the stop words.

[0107] Optionally, embodiments of this application also provide a flowchart of a method for determining word weights in a query sentence, such as... Figure 3 As shown.

[0108] The background scheduled task is to build an IDF dictionary. The steps include reading documents (queries) from the index, performing word segmentation, calculating the IDF value of the documents and normalizing it as an IDF dictionary. When calculating the weight of query words later, the IDF value can be obtained by directly looking up the word segmentation table.

[0109] Obtain the query text and segment it using algorithms such as CRF. Calculate the weights of the segmentation results in three aspects:

[0110] 1. Use algorithms such as HMM to label parts of speech and calculate part-of-speech weight scores;

[0111] 2. Use algorithms such as CRF and deep learning to extract entities from the query;

[0112] 3. Obtain the IDF value by looking up the table in the IDF dictionary, and calculate the IDF weight score;

[0113] Finally, the three weighted scores are summed and normalized to obtain the final weighted score.

[0114] Based on the same technical concept, embodiments of this application also provide a device for determining the word weight in a query sentence, such as... Figure 4 As shown, the device includes:

[0115] The first determining module 401 is used to determine the target words in the query phrase, wherein the target words include target word segments or entity words, and the target word segments and entity words are not necessarily the same;

[0116] The second determining module 402 is used to determine the part-of-speech score and IDF score of the target word based on the preset correspondence between word segmentation and score;

[0117] The third determining module 403 is used to determine the entity score of an entity word based on its entity type.

[0118] The fourth determining module 404 is used to determine the lexical weight of the target word by weighted summation based on at least one of the part-of-speech score, IDF score and entity score.

[0119] Optionally, the fourth determining module 404 is used for:

[0120] If the entity words exist in the preset word segmentation list, the target word segmentation is determined to be the same as the entity words, wherein the word segmentation list contains each word segmentation.

[0121] Obtain the preset part-of-speech weights, IDF weights, and entity weights;

[0122] Based on the part-of-speech score, part-of-speech weight, IDF score, IDF weight, entity score, and entity weight, a weighted sum is performed, and the weighted sum is used as the lexical weight of the target word.

[0123] Optionally, the fourth determining module 404 is used for:

[0124] When the entity word does not exist in the preset word segmentation list and the target word is the target word segmentation, obtain the preset part-of-speech weight and IDF weight;

[0125] The part-of-speech score, part-of-speech weight, IDF score, and IDF weight are weighted and summed, and the weighted sum is used as the lexical weight of the target word.

[0126] Optionally, the fourth determining module 404 is used for:

[0127] If the entity word does not exist in the preset word segmentation list and the target word is an entity word, obtain the preset entity weight;

[0128] The entity scores and entity weights are weighted and summed, and the sum is used as the word weight of the target word.

[0129] Optionally, the first determining module 401 is used for:

[0130] The query terms are segmented using a word segmentation tool, and each segmented term exists in a preset word segmentation list;

[0131] Entity recognition schemes are used to extract entities from query terms;

[0132] If the entity word exists in the preset word segmentation list, then the target word segmentation and the entity word are determined to be the same, and the target word includes either the target word segmentation or the entity word.

[0133] If the entity word does not exist in the preset word segmentation list, it is determined that the target word segmentation and the entity word are different, and the target word includes either the target word segmentation or the entity word.

[0134] Optionally, the second determining module 402 is used for:

[0135] The word segments in the query sentence are tagged with parts of speech using a preset part-of-speech tagging method.

[0136] Based on the pre-defined correspondence between the part-of-speech and part-of-speech scores of the target word, the part-of-speech score corresponding to the part-of-speech of the target word is determined.

[0137] Optionally, the second determining module 402 is used for:

[0138] If the target word is found to be in the preset IDF dictionary, the IDF score of the target word is determined according to the correspondence between word segments and IDF scores in the IDF dictionary.

[0139] If the target word is not found in the preset IDF dictionary, the preset IDF mean will be used as the IDF score of the target word.

[0140] Optionally, the device is also used for:

[0141] By using a preset normalization method, the word weights of each word in the query phrase are normalized to a preset range.

[0142] According to another aspect of the embodiments of this application, this application provides an electronic device, such as... Figure 5 As shown, the system includes a memory 503, a processor 501, a communication interface 502, and a communication bus 504. The memory 503 stores a computer program that can run on the processor 501. The memory 503 and the processor 501 communicate through the communication interface 502 and the communication bus 504. When the processor 501 executes the computer program, it implements the steps of the above method.

[0143] The memory and processor in the aforementioned electronic devices communicate with each other via a communication bus and a communication interface. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0144] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0145] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0146] According to another aspect of the embodiments of this application, a computer-readable medium having processor-executable non-volatile program code is also provided.

[0147] Optionally, in embodiments of this application, the computer-readable medium is configured to store program code for the processor to execute the above-described method:

[0148] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0149] In specific implementation, the embodiments of this application can be referred to the above embodiments and have corresponding technical effects.

[0150] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0151] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0152] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0153] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0154] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0155] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0156] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0157] If the aforementioned function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks. It should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0158] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for determining the weight of a word in a query phrase, characterized in that, The method comprises: acquiring a query sentence, and performing word segmentation and entity recognition on the query sentence to obtain each word segment and entity vocabulary in the query sentence; determining a target vocabulary in the query sentence, wherein the target vocabulary comprises a target word segment or an entity vocabulary, and the target word segment and the entity vocabulary are not necessarily the same; determining a part-of-speech score and an IDF score of the target word segment according to a preset correspondence between a word segment and a score; determining an entity score of the entity vocabulary according to an entity type of the entity vocabulary; determining a vocabulary weight of the target vocabulary by weighted summation according to at least one of the part-of-speech score, the IDF score and the entity score; wherein if the target vocabulary is both a target word segment and an entity vocabulary, the part-of-speech score, the IDF score and the entity score of the target vocabulary are weighted and summed to determine the vocabulary weight; if the target vocabulary is a target word segment but not an entity vocabulary, the part-of-speech score and the IDF score of the target vocabulary are weighted and summed to determine the vocabulary weight; if the target vocabulary is an entity vocabulary but not a target word segment, the entity score of the target vocabulary is weighted and summed to determine the vocabulary weight.

2. The method of claim 1, wherein, The determination of the vocabulary weight of the target vocabulary by weighted summation according to at least one of the part-of-speech score, the IDF score and the entity score comprises: in the case that the entity vocabulary exists in a preset word segment list, determining that the target word segment and the entity vocabulary are the same, wherein the word segment list contains each word segment; acquiring preset part-of-speech weight, IDF weight and entity weight; performing weighted summation according to the part-of-speech score, the part-of-speech weight, the IDF score, the IDF weight, the entity score and the entity weight, and taking the weighted summation result as the vocabulary weight of the target vocabulary.

3. The method of claim 1, wherein, The determination of the vocabulary weight of the target vocabulary by weighted summation according to at least one of the part-of-speech score, the IDF score and the entity score comprises: in the case that the entity vocabulary does not exist in a preset word segment list and the target vocabulary is the target word segment, acquiring preset part-of-speech weight and IDF weight; performing weighted summation according to the part-of-speech score, the part-of-speech weight, the IDF score and the IDF weight, and taking the weighted summation result as the vocabulary weight of the target vocabulary.

4. The method of claim 1, wherein, The determination of the vocabulary weight of the target vocabulary by weighted summation according to at least one of the part-of-speech score, the IDF score and the entity score comprises: in the case that the entity vocabulary does not exist in a preset word segment list and the target vocabulary is the entity vocabulary, acquiring preset entity weight; performing weighted summation according to the entity score and the entity weight, and taking the weighted summation result as the vocabulary weight of the target vocabulary.

5. The method of claim 1, wherein, The determination of the target vocabulary in the query sentence comprises: performing word segmentation on the query sentence by using a word segmentation tool, wherein each word segment exists in a preset word segment list; extracting entities in the query sentence by using an entity recognition scheme; If the entity word exists in the preset segmentation list, it is determined that the target segmentation and the entity word are the same, and the target word includes any one of the target segmentation and the entity word; If the entity word does not exist in the preset segmentation list, it is determined that the target segmentation and the entity word are not the same, and the target word includes the target segmentation or the entity word.

6. The method of claim 1, wherein, According to a preset corresponding relationship between the segmentation and the score, the part-of-speech score of the target segmentation is determined to include: The part-of-speech of the segmentation in the query sentence is tagged by using a preset part-of-speech tagging mode; According to a preset corresponding relationship between the part-of-speech of the segmentation and the part-of-speech score, the part-of-speech corresponding part-of-speech score of the target segmentation is determined.

7. The method of claim 1, wherein, According to a preset corresponding relationship between the segmentation and the score, the IDF score of the target segmentation is determined to include: If the target segmentation is located in the preset IDF dictionary, the IDF score of the target segmentation is determined according to a corresponding relationship between the segmentation and the IDF score in the IDF dictionary; If the target segmentation is not located in the preset IDF dictionary, a preset IDF average value is taken as the IDF score of the target segmentation.

8. The method according to any one of claims 1 to 4, characterized in that, After the word weight of the target word is determined by the weighted summation, the method further includes: The word weight of each word in the query sentence is normalized to a preset range by using a preset normalization operation mode.

9. A device for determining the weight of words in a query phrase, characterized in that, The device includes: A first determination module is configured to obtain a query sentence, segment and entity recognize the query sentence, obtain each segmentation and entity word in the query sentence, and determine a target word in the query sentence, wherein the target word includes a target segmentation or an entity word, and the target segmentation and the entity word are not necessarily the same; A second determination module is configured to determine a part-of-speech score and an IDF score of the target segmentation according to a preset corresponding relationship between the segmentation and the score; A third determination module is configured to determine an entity score of the entity word according to an entity type of the entity word; A fourth determination module is configured to determine a word weight of the target word by weighted summation according to at least one of the part-of-speech score, the IDF score and the entity score; If the target word is the target segmentation and the entity word, the part-of-speech score, the IDF score and the entity score of the target word are weighted and summed to determine the word weight; If the target word is the target segmentation and not the entity word, the part-of-speech score and the IDF score of the target word are weighted and summed to determine the word weight; If the target word is the entity word and not the target segmentation, the entity score of the target word is weighted and summed to determine the word weight. The device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; 10. An electronic device, comprising: The memory is configured to store a computer program; The processor is configured to execute the program stored in the memory to implement the method steps in any one of claims 1-8. The device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; The memory is configured to store a computer program; The processor is configured to execute the program stored in the memory to implement the method steps in any one of claims 1-8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method steps in any one of claims 1-8.

Images