Textual recall method and apparatus

Abstract

The present disclosure relates to the technical field of text recall, and provides a text recall method and device. The method comprises: expanding a query sentence by using a sentence expansion model to obtain an expanded sentence; preprocessing the query sentence and the expanded sentence respectively; performing new word discovery processing on the preprocessed query sentence and the expanded sentence respectively by using a Viterbi algorithm; performing word segmentation on the query sentence and the expanded sentence after new word discovery processing respectively by using a Jieba word segmentation algorithm to obtain first and second word segmentation results; marking the words in the first and second word segmentation results according to the word fields according to a domain word library; generating a search tree corresponding to the query sentence according to the query sentence, the expanded sentence, the preprocessed query sentence and the expanded sentence, the first and second word segmentation results, and the first and second word segmentation results after marking; and performing text recall according to the search tree to determine the query result of a target user.

Description

Technical Field

[0001] This disclosure relates to the field of text recall technology, and in particular to a text recall method and apparatus. Background Technology

[0002] Text recall is the process of matching the most relevant text information to a user based on their query. It's applied in various fields, such as product search and property search. Currently, common text recall methods are simply text matching, using search engines to retrieve results and ranking them according to TF-IDF scores before returning the ranking to the user. Common text recall methods divide the entire process into several major steps: intent understanding, multi-path recall, coarse ranking, fine ranking, and re-ranking. In multi-path recall, text TF-IDF recall is the most basic text hit path. However, due to the inaccuracy of the basic TF-IDF, it often produces many bad results in the search results. If subsequent fine ranking and re-ranking layers cannot handle these bad results, it will significantly impact the online user experience.

[0003] In realizing the present invention, the inventors discovered at least the following technical problems in the related technologies: the low accuracy of traditional text recall methods. Summary of the Invention

[0004] In view of this, the present disclosure provides a text recall method, apparatus, electronic device, and computer-readable storage medium to solve the problem of low accuracy in traditional text recall methods in the prior art.

[0005] A first aspect of this disclosure provides a text recall method, comprising: obtaining a query statement from a target user; expanding the query statement using a statement expansion model to obtain an expanded statement; preprocessing the query statement and the expanded statement respectively; performing new word discovery processing on the preprocessed query statement and the expanded statement respectively using the Viterbi algorithm; segmenting the query statement and the expanded statement after new word discovery processing respectively using the Jieba word segmentation algorithm to obtain a first word segmentation result and a second word segmentation result; labeling the words in the first word segmentation result and the second word segmentation result according to word domains based on a domain thesaurus; generating a search tree corresponding to the query statement based on the query statement, the expanded statement, the preprocessed query statement and the expanded statement, the first word segmentation result, the second word segmentation result, the labeled first word segmentation result and the second word segmentation result; and performing text recall based on the search tree to determine the query results of the target user.

[0006] A second aspect of this disclosure provides a text retrieval device, comprising: an expansion module configured to acquire a query statement from a target user and expand the query statement using a statement expansion model to obtain an expanded statement; a preprocessing module configured to preprocess the query statement and the expanded statement respectively; a new word module configured to perform new word discovery processing on the preprocessed query statement and the expanded statement respectively using a Viterbi algorithm; a word segmentation module configured to perform word segmentation on the query statement and the expanded statement after new word discovery processing respectively using a Jieba word segmentation algorithm to obtain a first word segmentation result and a second word segmentation result; an annotation module configured to annotate the words in the first word segmentation result and the second word segmentation result according to word domains based on a domain thesaurus; a generation module configured to generate a search tree corresponding to the query statement based on the query statement, the expanded statement, the preprocessed query statement and the expanded statement, the first word segmentation result, the second word segmentation result, and the annotated first word segmentation result and the second word segmentation result; and a retrieval module configured to perform text retrieval based on the search tree to determine the query results for the target user.

[0007] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0008] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0009] The beneficial effects of this embodiment compared to the prior art are as follows: This embodiment obtains the target user's query statement, expands the query statement using a statement expansion model to obtain an expanded statement; preprocesses both the query statement and the expanded statement; uses the Viterbi algorithm to perform new word discovery processing on the preprocessed query statement and the expanded statement; uses the Jieba word segmentation algorithm to segment the query statement and the expanded statement after new word discovery processing to obtain a first segmentation result and a second segmentation result; labels the words in the first and second segmentation results according to their respective word domains based on a domain thesaurus; generates a search tree corresponding to the query statement based on the query statement, the expanded statement, the preprocessed query statement and the expanded statement, the first segmentation result, the second segmentation result, and the labeled first and second segmentation results; and performs text retrieval based on the search tree to determine the target user's query results. Therefore, by employing the above technical means, the problem of low accuracy in traditional text retrieval methods in the prior art can be solved, thereby improving the accuracy of text retrieval. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure; Figure 2 This is a flowchart illustrating a text recall method provided in an embodiment of this disclosure; Figure 3 This is a schematic diagram of the structure of a text recall device provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0012] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0013] A text recall method and apparatus according to embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings.

[0014] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure. The application scenario may include terminal devices 101, 102, and 103, server 104, and network 105.

[0015] Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays that support communication with server 104, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. Terminal devices 101, 102, and 103 can be implemented as multiple software programs or software modules, or as a single software program or software module; this disclosure does not impose any limitations on this. Furthermore, various applications can be installed on terminal devices 101, 102, and 103, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.

[0016] Server 104 can be a server that provides various services, such as a backend server that receives requests sent by terminal devices with which it has established communication connections. This backend server can receive and analyze the requests sent by the terminal devices and generate processing results. Server 104 can be a single server, a server cluster consisting of several servers, or a cloud computing service center. This embodiment of the disclosure does not impose any limitations on these aspects.

[0017] It should be noted that server 104 can be either hardware or software. When server 104 is hardware, it can be various electronic devices that provide various services to terminal devices 101, 102, and 103. When server 104 is software, it can be multiple software programs or software modules that provide various services to terminal devices 101, 102, and 103, or it can be a single software program or software module that provides various services to terminal devices 101, 102, and 103. This disclosure does not limit the scope of the embodiments.

[0018] Network 105 can be a wired network using coaxial cable, twisted pair, and fiber optic connection, or it can be a wireless network that enables interconnection of various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), Infrared, etc. This disclosure does not limit the scope of the network.

[0019] Users can establish a communication connection with server 104 via network 105 through terminal devices 101, 102, and 103 to receive or send information, etc. It should be noted that the specific types, quantities, and combinations of terminal devices 101, 102, and 103, server 104, and network 105 can be adjusted according to the actual needs of the application scenario, and this disclosure embodiment does not impose any limitations on this.

[0020] Figure 2 This is a flowchart illustrating a text recall method provided in an embodiment of this disclosure. Figure 2 Text recall methods can be derived from Figure 1 The computer or server, or the software on the computer or server, executes the command. For example... Figure 2 As shown, the text recall method includes: S201, Obtain the query statement of the target user, expand the query statement using the statement expansion model, and obtain the expanded statement; S202, preprocess the query statement and the expanded statement respectively, and use the Viterbi algorithm to perform new word discovery processing on the preprocessed query statement and the expanded statement respectively; S203, the Jieba word segmentation algorithm is used to segment the query statement and the expanded statement after new word discovery processing to obtain the first word segmentation result and the second word segmentation result; S204, Based on the domain lexicon, the words in the first and second word segmentation results are labeled according to the word domain; S205. Generate a search tree corresponding to the query statement based on the query statement, the expanded statement, the preprocessed query statement and the expanded statement, the first word segmentation result, the second word segmentation result, and the first and second word segmentation results after annotation. S206, perform text recall based on the search tree to determine the query results for the target user.

[0021] The embodiments disclosed herein can be used in any text retrieval scenario, such as product search and property search. Product search and property search also involve matching query statements with textual descriptions of products or properties, thus they are also text retrieval scenarios. This disclosure uses product search as an example for explanation.

[0022] The Viterbi algorithm, also known as the Hidden Markov Model (HMM), is one of the three major algorithms for Hidden Markov Models. It's a special but widely used dynamic programming algorithm that solves the shortest path problem in any graph using dynamic programming, and can be used for new word discovery. New word discovery helps identify words that ordinary word segmentation cannot, essentially providing candidate words for subsequent segmentation.

[0023] The Jieba word segmentation algorithm, also known as the Jieba word segmentation algorithm, uses a prefix dictionary to achieve efficient word graph scanning, generating a directed acyclic graph (DAG) containing all possible word combinations for Chinese characters in a sentence. It then employs dynamic programming to find the maximum probability path and identify the maximum segmentation combination based on word frequency. For unregistered words, it uses an Hidden Markov Model (HMM) based on the word-forming ability of Chinese characters, employing the Viterbi algorithm. This disclosed embodiment utilizes the Jieba word segmentation algorithm to implement a multi-granularity word segmentation method.

[0024] According to the technical solution provided in this disclosure, the query statement of the target user is obtained, and the query statement is expanded using a statement expansion model to obtain an expanded statement; the query statement and the expanded statement are preprocessed respectively; the Viterbi algorithm is used to perform new word discovery processing on the preprocessed query statement and the expanded statement respectively; the Jieba word segmentation algorithm is used to segment the query statement and the expanded statement after new word discovery processing to obtain a first word segmentation result and a second word segmentation result; the words in the first word segmentation result and the second word segmentation result are labeled according to word domains based on the domain thesaurus; a search tree corresponding to the query statement is generated based on the query statement, the expanded statement, the preprocessed query statement and the expanded statement, the first word segmentation result, the second word segmentation result, and the labeled first word segmentation result and second word segmentation result; text retrieval is performed based on the search tree to determine the query result of the target user. Therefore, by adopting the above technical means, the problem of low accuracy of traditional text retrieval methods in the prior art can be solved, thereby improving the accuracy of text retrieval.

[0025] The query statement and the extended statement are preprocessed separately, including: case conversion, full-width / half-width conversion, traditional / simplified Chinese conversion, length truncation, word count truncation, and punctuation processing for the query statement and the extended statement respectively.

[0026] Based on the domain thesaurus, the words in the first and second word segmentation results are labeled according to the word domain, including: whether the words in the first and second word segmentation results are labeled as follows: whether it is a Chinese brand, whether it is an English brand, product category level, whether it is a core word, product alias, and virtual category.

[0027] The domain thesaurus stores the rules and related knowledge for word tagging. The domain thesaurus divides product categories into three levels: primary categories, secondary categories, and tertiary categories. Primary categories reflect the industry division of production and distribution, such as hardware, chemicals, food, and aquatic products. Secondary categories represent the general term for products with several common properties or characteristics; for example, food products can be further divided into vegetables and fruits, meat and meat products, milk and dairy products, eggs and egg products, etc. Tertiary categories are a further subdivision of these categories, reflecting specific product names. For example, alcoholic beverages are divided into baijiu (Chinese liquor), beer, wine, and fruit wine.

[0028] Product aliases are the descriptive names people use for a product in everyday life. The virtual category primarily targets virtual goods, such as games.

[0029] This disclosure provides an additional domain-specific thesaurus, which labels words according to their domains, thereby improving the accuracy of the final text retrieval.

[0030] Based on the query statement, expanded statement, preprocessed query statement and expanded statement, first word segmentation result, second word segmentation result, and annotated first word segmentation result and second word segmentation result, a search tree corresponding to the query statement is generated. This includes: using a first dynamic neural network model to process the words in the first word segmentation result and the second word segmentation result respectively to obtain a first processing result and a second processing result; generating a search tree corresponding to the query statement based on the query statement, expanded statement, preprocessed query statement and expanded statement, first word segmentation result, second word segmentation result, annotated first word segmentation result and second word segmentation result, first processing result and second processing result; wherein, the first dynamic neural network model has been trained and can be used for core word extraction, product category prediction, word weight analysis and word density analysis.

[0031] The dynamic neural network model is a NER model. Word weight analysis, similar to the TF-IDF method, is a method for evaluating the importance of a word. The first dynamic neural network model can use four models to respectively implement core word extraction, product category prediction, word weight analysis, and word density analysis. The model training methods in this regard are the same as existing methods and will not be elaborated further.

[0032] Based on the query statement, expanded statement, preprocessed query statement and expanded statement, first word segmentation result, second word segmentation result, and labeled first word segmentation result and second word segmentation result, a search tree corresponding to the query statement is generated. This includes: using a second dynamic neural network model to process the words in the first word segmentation result and the second word segmentation result respectively to obtain a third processing result and a fourth processing result; generating a search tree corresponding to the query statement based on the query statement, expanded statement, preprocessed query statement and expanded statement, first word segmentation result, second word segmentation result, labeled first word segmentation result and second word segmentation result, third processing result and fourth processing result; wherein, the second dynamic neural network model has been trained and can be used for intent classification and named entity recognition.

[0033] The first dynamic neural network model can use two models to implement intent classification and named entity recognition respectively. The training method for this model is the same as existing methods and will not be described in detail. Named entity recognition (NER), also known as proper name recognition, refers to the identification of entities with specific meanings in text, mainly including people's names, place names, organization names, proper nouns, etc.

[0034] Text retrieval based on the search tree is used to determine the query results for the target user, including: identifying the target object with the highest similarity to the search tree and identifying the target object as the query result for the target user.

[0035] Before expanding a query statement using a statement expansion model to obtain an expanded statement, the method further includes: training the statement expansion model so that the statement expansion model can process and obtain the embedding vector of the query statement, and querying the expanded statement that is similar to the query statement based on the embedding vector.

[0036] The statement expansion model can be any commonly used neural network model, and it queries expanded statements from a large-scale database.

[0037] For example, the query is "Ouyang Moumou's same style gold bar," and the expanded queries are "Ouyang Moumou's same style YSL" and "Ouyang Moumou's same style lipstick." There are no other changes to the preprocessing and new word discovery processing; the query and expanded queries after new word discovery processing remain "Ouyang Moumou's same style gold bar," "Ouyang Moumou's same style YSL," and "Ouyang Moumou's same style lipstick." The first word segmentation result is "Ouyang Moumou," "same style," and "gold bar." The second word segmentation result is "Ouyang Moumou," "same style," "YSL," and "Ouyang Moumou," "same style," and "lipstick." The words in the first and second word segmentation results are labeled according to their word domains, resulting in: "entities":[ {"type":"brand","tokens":["ysl"]}, {"type":"classNameL1":"tokens":[]}, {"type":"classNameL2":"tokens":[]}, {"type":"classNameL3":"tokens":["lipstick"]}, {"type":"tags":"tokens":[]}, {"type":"keywords":"tokens":["Ouyang Moumou","small gold bar","ysl","lipstick"]}, {"type":"alias":"tokens":["small gold bars"]} ] The search tree is { "originalQuery":"Same style gold bar as Ouyang Moumou!" "processedQuery":"Ouyang Moumou's same style small gold bar", "segments":["Ouyang Moumou","same style","small gold bar"], "rewriteQueries":[ {"query":"Ouyang Moumou's same style small gold bar", "processedQuery":"Ouyang Moumou's same style small gold bar", "segments":["Ouyang Moumou","same style","small gold bar"], "rewriteType":"queryCorrect" }, {"query":"Ouyang Moumou's YSL style", "processedQuery":"Ouyang Moumou's same style YSL", "segments":["Ouyang Moumou","same style","ysl"], "rewriteType":"synonymReplace" }, {"query":"Ouyang Moumou's lipstick", "processedQuery":"Ouyang Moumou's lipstick", "segments":["Ouyang Moumou","same style","lipstick"], "rewriteType":"synonymReplace" } ], "entities":[ {"type":"brand","tokens":["ysl"]}, {"type":"classNameL1","tokens":[]}, {"type":"classNameL2","tokens":[]}, {"type":"classNameL3","tokens":["lipstick"]}, {"type":"tags","tokens":[]}, {"type":"keywords","tokens":["Ouyang Moumou","small gold bar","ysl","lipstick"]}, {"type":"alias","tokens":["small gold bars"]} ], "embedding":[ } All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0038] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0039] Figure 3 This is a schematic diagram of a text recall device provided in an embodiment of this disclosure. Figure 3 As shown, the text recall device includes: The extension module 301 is configured to obtain the query statement of the target user, and expand the query statement using the statement expansion model to obtain the expanded statement; The preprocessing module 302 is configured to preprocess the query statement and the expanded statement respectively, and to use the Viterbi algorithm to perform new word discovery processing on the preprocessed query statement and the expanded statement respectively; The word segmentation module 303 is configured to use the Jieba word segmentation algorithm to segment the query statement and the expanded statement after the new word discovery process, respectively, to obtain the first word segmentation result and the second word segmentation result. The annotation module 304 is configured to annotate the words in the first word segmentation result and the second word segmentation result according to the word domain based on the domain lexicon; The generation module 305 is configured to generate a search tree corresponding to the query statement based on the query statement, the expanded statement, the preprocessed query statement and the expanded statement, the first word segmentation result, the second word segmentation result, and the first word segmentation result and the second word segmentation result after annotation. The recall module 306 is configured to perform text recall based on the search tree to determine the query results for the target user.

[0040] This disclosure applies to the field of text-based product recall. Products include: electronic goods, online advertisements, online rental room resources, and online home purchase room resources, etc. Target user information includes: the model of the target user's mobile phone or computer, or other electronic devices the target user is using, and the target user's location, etc. Target product information includes: data information such as the price and sales volume of the target product. Interaction between the target product and the target user refers to actions such as clicking, saving, and browsing the target product's webpage, link, or window. It should be noted that the term "target" in "target user," "target user information," "target product," and "target product information" is only used for distinction and has no other meaning. For example, in the following text, "user" refers to the object used in training the model, while "target user" refers to the object to which products are recommended.

[0041] The embodiments disclosed herein can be used in any text retrieval scenario, such as product search and property search. Product search and property search also involve matching query statements with textual descriptions of products or properties, thus they are also text retrieval scenarios. This disclosure uses product search as an example for explanation.

[0042] The Viterbi algorithm, also known as the Hidden Markov Model (HMM), is one of the three major algorithms for Hidden Markov Models. It's a special but widely used dynamic programming algorithm that solves the shortest path problem in any graph using dynamic programming, and can be used for new word discovery. New word discovery helps identify words that ordinary word segmentation cannot, essentially providing candidate words for subsequent segmentation.

[0043] The Jieba word segmentation algorithm, also known as the Jieba word segmentation algorithm, uses a prefix dictionary to achieve efficient word graph scanning, generating a directed acyclic graph (DAG) containing all possible word combinations for Chinese characters in a sentence. It then employs dynamic programming to find the maximum probability path and identify the maximum segmentation combination based on word frequency. For unregistered words, it uses an Hidden Markov Model (HMM) based on the word-forming ability of Chinese characters, employing the Viterbi algorithm. This disclosed embodiment utilizes the Jieba word segmentation algorithm to implement a multi-granularity word segmentation method.

[0044] According to the technical solution provided in this disclosure, the query statement of the target user is obtained, and the query statement is expanded using a statement expansion model to obtain an expanded statement; the query statement and the expanded statement are preprocessed respectively; the Viterbi algorithm is used to perform new word discovery processing on the preprocessed query statement and the expanded statement respectively; the Jieba word segmentation algorithm is used to segment the query statement and the expanded statement after new word discovery processing to obtain a first word segmentation result and a second word segmentation result; the words in the first word segmentation result and the second word segmentation result are labeled according to word domains based on the domain thesaurus; a search tree corresponding to the query statement is generated based on the query statement, the expanded statement, the preprocessed query statement and the expanded statement, the first word segmentation result, the second word segmentation result, and the labeled first word segmentation result and second word segmentation result; text retrieval is performed based on the search tree to determine the query result of the target user. Therefore, by adopting the above technical means, the problem of low accuracy of traditional text retrieval methods in the prior art can be solved, thereby improving the accuracy of text retrieval.

[0045] Optionally, the preprocessing module 302 is also configured to perform case conversion, full-width / half-width conversion, simplified / traditional character conversion, length truncation, character count truncation, and punctuation processing on the query statement and the extended statement, respectively.

[0046] Optionally, the annotation module 304 is also configured to annotate the words in the first and second word segmentation results as follows: whether it is a Chinese brand, whether it is an English brand, product category level, whether it is a core word, product alias, and virtual category.

[0047] The domain thesaurus stores the rules and related knowledge for word tagging. The domain thesaurus divides product categories into three levels: primary categories, secondary categories, and tertiary categories. Primary categories reflect the industry division of production and distribution, such as hardware, chemicals, food, and aquatic products. Secondary categories represent the general term for products with several common properties or characteristics; for example, food products can be further divided into vegetables and fruits, meat and meat products, milk and dairy products, eggs and egg products, etc. Tertiary categories are a further subdivision of these categories, reflecting specific product names. For example, alcoholic beverages are divided into baijiu (Chinese liquor), beer, wine, and fruit wine.

[0048] Product aliases are the descriptive names people use for a product in everyday life. The virtual category primarily targets virtual goods, such as games.

[0049] This disclosure provides an additional domain-specific thesaurus, which labels words according to their domains, thereby improving the accuracy of the final text retrieval.

[0050] Optionally, the generation module 305 is further configured to use a first dynamic neural network model to process the words in the first word segmentation result and the second word segmentation result respectively, to obtain a first processing result and a second processing result; and to generate a search tree corresponding to the query statement based on the query statement, the expanded statement, the preprocessed query statement and the expanded statement, the first word segmentation result, the second word segmentation result, the labeled first word segmentation result and the second word segmentation result, the first processing result and the second processing result; wherein, the first dynamic neural network model has been trained and can be used for core word extraction, product category prediction, word weight analysis and word density analysis.

[0051] The dynamic neural network model is a NER model. Word weight analysis, similar to the TF-IDF method, is a method for evaluating the importance of a word. The first dynamic neural network model can use four models to respectively implement core word extraction, product category prediction, word weight analysis, and word density analysis. The model training methods in this regard are the same as existing methods and will not be elaborated further.

[0052] Optionally, the generation module 305 is further configured to use a second dynamic neural network model to process the words in the first and second word segmentation results respectively, to obtain a third and a fourth processing result; and to generate a search tree corresponding to the query statement based on the query statement, the expanded statement, the preprocessed query statement and the expanded statement, the first word segmentation result, the second word segmentation result, the labeled first and second word segmentation results, the third processing result and the fourth processing result; wherein the second dynamic neural network model has been trained and can be used for intent classification and named entity recognition.

[0053] The first dynamic neural network model can use two models to implement intent classification and named entity recognition respectively. The training method for this model is the same as existing methods and will not be described in detail. Named entity recognition (NER), also known as proper name recognition, refers to the identification of entities with specific meanings in text, mainly including people's names, place names, organization names, proper nouns, etc.

[0054] Optionally, the recall module 306 is also configured to identify the target object with the highest similarity to the search tree and determine the target object as the query result of the target user.

[0055] Optionally, the extension module 301 is further configured to train the statement extension model so that the statement extension model can process and obtain the embedding vector of the query statement, and query the extended statement that is similar to the query statement based on the embedding vector.

[0056] The statement expansion model can be any commonly used neural network model, and it queries expanded statements from a large-scale database.

[0057] For example, the query is "Ouyang Moumou's same style gold bar," and the expanded queries are "Ouyang Moumou's same style YSL" and "Ouyang Moumou's same style lipstick." There are no other changes to the preprocessing and new word discovery processing; the query and expanded queries after new word discovery processing remain "Ouyang Moumou's same style gold bar," "Ouyang Moumou's same style YSL," and "Ouyang Moumou's same style lipstick." The first word segmentation result is "Ouyang Moumou," "same style," and "gold bar." The second word segmentation result is "Ouyang Moumou," "same style," "YSL," and "Ouyang Moumou," "same style," and "lipstick." The words in the first and second word segmentation results are labeled according to their word domains, resulting in: "entities":[ {"type":"brand","tokens":["ysl"]}, {"type":"classNameL1":"tokens":[]}, {"type":"classNameL2":"tokens":[]}, {"type":"classNameL3":"tokens":["lipstick"]}, {"type":"tags":"tokens":[]}, {"type":"keywords":"tokens":["Ouyang Moumou","small gold bar","ysl","lipstick"]}, {"type":"alias":"tokens":["small gold bars"]} ] The search tree is { "originalQuery":"Same style gold bar as Ouyang Moumou!" "processedQuery":"Ouyang Moumou's same style small gold bar", "segments":["Ouyang Moumou","same style","small gold bar"], "rewriteQueries":[ {"query":"Ouyang Moumou's same style small gold bar", "processedQuery":"Ouyang Moumou's same style small gold bar", "segments":["Ouyang Moumou","same style","small gold bar"], "rewriteType":"queryCorrect" }, {"query":"Ouyang Moumou's YSL style", "processedQuery":"Ouyang Moumou's same style YSL", "segments":["Ouyang Moumou","same style","ysl"], "rewriteType":"synonymReplace" }, {"query":"Ouyang Moumou's lipstick", "processedQuery":"Ouyang Moumou's lipstick", "segments":["Ouyang Moumou","same style","lipstick"], "rewriteType":"synonymReplace" } ], "entities":[ {"type":"brand","tokens":["ysl"]}, {"type":"classNameL1","tokens":[]}, {"type":"classNameL2","tokens":[]}, {"type":"classNameL3","tokens":["lipstick"]}, {"type":"tags","tokens":[]}, {"type":"keywords","tokens":["Ouyang Moumou","small gold bar","ysl","lipstick"]}, {"type":"alias","tokens":["small gold bars"]} ], "embedding":[ } It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0058] Figure 4 This is a schematic diagram of the electronic device 4 provided in an embodiment of this disclosure. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, it implements the steps in the various method embodiments described above. Alternatively, when the processor 401 executes the computer program 403, it implements the functions of each module / unit in the various device embodiments described above.

[0059] Electronic device 4 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 4 may include, but is not limited to, processor 401 and memory 402. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 4 and does not constitute a limitation on electronic device 4. It may include more or fewer components than shown, or different components.

[0060] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0061] The memory 402 can be an internal storage unit of the electronic device 4, such as a hard disk or RAM of the electronic device 4. The memory 402 can also be an external storage device of the electronic device 4, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 4. The memory 402 can also include both internal and external storage units of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device.

[0062] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0063] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in a computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0064] The above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit it. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be included within the protection scope of this disclosure.

Claims

1. A text recall method, characterized in that, include: Obtain the query statement of the target user, and expand the query statement using the statement expansion model to obtain the expanded statement; The query statement and the expanded statement are preprocessed respectively. The Viterbi algorithm is used to perform new word discovery processing on the preprocessed query statement and the expanded statement respectively. The preprocessing includes case conversion, full-width and half-width conversion, traditional and simplified character conversion, length truncation, truncation according to the number of characters, and punctuation processing. The purpose of the new word discovery processing is to find words that cannot be segmented by ordinary word segmentation to provide candidate words for word segmentation. The Jieba word segmentation algorithm was used to segment the query statement and the expanded statement after the new word discovery process, respectively, to obtain the first word segmentation result and the second word segmentation result. Based on the domain thesaurus, the words in the first and second word segmentation results are labeled according to the word domain as follows: whether it is a Chinese brand, whether it is an English brand, product category level, whether it is a core word, product alias, and virtual category; Based on the query statement, the expanded statement, the preprocessed query statement and expanded statement, the first word segmentation result, the second word segmentation result, and the labeled first word segmentation result and second word segmentation result, a search tree corresponding to the query statement is generated. Text retrieval is performed based on the search tree to determine the query results for the target user. Based on the query statement, the expanded statement, the preprocessed query statement and expanded statement, the first word segmentation result, the second word segmentation result, and the labeled first word segmentation result and second word segmentation result, a search tree corresponding to the query statement is generated, including: The words in the first and second word segmentation results are processed using a first dynamic neural network model to obtain a first processing result and a second processing result. Based on the query statement, the expanded statement, the preprocessed query statement and expanded statement, the first word segmentation result, the second word segmentation result, the labeled first and second word segmentation results, the first processing result, and the second processing result, a search tree corresponding to the query statement is generated. The first dynamic neural network model has been trained, and four models are used to implement core word extraction, product category prediction, word weight analysis, and word density analysis, respectively. or, The words in the first and second word segmentation results are processed using a second dynamic neural network model to obtain a third and a fourth processing result. Based on the query statement, the expanded statement, the preprocessed query statement and expanded statement, the first word segmentation result, the second word segmentation result, the labeled first and second word segmentation results, the third processing result, and the fourth processing result, a search tree corresponding to the query statement is generated. The second dynamic neural network model has been trained, and two models are used to implement intent classification and named entity recognition, respectively.

2. The method according to claim 1, characterized in that, Text retrieval based on the search tree to determine the query results for the target user includes: The target object with the highest similarity to the search tree is identified, and the target object is identified as the query result of the target user.

3. The method according to claim 1, characterized in that, Before expanding the query statement using a statement expansion model to obtain the expanded statement, the method further includes: The statement expansion model is trained so that it can process and obtain the embedding vector of the query statement, and query the expanded statement that is similar to the query statement based on the embedding vector.

4. A text recall device, characterized in that, include: The extension module is configured to obtain the query statement of the target user, and expand the query statement using the statement expansion model to obtain the expanded statement; The preprocessing module is configured to preprocess the query statement and the expanded statement respectively, and to use the Viterbi algorithm to perform new word discovery processing on the preprocessed query statement and the expanded statement respectively. The preprocessing includes case conversion, full-width / half-width conversion, simplified / traditional character conversion, length truncation, truncation according to the number of characters, and punctuation processing. The purpose of the new word discovery processing is to find words that cannot be segmented by ordinary word segmentation to provide candidate words for word segmentation. The word segmentation module is configured to use the Jieba word segmentation algorithm to segment the query statement and the expanded statement after the new word discovery processing, respectively, to obtain the first word segmentation result and the second word segmentation result. The annotation module is configured to annotate the words in the first and second word segmentation results according to the word domain based on the domain thesaurus as follows: whether it is a Chinese brand, whether it is an English brand, product category level, whether it is a core word, product alias, and virtual category; The generation module is configured to generate a search tree corresponding to the query statement based on the query statement, the expanded statement, the preprocessed query statement and expanded statement, the first word segmentation result, the second word segmentation result, and the labeled first word segmentation result and second word segmentation result. The recall module is configured to perform text recall based on the search tree to determine the query results for the target user. The generation module is specifically configured to: process the words in the first word segmentation result and the second word segmentation result using a first dynamic neural network model to obtain a first processing result and a second processing result; generate a search tree corresponding to the query statement based on the query statement, the expanded statement, the preprocessed query statement and the expanded statement, the first word segmentation result, the second word segmentation result, the labeled first word segmentation result and the second word segmentation result, the first processing result and the second processing result; wherein, the first dynamic neural network model has been trained for core word extraction, product category prediction, word weight analysis and word density analysis; Alternatively, the generation module is specifically configured to: process the words in the first and second word segmentation results using a second dynamic neural network model to obtain a third and a fourth processing result; generate a search tree corresponding to the query statement based on the query statement, the expanded statement, the preprocessed query statement and expanded statement, the first word segmentation result, the second word segmentation result, the labeled first and second word segmentation results, the third processing result and the fourth processing result; wherein the second dynamic neural network model has been trained for intent classification and named entity recognition.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 3.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 3.

Images