Search engine word segmentation model construction method and related device

Abstract

The embodiment of the application belongs to the technical field of artificial intelligence, is applied to the technical field of search engine word segmentation processing, relates to a search engine word segmentation model construction method and related equipment thereof, and comprises the following steps: obtaining search data of historical search users; adopting an N-Gram algorithm and a PMI new word discovery mechanism to construct a word segmentation dictionary; obtaining a word segmentation set corresponding to each search field according to the word segmentation dictionary; dividing word segmentation in the word segmentation set into two categories of important class word segmentation and ordinary class word segmentation based on an IDF algorithm; constructing a model input triple; training an initialized search engine word segmentation model by taking the model input triple as a model input value to obtain a trained search engine word segmentation model, and completing construction of the search engine word segmentation model. The search engine word segmentation model is constructed in a PMI, N-Gram algorithm, IDF inverse document frequency and supervised contrast learning manner, so that the search results recalled during search are more accurate.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method for constructing a word segmentation model for a search engine and related equipment. Background Technology

[0002] The goal of a search engine is to retrieve the most valuable information for the user from available online resources based on the user's request. Search engines are mainly divided into three major modules: semantic understanding, recall, and ranking. The word weighting module is used in the semantic understanding stage, which mainly performs word segmentation on the words or phrases in the user's query field.

[0003] In existing technologies, word segmentation processing in search engines is implemented using two methods: static word weighting and dynamic word weighting. Static word weighting is generally calculated using methods such as TF-IDF / BM25, which cannot adapt to contextual changes and therefore performs poorly. Dynamic word weighting mainly employs deep semantic models such as DeepCT / DeepRT. DeepCT directly uses the recall weight of words to obtain the target score for each word, and then uses modeling to fit the score distribution of all segmented words in the entire search field, increasing the complexity of modeling and potentially leading to the cumulative propagation of errors. Therefore, existing technologies still suffer from inaccurate search results when performing search retrieval. Summary of the Invention

[0004] The purpose of this application is to propose a method for constructing a word segmentation model for a search engine and related equipment, so as to solve the problem that search results are not accurate enough when performing search retrieval in the prior art.

[0005] To address the aforementioned technical problems, this application provides a method for constructing a search engine word segmentation model, employing the following technical solution:

[0006] A method for constructing a word segmentation model for a search engine includes the following steps:

[0007] Step 201: Obtain search data from historical search users, wherein the search data includes historical search fields and search return logs;

[0008] Step 202: Use the N-Gram algorithm and PMI word discovery mechanism to perform word segmentation prediction on each search field in the historical search fields, obtain the word segmentation prediction results, and construct a word segmentation dictionary based on the word segmentation prediction results corresponding to all historical search fields;

[0009] Step 203: Perform word segmentation on each search field in the historical search fields according to the word segmentation dictionary, and obtain the word segmentation set corresponding to each search field;

[0010] Step 204: Based on the search return log, the preset IDF algorithm, and the preset classification threshold, divide the word segments in the word segmentation set into two categories: important word segments and ordinary word segments;

[0011] Step 205: Obtain the important class segmentation and ordinary class segmentation corresponding to each search field, and construct the model input triplet, wherein the format of the model input triplet is: [current search field: important class segmentation in the current search field: ordinary class segmentation in the current search field];

[0012] Step 206: Obtain the model input triplet corresponding to each search field as the model input value to train the initialized search engine word segmentation model, and obtain the trained search engine word segmentation model, thus completing the construction of the search engine word segmentation model.

[0013] Furthermore, the step of using the N-Gram algorithm and PMI word discovery mechanism to perform word segmentation prediction for each search field in the historical search fields specifically includes:

[0014] The N-Gram algorithm is used to filter out the pause words in each search field of the historical search fields, and the pause words are then deleted.

[0015] The PMI word discovery mechanism is used to filter out consecutively occurring words in each search field of the historical search fields, and it is determined whether the consecutive number of consecutive occurrences of the words is greater than a preset number threshold.

[0016] If the number of consecutive occurrences exceeds a preset threshold, the consecutively occurring words are marked and treated as a word segmentation field.

[0017] If the number of consecutive occurrences is not greater than a preset threshold, then the consecutively occurring words will be used as different word segmentation fields.

[0018] Furthermore, the search return log includes identification information of clicked documents and identification information of unclicked documents. Before performing the step of dividing the word segments in the word segmentation set into two categories, important word segments and ordinary word segments, based on the search return log, a preset IDF algorithm, and a preset classification threshold, the method further includes:

[0019] Based on the identification information of the clicked document and the identification information of the unclicked document, obtain the clicked document and the unclicked document.

[0020] Furthermore, the step of dividing the word segments in the word segmentation set into two categories—important word segments and ordinary word segments—based on the search return logs, a preset IDF algorithm, and a preset classification threshold specifically includes:

[0021] The inverse document frequency (IDF) of each word in the word segmentation set in the exposed clicked document is obtained by using the IDF algorithm and is denoted as the first inverse document frequency.

[0022] The inverse document frequency (IDF) of each word in the word segmentation set in the exposed but unclicked document is obtained using the IDF algorithm and is denoted as the second inverse document frequency.

[0023] Based on the preset absolute value algorithm, the first inverse document frequency and the second inverse document frequency, the frequency difference of each word segment between the exposed clicked document and the exposed unclicked document is obtained;

[0024] Based on the frequency difference and the preset frequency difference threshold, the word segmentation set is divided into two categories: important word segmentation and ordinary word segmentation.

[0025] Furthermore, the step of obtaining the inverse document frequency (IVF) of each word in the segmentation set in the exposed clicked document using the IDF algorithm, denoted as the first IDF, specifically includes:

[0026] According to the preset IDF algorithm formula: Obtain the inverse document frequency (INV) of each segment in the segmentation set within the exposed clicked document, where j represents the segmentation number in the segmentation set, and D represents the total number of exposed clicked documents and exposed unclicked documents. 1, This indicates the number of documents that were clicked during the exposure.

[0027] Furthermore, the step of obtaining the inverse document frequency (IDF) of each word in the segmentation set in the exposed unclicked document using the IDF algorithm, denoted as the second inverse document frequency, specifically includes:

[0028] According to the preset IDF algorithm formula: Obtain the inverse document frequency (IVF) of each segment in the segmentation set within the exposed but unclicked documents, where j represents the segmentation number in the segmentation set, and D represents the total number of exposed clicked documents and exposed unclicked documents. 2, This indicates the number of documents that were exposed but not clicked.

[0029] Furthermore, the step of obtaining the frequency difference between the exposed clicked documents and the exposed unclicked documents based on a preset absolute value algorithm, the first inverse document frequency, and the second inverse document frequency specifically includes:

[0030] According to the preset absolute value algorithm formula: L=|idf 1, -df 2,j |, obtain the frequency difference of each word between the clicked document and the unclicked document, where j represents the number of each word in the word set, idf 1, The first inverse document frequency (IDF) of the segment with ID j. 2, This represents the second inverse document frequency of the segment with ID j.

[0031] Furthermore, the step of dividing the word segmentation set into two categories—important word segmentation and ordinary word segmentation—based on the frequency difference and a preset frequency difference threshold specifically includes:

[0032] Determine whether the frequency difference is greater than a preset frequency difference threshold;

[0033] If the frequency difference is greater than the preset frequency difference threshold, then the word segment corresponding to the frequency difference is obtained, and the word segment is cached in the pre-constructed important class word segmentation set;

[0034] If the frequency difference is not greater than the preset frequency difference threshold, then the word segment corresponding to the frequency difference is obtained, and the word segment is cached in the pre-constructed ordinary word segmentation set.

[0035] Furthermore, the step of obtaining the model input triplet corresponding to each search field as the model input value to train the initialized search engine word segmentation model, thereby obtaining the trained search engine word segmentation model and completing the construction of the search engine word segmentation model, specifically includes:

[0036] Based on the cosine similarity algorithm and the model input triplet corresponding to each search field, the similarity between each search field and its corresponding important class segmentation and the similarity between the important class segmentation in each search field and each ordinary class segmentation in each search field are obtained.

[0037] Based on the preset loss function: Obtain the loss value of each search field in the word segmentation model of the search engine, where exp(q·k) + / τ) represents the similarity between the current search field and its corresponding important word segmentation, exp(q·k) i / τ) represents the similarity between important word segments in the current search field and each ordinary word segment in the current search field, k i Let k represent the i-th ordinary class word segmentation. + τ represents the important class of word segmentation, k represents the number of words in the word segmentation set, and τ represents a preset hyperparameter used to control the model's ability to distinguish ordinary class words.

[0038] Count the number of loss values ​​that satisfy the preset loss threshold;

[0039] Determine whether the number of loss values ​​that satisfy a preset loss threshold is greater than a preset number threshold;

[0040] If the number of loss values ​​that satisfy the preset loss threshold is not greater than the preset number threshold, then the word segments in the word segmentation set are re-divided by adjusting the frequency difference threshold to obtain important class words and ordinary class words after re-division.

[0041] Steps 205 to 206 are executed again to build and update the search engine word segmentation model until the number of loss values ​​that meet the preset loss threshold is greater than the preset number threshold, at which point the search engine word segmentation model is completed.

[0042] To address the aforementioned technical problems, this application also provides a search engine word segmentation model construction device, which employs the following technical solution:

[0043] A device for constructing a word segmentation model for a search engine, comprising:

[0044] The search data acquisition module is used to acquire search data of historical search users, wherein the search data includes historical search fields and search return logs;

[0045] The word segmentation dictionary construction module is used to perform word segmentation prediction on each search field in the historical search fields using the N-Gram algorithm and PMI word discovery mechanism, obtain word segmentation prediction results, and construct a word segmentation dictionary based on the word segmentation prediction results corresponding to all historical search fields.

[0046] The word segmentation set acquisition module is used to segment each search field in the historical search fields according to the word segmentation dictionary, and obtain the word segmentation set corresponding to each search field.

[0047] The word segmentation category division module is used to divide the word segments in the word segmentation set into two categories: important word segments and ordinary word segments based on the search return log, the preset IDF algorithm, and the preset classification threshold.

[0048] The triplet construction module is used to obtain the important class segmentation and ordinary class segmentation corresponding to each search field, and construct the model input triplet. The format of the model input triplet is: [current search field: important class segmentation in the current search field: ordinary class segmentation in the current search field];

[0049] A word segmentation model construction module, configured to obtain a model input triplet corresponding to each of the search fields respectively as a model input value to train an initialized search engine word segmentation model, so as to obtain a trained search engine word segmentation model and complete the construction of the search engine word segmentation model.

[0050] To solve the above technical problem, the embodiments of the present application further provide a computer device, which adopts the following technical solutions as described below:

[0051] A computer device comprises a memory and a processor, wherein computer-readable instructions are stored in the memory, and when the processor executes the computer-readable instructions, the steps of the above-mentioned search engine word segmentation model construction method are implemented.

[0052] To solve the above technical problem, the embodiments of the present application further provide a computer-readable storage medium, which adopts the following technical solutions as described below:

[0053] A computer-readable storage medium, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by a processor, the steps of the above-mentioned search engine word segmentation model construction method are implemented.

[0054] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects:

[0055] The search engine word segmentation model construction method according to the embodiments of the present application adopts a combination of N-Gram, PMI and IDF for word annotation. Compared with the DeepCTR model for word annotation, it has the following advantages: when performing word annotation, the DeepCTR model only considers the word frequency of words in related documents, and will also mark the weights of words irrelevant to the core semantics in the search fields as high importance, which will lead to the recall of some results irrelevant to user intent. In the search field word segmentation stage using N-Gram and PMI, not only new words in the search field can be identified through the new word discovery mechanism of PMI, for example, the new word "iPhone" combined from "apple" and "phone", but also stop words in the search field can be filtered out through the N-Gram algorithm, such as predicates (de, di) and auxiliary words (a), etc. Then, by combining the calculation mode of documents and IDF inverse document frequency, a more accurate word annotation result is jointly predicted, so that the search results recalled during search are more accurate. Brief Description of the Drawings

[0056] To illustrate the solutions in the present application more clearly, the accompanying drawings required in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application, and for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0057] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0058] Figure 2 A flowchart of an embodiment of the search engine word segmentation model construction method according to this application;

[0059] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 204 shown.

[0060] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 206 shown;

[0061] Figure 5 A schematic diagram of the structure of an embodiment of the search engine word segmentation model construction apparatus according to this application;

[0062] Figure 6 A schematic diagram of the structure of an embodiment of the computer device according to this application. Detailed Implementation

[0063] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0064] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0065] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0066] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0067] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0068] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0069] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0070] It should be noted that the search engine word segmentation model construction method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the search engine word segmentation model construction device is generally set in the server / terminal device.

[0071] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0072] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of the search engine word segmentation model construction method according to this application. The search engine word segmentation model construction method includes the following steps:

[0073] Step 201: Obtain search data from historical search users.

[0074] In this embodiment, the search data includes historical search fields and search return logs.

[0075] In this embodiment, the search return log includes identification information of exposed and clicked documents and identification information of exposed but unclicked documents.

[0076] Step 202: Perform word segmentation prediction on each search term in the historical search fields using the N-Gram algorithm and the PMI word discovery mechanism to obtain a word segmentation prediction result, and construct a word segmentation dictionary according to the word segmentation prediction results corresponding to all historical search fields.

[0077] In this embodiment, the step of performing word segmentation prediction on each search field in the historical search fields using the N-Gram algorithm and the PMI word discovery mechanism specifically includes: filtering out pause words in each search field in the historical search fields using the N-Gram algorithm, and deleting the pause words; filtering out consecutively appearing words in each search field in the historical search fields using the PMI word discovery mechanism, and determining whether the consecutive occurrence count of the consecutively appearing words is greater than a preset count threshold; if the consecutive occurrence count is greater than the preset count threshold, marking the consecutively appearing words and taking the consecutively appearing words as one word segmentation field; if the consecutive occurrence count is not greater than the preset count threshold, taking the consecutively appearing words as different word segmentation fields respectively.

[0078] Filtering out pause words such as modal particles and predicates through the N-Gram algorithm further simplifies the search fields and improves search efficiency. The N-Gram model is an N-gram language model. Meanwhile, the PMI (pointwise mutual information) word discovery mechanism is used for new word segmentation combinations, for example, "Ping An" and "Technology" respectively represent different meanings, but in many insurance scenarios, "Ping An Technology" can be used as a single word segmentation, representing a company name; "Xiao Mi" and "Mobile Phone" respectively represent different word segmentation meanings, but in the field of mobile phone brands, "Xiao Mi Mobile Phone" can also be used as a separate word segmentation, representing a mobile phone brand. Performing word segmentation prediction using the N-Gram algorithm and the PMI word discovery mechanism makes the word segmentation results more accurate, and ensures the accuracy of search recall results.

[0079] Step 203: Perform word segmentation on each search field in the historical search fields according to the word segmentation dictionary, to obtain a word segmentation set corresponding to each search field respectively.

[0080] Step 204: Classify the word segments in the word segmentation set into two categories, namely important word segments and ordinary word segments, based on the search return log, a preset IDF algorithm and a preset classification threshold.

[0081] In this embodiment, before performing the step of dividing the word segments in the word segmentation set into two categories, important word segments and ordinary word segments, based on the search return log, the preset IDF algorithm, and the preset classification threshold, the method further includes: obtaining the exposed clicked documents and the exposed unclicked documents according to the identification information of the exposed clicked documents and the identification information of the exposed unclicked documents.

[0082] IDF (Inverse Document Frequency) represents the inverse document frequency, and the IDF algorithm represents the algorithm for obtaining the inverse document frequency of segmented words.

[0083] Continue to refer to Figure 3 , Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 204 shown includes:

[0084] Step 301: Use the IDF algorithm to obtain the inverse document frequency of each word in the word segmentation set in the exposed clicked document, and record it as the first inverse document frequency;

[0085] In this embodiment, the step of obtaining the inverse document frequency (IRF) of each word in the segmentation set in the exposed clicked document using the IDF algorithm, denoted as the first IRF, specifically includes: according to the preset IDF algorithm formula: Obtain the inverse document frequency (INV) of each segment in the segmentation set within the exposed clicked document, where j represents the segmentation number in the segmentation set, and D represents the total number of exposed clicked documents and exposed unclicked documents. 1, This indicates the number of documents that were clicked during the exposure.

[0086] Step 302: Use the IDF algorithm to obtain the inverse document frequency of each word in the word segmentation set in the exposed unclicked document, and record it as the second inverse document frequency;

[0087] In this embodiment, the step of obtaining the inverse document frequency (IDF) of each word in the segmentation set in the exposed unclicked document using the IDF algorithm, denoted as the second IDF, specifically includes: according to the preset IDF algorithm formula: Obtain the inverse document frequency (IVF) of each segment in the segmentation set within the exposed but unclicked documents, where j represents the segmentation number in the segmentation set, and D represents the total number of exposed clicked documents and exposed unclicked documents. 2, This indicates the number of documents that were exposed but not clicked.

[0088] Step 303: Based on the preset absolute value algorithm, the first inverse document frequency and the second inverse document frequency, obtain the frequency difference of each word segment between the exposed clicked document and the exposed unclicked document;

[0089] In this embodiment, the step of obtaining the frequency difference between the exposed clicked documents and the exposed unclicked documents based on a preset absolute value algorithm, the first inverse document frequency, and the second inverse document frequency specifically includes: according to the preset absolute value algorithm formula: L=|idf 1, -df 2, |, obtain the frequency difference of each word between the clicked document and the unclicked document, where j represents the number of each word in the word set, idf 1,j The first inverse document frequency (IDF) of the segment with ID j. 2, This represents the second inverse document frequency of the segment with ID j.

[0090] Step 304: Based on the frequency difference and the preset frequency difference threshold, the word segmentation set is divided into two categories: important word segmentation and ordinary word segmentation.

[0091] In this embodiment, the step of dividing the word segmentation set into two categories, important word segmentation and ordinary word segmentation, based on the frequency difference and a preset frequency difference threshold specifically includes: determining whether the frequency difference is greater than the preset frequency difference threshold; if the frequency difference is greater than the preset frequency difference threshold, obtaining the word segmentation corresponding to the frequency difference and caching the word segmentation to a pre-constructed important word segmentation set; if the frequency difference is not greater than the preset frequency difference threshold, obtaining the word segmentation corresponding to the frequency difference and caching the word segmentation to a pre-constructed ordinary word segmentation set.

[0092] By transforming the word segmentation process in search engines from a weighted scoring prediction problem into a classification problem, the word segmentation results in the search field are divided into two categories: important words and ordinary words. Through continuous adjustment of the word segmentation in these two categories, the search engine word segmentation model is improved. This avoids directly using deepCT to obtain the recall weights of words to get the target score for each word, and then fitting the score distribution of all words through modeling. This reduces the complexity of modeling and prevents the cumulative propagation of erroneous modeling results.

[0093] Step 205: Obtain the important class segmentation and ordinary class segmentation corresponding to each search field, and construct the model input triplet, wherein the format of the model input triplet is: [current search field: important class segmentation in the current search field: ordinary class segmentation in the current search field];

[0094] Step 206: Obtain the model input triplet corresponding to each search field as the model input value to train the initialized search engine word segmentation model, and obtain the trained search engine word segmentation model, thus completing the construction of the search engine word segmentation model.

[0095] Continue to refer to Figure 4 , Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 206 shown includes:

[0096] Step 401: Based on the cosine similarity algorithm and the model input triplet corresponding to each search field, obtain the similarity between each search field and its corresponding important class segmentation word, and the similarity between the important class segmentation word in each search field and each ordinary class segmentation word in each search field.

[0097] Step 402, according to the preset loss function: Obtain the loss value of each search field in the word segmentation model of the search engine, where exp(q·k) + / τ) represents the similarity between the current search field and its corresponding important word segmentation, exp(q·k) i / τ) represents the similarity between important word segments in the current search field and each ordinary word segment in the current search field, k i Let k represent the i-th ordinary class word segmentation. + τ represents the important class of word segmentation, k represents the number of words in the word segmentation set, and τ represents a preset hyperparameter used to control the model's ability to distinguish ordinary class words.

[0098] Step 403: Count the number of loss values ​​that satisfy the preset loss threshold;

[0099] Step 404: Determine whether the number of loss values ​​that satisfy a preset loss threshold is greater than a preset number threshold;

[0100] Step 405: If the number of loss values ​​that satisfy the preset loss threshold is not greater than the preset number threshold, then the word segments in the word segmentation set are re-divided by adjusting the frequency difference threshold to obtain the important class words and ordinary class words after re-division.

[0101] Step 406: Repeat steps 205 to 206 to build and update the search engine word segmentation model until the number of loss values ​​that meet the preset loss threshold is greater than the preset number threshold, at which point the search engine word segmentation model is completed.

[0102] By utilizing a loss function, a supervised contrastive learning approach is introduced to supervise the construction process of the search engine word segmentation model. The result of the loss function is used to determine whether the model has been successfully built, and the model is continuously optimized to ensure the accuracy of the final search engine word segmentation model, thereby making the search recall results more accurate.

[0103] This application acquires search data from historical users; constructs a word segmentation dictionary using the N-Gram algorithm and PMI word discovery mechanism; obtains a word segmentation set corresponding to each search field based on the word segmentation dictionary; divides the word segments in the word segmentation set into two categories: important word segments and ordinary word segments based on the IDF algorithm; constructs model input triples; and trains the initialized search engine word segmentation model using the model input triples as model input values ​​to obtain the trained search engine word segmentation model, thus completing the construction of the search engine word segmentation model. By employing N-Gram and PMI in the search field word segmentation stage, it can identify new words in the search field through the PMI new word discovery mechanism and filter out stop words in the search field through the N-Gram algorithm. Then, by combining document and IDF inverse document frequency calculation, it jointly predicts more accurate word labeling results. Finally, by introducing supervised comparative learning, it constructs the search engine word segmentation model, making the retrieved search results more accurate.

[0104] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0105] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0106] In this embodiment, N-Gram and PMI are used in the word segmentation stage of the search field. The PMI new word discovery mechanism can identify new words in the search field, and the N-Gram algorithm can filter out pause words in the search field. Then, by combining document and IDF inverse document frequency calculation, more accurate word labeling results are predicted. Finally, by introducing supervised contrastive learning, the word segmentation model of the search engine is constructed, making the search results recalled during the search more accurate.

[0107] Further reference Figure 5 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a search engine word segmentation model construction device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0108] like Figure 5 As shown, the search engine word segmentation model construction device 500 described in this embodiment includes: a search data acquisition module 501, a word segmentation dictionary construction module 502, a word segmentation set acquisition module 503, a word segmentation category division module 504, a triplet construction module 505, and a word segmentation model construction module 506.

[0109] in:

[0110] The search data acquisition module 501 is used to acquire search data of historical search users, wherein the search data includes historical search fields and search return logs;

[0111] The word segmentation dictionary construction module 502 is used to perform word segmentation prediction on each search field in the historical search fields using the N-Gram algorithm and PMI word discovery mechanism, obtain word segmentation prediction results, and construct a word segmentation dictionary based on the word segmentation prediction results corresponding to all historical search fields.

[0112] The word segmentation set acquisition module 503 is used to segment each search field in the historical search fields according to the word segmentation dictionary, and obtain the word segmentation set corresponding to each search field.

[0113] The word segmentation category division module 504 is used to divide the word segments in the word segmentation set into two categories: important word segments and ordinary word segments based on the search return log, the preset IDF algorithm and the preset classification threshold.

[0114] The triplet construction module 505 is used to obtain the important class segmentation and ordinary class segmentation corresponding to each search field, and construct the model input triplet, wherein the format of the model input triplet is: [current search field: important class segmentation in the current search field: ordinary class segmentation in the current search field];

[0115] The word segmentation model construction module 506 is used to obtain the model input triplet corresponding to each search field as the model input value to train the initialized search engine word segmentation model, obtain the trained search engine word segmentation model, and complete the construction of the search engine word segmentation model.

[0116] This application acquires search data from historical users; constructs a word segmentation dictionary using the N-Gram algorithm and PMI word discovery mechanism; obtains a word segmentation set corresponding to each search field based on the word segmentation dictionary; divides the word segments in the word segmentation set into two categories: important word segments and ordinary word segments based on the IDF algorithm; constructs model input triples; and trains the initialized search engine word segmentation model using the model input triples as model input values ​​to obtain the trained search engine word segmentation model, thus completing the construction of the search engine word segmentation model. By employing N-Gram and PMI in the search field word segmentation stage, it can identify new words in the search field through the PMI new word discovery mechanism and filter out stop words in the search field through the N-Gram algorithm. Then, by combining document and IDF inverse document frequency calculation, it jointly predicts more accurate word labeling results. Finally, by introducing supervised comparative learning, it constructs the search engine word segmentation model, making the retrieved search results more accurate.

[0117] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0118] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0119] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0120] The computer device 6 includes a memory 6a, a processor 6b, and a network interface 6c that are interconnected via a system bus. It should be noted that only the computer device 6 with components 6a-6c is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0121] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0122] The memory 6a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 6a may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 6a may also be an external storage device of the computer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 6. Of course, the memory 6a may also include both the internal storage unit and its external storage device of the computer device 6. In this embodiment, the memory 6a is typically used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions for building a search engine word segmentation model. In addition, the memory 6a can also be used to temporarily store various types of data that have been output or will be output.

[0123] In some embodiments, the processor 6b may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 6b is typically used to control the overall operation of the computer device 6. In this embodiment, the processor 6b is used to execute computer-readable instructions stored in the memory 6a or to process data, for example, to execute computer-readable instructions for the search engine word segmentation model construction method.

[0124] The network interface 6c may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 6 and other electronic devices.

[0125] The computer device proposed in this embodiment belongs to the field of word segmentation technology. This application acquires search data from historical search users; constructs a word segmentation dictionary using the N-Gram algorithm and the PMI word discovery mechanism; obtains the word segmentation set corresponding to each search field based on the word segmentation dictionary; divides the word segmentation in the word segmentation set into two categories: important word segmentation and ordinary word segmentation based on the IDF algorithm; constructs model input triples; and trains the initialized search engine word segmentation model using the model input triples as model input values ​​to obtain the trained search engine word segmentation model, thus completing the construction of the search engine word segmentation model. By using N-Gram and PMI in the search field word segmentation stage, it can identify new words in the search field through the PMI new word discovery mechanism and filter out pause words in the search field through the N-Gram algorithm. Then, by combining document and IDF inverse document frequency calculation, it jointly predicts more accurate word labeling results. Finally, by introducing supervised comparative learning, it constructs the search engine word segmentation model, making the retrieved search results more accurate.

[0126] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the search engine word segmentation model construction method described above.

[0127] The computer-readable storage medium proposed in this embodiment belongs to the field of word segmentation technology. This application acquires search data from historical search users; constructs a word segmentation dictionary using the N-Gram algorithm and the PMI word discovery mechanism; obtains the word segmentation set corresponding to each search field based on the word segmentation dictionary; divides the word segments in the word segmentation set into two categories—important word segments and ordinary word segments—based on the IDF algorithm; constructs model input triples; and trains the initialized search engine word segmentation model using the model input triples as model input values ​​to obtain the trained search engine word segmentation model, thus completing the construction of the search engine word segmentation model. By employing N-Gram and PMI in the search field word segmentation stage, it can identify new words in the search field through the PMI new word discovery mechanism and filter out pause words in the search field through the N-Gram algorithm. Then, by combining document and IDF inverse document frequency calculation methods, it jointly predicts more accurate word labeling results. Finally, by introducing supervised comparative learning, it constructs the search engine word segmentation model, making the retrieved search results more accurate.

[0128] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0129] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A method for constructing a word segmentation model for a search engine, characterized in that, Includes the following steps: Step 201: Obtain search data from historical search users, wherein the search data includes historical search fields and search return logs; Step 202: Use the N-Gram algorithm and PMI word discovery mechanism to perform word segmentation prediction on each search field in the historical search fields, obtain the word segmentation prediction results, and construct a word segmentation dictionary based on the word segmentation prediction results corresponding to all historical search fields; Step 203: Perform word segmentation on each search field in the historical search fields according to the word segmentation dictionary, and obtain the word segmentation set corresponding to each search field; Step 204: Based on the search return log, a preset IDF algorithm, and a preset classification threshold, the word segments in the word segmentation set are divided into two categories: important word segments and ordinary word segments. The search return log includes identifiers of clicked documents and identifiers of unclicked documents. Before performing the step of dividing the word segments in the word segmentation set into important and ordinary categories based on the search return log, the preset IDF algorithm, and the preset classification threshold, the method further includes: Based on the identification information of the clicked document and the identification information of the unclicked document, obtain the clicked document and the unclicked document. The step of dividing the word segments in the word segmentation set into two categories—important word segments and ordinary word segments—based on the search return logs, a preset IDF algorithm, and a preset classification threshold specifically includes: The inverse document frequency (IDF) of each word in the word segmentation set in the exposed clicked document is obtained by using the IDF algorithm and is denoted as the first inverse document frequency. The inverse document frequency (IDF) of each word in the word segmentation set in the exposed but unclicked document is obtained using the IDF algorithm and is denoted as the second inverse document frequency. Based on the preset absolute value algorithm, the first inverse document frequency and the second inverse document frequency, the frequency difference of each word segment between the exposed clicked document and the exposed unclicked document is obtained; Based on the frequency difference and the preset frequency difference threshold, the word segmentation set is divided into two categories: important word segmentation and ordinary word segmentation. Step 205: Obtain the important class segmentation and ordinary class segmentation corresponding to each search field, and construct the model input triplet, wherein the format of the model input triplet is: [current search field: important class segmentation in the current search field: ordinary class segmentation in the current search field]; Step 206: Obtain the model input triplet corresponding to each search field as the model input value to train the initialized search engine word segmentation model, thereby obtaining the trained search engine word segmentation model and completing the construction of the search engine word segmentation model. Specifically, this includes: Based on the cosine similarity algorithm and the model input triplet corresponding to each search field, the similarity between each search field and its corresponding important class segmentation and the similarity between the important class segmentation in each search field and each ordinary class segmentation in each search field are obtained. Based on the preset loss function: Obtain the loss value of each search field in the word segmentation model of the search engine, wherein, This indicates the similarity between the current search field and its corresponding important word segmentation. This indicates the similarity between important word segmentation in the current search field and each ordinary word segmentation in the current search field. Indicates the first Ordinary word segmentation, Indicates important word categories, This indicates the number of words in the word segmentation set. This represents the preset hyperparameters used to control the model's discriminative power for ordinary word segmentation; Count the number of loss values ​​that satisfy the preset loss threshold; Determine whether the number of loss values ​​that satisfy a preset loss threshold is greater than a preset number threshold; If the number of loss values ​​that satisfy the preset loss threshold is not greater than the preset number threshold, then the word segments in the word segmentation set are re-divided by adjusting the frequency difference threshold to obtain important class words and ordinary class words after re-division. Steps 205 to 206 are executed again to build and update the search engine word segmentation model until the number of loss values ​​that meet the preset loss threshold is greater than the preset number threshold, at which point the search engine word segmentation model is completed.

2. The method for constructing a search engine word segmentation model according to claim 1, characterized in that, The step of using the N-Gram algorithm and PMI word discovery mechanism to perform word segmentation prediction for each search field in the historical search fields specifically includes: The N-Gram algorithm is used to filter out the pause words in each search field of the historical search fields, and the pause words are then deleted. The PMI word discovery mechanism is used to filter out consecutively occurring words in each search field of the historical search fields, and it is determined whether the consecutive number of consecutive occurrences of the words is greater than a preset number threshold. If the number of consecutive occurrences exceeds a preset threshold, the consecutively occurring words are marked and treated as a word segmentation field. If the number of consecutive occurrences is not greater than a preset threshold, then the consecutively occurring words will be used as different word segmentation fields.

3. The method for constructing a search engine word segmentation model according to claim 1, characterized in that, The step of obtaining the inverse document frequency (IVF) of each word in the word segmentation set in the exposed clicked document using the IDF algorithm, denoted as the first IVF, specifically includes: According to the preset IDF algorithm formula: , respectively obtain the inverse document frequency of each word in the word segmentation set in the exposed clicked document, wherein This indicates the number of each word in the word segmentation set. This represents the total number of documents whose exposure was clicked and the total number of documents whose exposure was not clicked. This indicates the number of documents that were clicked upon exposure.

4. The method for constructing a search engine word segmentation model according to claim 1, characterized in that, The step of obtaining the inverse document frequency (IRF) of each word in the word segmentation set in the exposed but unclicked document using the IDF algorithm, denoted as the second IRF, specifically includes: According to the preset IDF algorithm formula: , respectively obtain the inverse document frequency of each word in the word segmentation set in the exposed unclicked document, wherein, This indicates the number of each word in the word segmentation set. This represents the total number of documents whose exposure was clicked and the total number of documents whose exposure was not clicked. This indicates the number of documents that were exposed but not clicked.

5. The method for constructing a search engine word segmentation model according to claim 1, characterized in that, The step of obtaining the frequency difference between the exposed clicked documents and the exposed unclicked documents based on a preset absolute value algorithm, the first inverse document frequency, and the second inverse document frequency specifically includes: According to the preset absolute value algorithm formula: The frequency difference of each word segment between the clicked documents and the unclicked documents is obtained. This indicates the number of each word in the word segmentation set. Indicates the number is The first inverse document frequency of the word segmentation, Indicates the number is The second inverse document frequency of the word segmentation.

6. The method for constructing a search engine word segmentation model according to claim 1, characterized in that, The step of dividing the word segmentation set into two categories, important word segmentation and ordinary word segmentation, based on the frequency difference and a preset frequency difference threshold, specifically includes: Determine whether the frequency difference is greater than a preset frequency difference threshold; If the frequency difference is greater than the preset frequency difference threshold, then the word segment corresponding to the frequency difference is obtained, and the word segment is cached in the pre-constructed important class word segmentation set; If the frequency difference is not greater than the preset frequency difference threshold, then the word segment corresponding to the frequency difference is obtained, and the word segment is cached in the pre-constructed ordinary word segmentation set.

7. A device for constructing a word segmentation model for a search engine, characterized in that, The search engine word segmentation model building device implements the steps of the search engine word segmentation model building method as described in any one of claims 1 to 6, and the search engine word segmentation model building device includes: The search data acquisition module is used to acquire search data of historical search users, wherein the search data includes historical search fields and search return logs; The word segmentation dictionary construction module is used to perform word segmentation prediction on each search field in the historical search fields using the N-Gram algorithm and PMI word discovery mechanism, obtain word segmentation prediction results, and construct a word segmentation dictionary based on the word segmentation prediction results corresponding to all historical search fields. The word segmentation set acquisition module is used to segment each search field in the historical search fields according to the word segmentation dictionary, and obtain the word segmentation set corresponding to each search field. The word segmentation category division module is used to divide the word segments in the word segmentation set into two categories: important word segments and ordinary word segments based on the search return log, the preset IDF algorithm, and the preset classification threshold. The triplet construction module is used to obtain the important class segmentation and ordinary class segmentation corresponding to each search field, and construct the model input triplet. The format of the model input triplet is: [current search field: important class segmentation in the current search field: ordinary class segmentation in the current search field]; The word segmentation model construction module is used to obtain the model input triplet corresponding to each search field as the model input value to train the initialized search engine word segmentation model, obtain the trained search engine word segmentation model, and complete the construction of the search engine word segmentation model.

8. A computer device comprising a memory and a processor, the memory storing computer-readable instructions, wherein the processor, when executing the computer-readable instructions, implements the steps of the search engine word segmentation model construction method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the search engine word segmentation model construction method as described in any one of claims 1 to 6.

Images