Model construction method, scoring method, search method, system, device, product and storage medium

Abstract

The embodiments of this specification relate to the field of artificial intelligence technology, and provide a model construction method, a scoring method, a search method, a system, a device, a product, and a storage medium. The model construction method includes: obtaining a training sample set; the training samples include: query word samples, recommended object texts including recommended object names and recommended object modifiers, and correct scoring texts representing the correlation scores between the query word samples and the recommended object texts; obtaining prompt content including query word samples, recommended object texts, and thought chain prompt texts; inputting the prompt content into a preset language model, so that the language model performs reasoning based on the thought chain, predicts a predicted scoring text of the correlation score between the query word sample and the recommended object text, and then trains based on the difference between the predicted scoring text and the correct scoring text. The language model constructed in this embodiment can output accurate correlation scores.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and particularly to a model construction method, a scoring method, a search method, a system, a device, a product, and a storage medium. Background Art

[0002] In modern information retrieval and recommendation systems, search relevance is a key factor in enhancing the user experience. Users expect to find the most relevant information or products in the shortest possible time. However, traditional search engines and recommendation systems often face great challenges when dealing with complex queries, understanding user intentions, and making relevance judgments by combining multi-source data. This results in low relevance and accuracy of search results, affecting user satisfaction and the competitiveness of the platform. Summary of the Invention

[0003] To overcome the problems existing in the related art, embodiments of this specification provide a model construction method, a scoring method, a search method, a system, a device, a product, and a storage medium.

[0004] According to the first aspect of the embodiments of this specification, a model construction method is provided for constructing a language model capable of performing relevance scoring. The method includes:

[0005] Obtain a training sample set; wherein, the training sample includes: a query word sample, a recommended object text including a recommended object name and a recommended object modifier, and a correct scoring text representing the relevance score between the query word sample and the recommended object text;

[0006] Obtain a prompt content including the query word sample, the recommended object text, and a thought chain prompt text; the thought chain prompt text is used to represent the following thought chain: identify whether the query word sample includes a query object name, whether the query word sample includes a query object modifier, whether the query object name matches the recommended object name, and whether the query object modifier matches the recommended object modifier, and determine the corresponding relevance score according to the identification result;

[0007] Input the prompt content into a preset language model, so that the language model performs reasoning based on the thought chain, predicts a predicted scoring text of the relevance score between the query word sample and the recommended object text, and then trains based on the difference between the predicted scoring text and the correct scoring text.

[0008] According to the second aspect of the embodiments of this specification, a scoring method is provided. The method is applied to a server, and the method includes:

[0009] Obtain the current query word and the recommended object text corresponding to the current query word;

[0010] Obtain a prompt content that includes the current query term, the recommended object text, and the thought chain prompt text; the thought chain prompt text is used to represent the following thought chain: identify whether the current query term includes a query object name, whether the current query term includes a query object modifier, whether the query object name matches the recommended object name, and whether the query object modifier matches the recommended object modifier, and determine a corresponding relevance score according to the recognition result;

[0011] Input the prompt content into a preset language model, so that the language model performs reasoning based on the thought chain and predicts a predicted score text of the relevance score between the query term sample and the recommended object text;

[0012] Wherein, the language model is constructed by using the steps of the method described in the first aspect.

[0013] According to the third aspect of the embodiments of the present specification, a search method is provided. The method is applied to a client, and the method includes:

[0014] Obtain the current query term input by the user;

[0015] After sending the current query term to the server, receive the sorting result sent by the server and display it;

[0016] Wherein, the server is used to recall each object text to be sorted corresponding to the current query term, generate a first relevance score text between the current query term and each object text to be sorted by using an online generation model, and then sort each object text to be sorted to obtain a sorting result of each object to be sorted; wherein, the online generation model is a language model constructed by using the method described in the first aspect.

[0017] According to the fourth aspect of the embodiments of the present specification, a search system is provided. The search system includes a server and a client. The server is used to execute the steps of the method described in the first aspect or the second aspect, and the client is used to execute the steps of the method described in the third aspect.

[0018] According to the fifth aspect of the embodiments of the present specification, a computer device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. Wherein, when the processor executes the computer program, it implements the steps of the method embodiments described in the foregoing first aspect, second aspect, or third aspect.

[0019] According to a sixth aspect of the embodiments of this specification, there is provided a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method embodiments described in the foregoing first aspect, second aspect, or third aspect are implemented.

[0020] According to a seventh aspect of the embodiments of this specification, there is provided a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the method embodiments described in the foregoing first aspect, second aspect, or third aspect are implemented.

[0021] The technical solutions provided by the embodiments of this specification may include the following beneficial effects:

[0022] In the embodiments of this specification, the prompt content includes a chain-of-thought prompt text, and the following chain of thought is represented in the chain-of-thought prompt text: identifying whether the query word sample contains a query object name, whether the query word sample contains a query object modifier, whether the query object name matches the recommended object name, and whether the query object modifier matches the recommended object modifier, and determining a corresponding relevance score according to the recognition result. Therefore, in this embodiment, a chain of thought is introduced into the model, which is equivalent to inserting a specific relevance score signal during the model inference process. Thus, when the model discriminates the relevance between the query word sample and the recommended object text, a control logic for how to discriminate the relevance score is provided for the model. The chain-of-thought prompt text is also equivalent to providing the model with self-reflection information, enabling the self-reflection mechanism to be directly incorporated and internalized into the large language model. Therefore, this embodiment can train a language model for accurate relevance scoring. In addition, the chain-of-thought prompt text provides accurate guidance and inference steps for the model to predict the relevance score, and can also improve the interpretability of the model, making the search experience interpretable and helping to understand the inference logic of the model.

[0023] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit this specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a schematic diagram of a search scenario shown according to an exemplary embodiment of this specification.

[0025] Figure 2A is a flowchart of a model construction method shown according to an exemplary embodiment of this specification.

[0026] Figure 2B is a schematic diagram of a chain of thought for relevance scoring shown according to an exemplary embodiment of this specification.

[0027] Figure 2CIt is a schematic diagram of a training framework shown in this specification according to an exemplary embodiment.

[0028] Figure 3A It is a flowchart of a scoring method shown in this specification according to an exemplary embodiment.

[0029] Figure 3B It is a flowchart of a search method shown in this specification according to an exemplary embodiment.

[0030] Figure 4 It is a hardware structure diagram of a computer device where a model building device / scoring device / search device is located shown in this specification according to an exemplary embodiment.

[0031] Figure 5 It is a block diagram of a model building device shown in this specification according to an exemplary embodiment.

[0032] Figure 6 It is a block diagram of a scoring device shown in this specification according to an exemplary embodiment.

[0033] Figure 7 It is a block diagram of a search device shown in this specification according to an exemplary embodiment. Detailed implementation manners

[0034] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all the implementation manners consistent with this specification. On the contrary, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0035] The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit this specification. The singular forms "a", "the", and "said" used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0036] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0037] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this specification are all information and data that have been authorized by the user or fully authorized by all parties. And the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entrances are provided for users to choose to authorize or refuse.

[0038] In scenarios such as search engines or recommendation systems, the server can obtain the user's current query term, and then recall a batch of objects to be sorted related to the current query term. The server can be provided with a sorting model, and the sorting model can predict the recommendation score between the information of the current query term and each object to be sorted based on the information of the current query term and the information of each object to be sorted. Among them, the degree of relevance between the current query term and the object to be sorted is one of the considerations for the sorting model to predict the recommendation score. Therefore, it is necessary to accurately determine the degree of relevance between the current query term and the object to be sorted. A classification neural network can be trained to identify the category to which the relevance between the current query term and the object to be sorted belongs.

[0039] Currently, generative large models can learn and generalize existing data to create new content and can also solve discriminative problems. Due to their huge number of parameters and deep network structures, generative large models can learn and understand more features and patterns, thus showing strong processing capabilities related to natural language understanding, intent recognition, reasoning, context modeling, language generation, etc. when dealing with complex tasks.

[0040] Therefore, the embodiments of this specification provide a model construction scheme to construct a language model that can perform relevance scoring, so as to combine generative AI (Artificial Intelligence) with relevance scoring, so that accurate relevance scores can be obtained in fields such as search or recommendation systems, thereby improving the relevance and accuracy of search results.

[0041] Among them, in this embodiment, the prompt content is designed to include the thought chain prompt text, and the following thought chain is represented in the thought chain prompt text: identifying whether the query word sample contains the query object name, whether the query word sample contains the query object modifier, whether the query object name matches the recommended object name, and whether the query object modifier matches the recommended object modifier, and determining the corresponding relevance score according to the recognition result. Therefore, this embodiment realizes introducing the thought chain into the model, which is equivalent to inserting a specific relevance scoring signal during the model inference process. Thus, when the model discriminates the relevance between the query word sample and the recommended object text, it provides the control logic for the model on how to discriminate the relevance score. The thought chain prompt text is also equivalent to providing the model with self-reflection information, enabling the self-reflection mechanism to be directly incorporated into the large language model. Therefore, this embodiment can train a language model capable of accurate relevance scoring. In addition, the thought chain prompt text provides the model with accurate guidance and reasoning steps for predicting the relevance score, which can also improve the interpretability of the model, making the search experience interpretable and helping to understand the reasoning logic of the model. Next, this embodiment will be described.

[0042] Please refer to Figure 1 , Figure 1 FIG. is a schematic diagram of a search scenario provided by an embodiment of this specification, which includes a server 100 and at least one client 110. Exemplarily, the client 110 can access the server 100 through a network to use the services provided by the server 100, including but not limited to search services or recommendation services, etc.; in the recommendation service, it can be the recommendation of any object, including entity commodity recommendation, virtual service recommendation, manual service recommendation, store recommendation, delivery service recommendation, or home service recommendation, etc.

[0043] The server 100 can be a program installed in a background device to provide services for users. Exemplarily, as Figure 1 shown, the background device can be a server, which can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0044] The client 110 can be a program installed in a user device to provide services for users. The client 110 includes but is not limited to application programs APP, Web pages, applets, plugins, or components, etc. As Figure 1As shown, the user equipment includes but is not limited to smart phones, personal digital assistants, tablet computers, personal computers, laptops, wearable devices, virtual reality terminal devices, augmented reality terminal devices, etc.

[0045] As Figure 2A shown, it is a flowchart of a model construction method provided by an embodiment of this specification, used to construct a language model capable of performing relevance scoring. The method may include the following steps:

[0046] In step 202, a training sample set is obtained.

[0047] Among them, the training sample includes: a query word sample, a recommended object text including a recommended object name and a recommended object modifier, and a correct scoring text representing the relevance score between the query word sample and the recommended object text.

[0048] In step 204, a prompt content including the query word sample, the recommended object text, and a thought chain prompt text is obtained.

[0049] The thought chain prompt text is used to represent the following thought chain: identifying whether the query word sample includes a query object name, whether the query word sample includes a query object modifier, whether the query object name matches the recommended object name, and whether the query object modifier matches the recommended object modifier, and determining the corresponding relevance score according to the identification result.

[0050] In step 206, the prompt content is input into a preset language model, so that the language model performs reasoning based on the thought chain, predicts a predicted scoring text of the relevance score between the query word sample and the recommended object text, and then is trained based on the difference between the predicted scoring text and the correct scoring text.

[0051] As an example, the method of this embodiment can be applied to Figure 1 the server in, and the constructed language model can be deployed and run in any computer device to perform relevance scoring.

[0052] As an example, this embodiment does not limit the specific recommended object. For example, in a trading platform, it can be an object recommended by the service platform side to the user, specifically it can be a commodity, a store, or any other custom object. In a search engine, it can be a search entry recommended by the server to the user.

[0053] In the fields of search and recommendation, the English terms "Query" and "doc" can be adopted. Query is the query term in this embodiment, directly expressing the information needs of users. The translation of "doc" is "document", which is the recommended object in this embodiment. For example, in a search engine, each doc contains information such as the website URL, title, and text summary of a web page. The search engine will sort and return results based on the relevance between Query and doc. In a recommendation system, "doc" can be analogous to "item", which is the specific object that the recommendation system needs to recommend to users.

[0054] As an example, at least two levels of relevance can be set, and different levels of relevance represent different degrees of high and low relevance. The relevance score corresponding to each relevance level can be set according to actual needs.

[0055] When a user needs to search, the user usually submits a query term Query. Query may or may not contain the name of the query object and its modifiers, while the recommended object text contains the object name and modifiers.

[0056] Among them, the object name refers to the main identifier of the recommended object; taking a commodity as an example, it is usually the name of the product; for example, in "coffee", "milk tea", or "ice cream", etc.

[0057] The modifier refers to the additional information describing the object, providing more detailed context, such as brand, size, flavor, or attribute, etc. For example, in "large cup of milk tea", "large cup" is the modifier.

[0058] As an example, taking three levels of relevance as an example, it can be:

[0059] 2 points indicate strong relevance; there can be two situations: when the modifier is not included in Query, the object name matches; when the modifier is included in Query, both the object name and the modifier match.

[0060] 1 point indicates weak relevance: the object name matches, but the modifier does not match.

[0061] 0 point indicates no relevance: the object name does not match.

[0062] As an example, as follows:

[0063] Suppose the recommended object text of a certain commodity is "Sweet Ice City large cup of milk tea", where "milk tea" is the object name and "Sweet Ice City large cup" is the modifier; then when the query is as follows, the relevance score is as follows:

[0064] query=“Kintetsu Plaza”: There is no object name or modifier, and it is a semantically irrelevant word. So what the product is does not affect the semantic relevance, and it can be counted as 2 points.

[0065] query=“奶茶”: The object name matches, and there are no other word constraints, which can be counted as 2 points.

[0066] query=“奶茶大杯”: the object name matches, and the modifiers also match, 2 points.

[0067] query=“奶茶小杯”: The object names are the same, but the modifiers do not match, so it is scored as 1 point.

[0068] query=“奶茶7分糖”: the object names are the same and the modifiers are irrelevant, so it can be counted as 2 points.

[0069] query=“Ice Cream Cup”: The object names are different, but the modifiers are the same, so the score is 0.

[0070] It is understandable that other correlation levels may be used in actual applications, and the specific score corresponding to each correlation level may be configured according to actual needs, which is not limited in this embodiment.

[0071] In related solutions, the model usually directly outputs the relevance score result, but it is not known how the model obtains this result. Based on this, the Chain of Thought (CoT) is introduced into the model in this embodiment. The Chain of Thought is a reasoning method that gradually decomposes the problem and gradually derives the solution. It emphasizes each step of the logical chain to ensure that each step is clear and unambiguous, thereby improving the ability to solve complex problems. Therefore, the model that introduces the chain of thought can provide the model with accurate guidance and reasoning steps for predicting relevance scores, and can also improve the interpretability of the model, so that the search experience can be explained. The introduction of the chain of thought allows you to see how the model reasoned at each step, which helps to understand the reasoning logic of the model.

[0072] In this embodiment, based on the aforementioned identification method of the correlation score between the query and the doc, a corresponding thought chain and a thought chain prompt text can be designed;

[0073] The chain of thought can be as follows: identify whether the query word sample contains the name of the query object, whether the query word sample contains the modifier of the query object, whether the name of the query object matches the name of the recommended object, and whether the modifier of the query object matches the modifier of the recommended object, and determine the corresponding relevance score according to the recognition results. Among them, the chain of thought contains 4 recognition operations. The order of the recognition operations can be set according to actual needs, as long as the recognition results can be mapped to the corresponding relevance scores. Different recognition results can correspond to different relevance scores, and the relevance scores corresponding to each recognition result can be set according to needs. This embodiment does not limit this.

[0074] In some examples, the chain of thought, as Figure 2B shown, is a schematic diagram of the chain of thought for a relevance score shown in this specification according to an exemplary embodiment, and may include the following logical path for determining the relevance score:

[0075] First, identify whether the query word sample contains the name of the query object and the modifier of the query object. If it does not contain the name of the query object, a strong relevance score is obtained; Figure 2B Taking 2 points as an example for the medium-strong relevance score;

[0076] If it contains the name of the query object, first identify whether the name of the query object matches the name of the recommended object. If it does not match, an irrelevance score is obtained; Figure 2B Taking 0 points as an example for the medium-strong relevance score;

[0077] If it matches, and in the case of determining that the query word sample does not contain the modifier of the query object, the strong relevance score is obtained; Figure 2B Taking 2 points as an example for the medium-strong relevance score;

[0078] In the case of determining that it contains the modifier of the query object, identify whether the modifier of the query object matches the modifier of the recommended object; if it matches, the strong relevance score is obtained, and if it does not match, a weak relevance score is obtained; Figure 2B Taking 1 point as an example for the weak relevance score.

[0079] As an example, the chain of thought text contained in the prompt content input to the model can be the above text content, or text related to the above text content, as long as it reflects the recognition logic of the above chain of thought.

[0080] In short, the following are several types of recognition result types:

[0081] No result: If the object name is not found in the recognized Query, directly give 2 points (representing a strong relevance score).

[0082] Object name mismatch: If the object names do not match, give 0 points (representing an irrelevant score).

[0083] The object names match but there are no modifiers in the Query, give 2 points.

[0084] The object names match but the modifiers do not match: If the object names match but the modifiers do not match, give 1 point (representing a weakly relevant score).

[0085] Both the object name and the modifiers match: If both the object name and the modifiers match, give 2 points.

[0086] In the above embodiments, three levels of strong relevance, weak relevance, and irrelevance are adopted. Among them, the strong relevance score includes three different situations, and all three different situations are represented by "2 points". Optionally, different levels can also be used to distinguish these three different situations, that is, three different values can be used to indicate these three different situations, or in practical applications, it can be set according to needs, and this embodiment does not limit this.

[0087] Therefore, the above chain of thought gives how to finally give a relevance score by gradually checking the degree of relevance between the Query and the doc, which is used to measure the relevance between the Query and the doc. In this way, through the above chain of thought, this embodiment provides a discriminant logic for the relevance score for the model, enabling the model to accurately generate the relevance score.

[0088] In actual implementation, it can also be considered to directly use an existing pre-trained large language model through a Prompt containing the chain of thought to score the relevance between the Query and the doc. However, in the pre-trained corpus of the large language model, there is no corpus for the relevance score between the Query and the doc. Without training the large language model, the accuracy rate is at most 90%; in the scenarios where the relevance score between the Query and the doc is required, in order to ensure the accuracy rate and the user search experience, the accuracy requirement for the relevance score needs to be 95%+ or even 98%+.

[0089] Based on this, a language model capable of performing relevance scoring needs to be trained separately. The existing base language model can be used (of course, it is also optional to build it by oneself, and this embodiment does not limit this), and training samples are constructed for fine-tuning. In this embodiment, the prompt content input to the language model includes the chain of thought prompt text, enabling the language model to implement the self-reflection mechanism.

[0090] Self-Reflection is a cognitive process through which individuals review and analyze their thoughts, behaviors, and emotions to enhance self-awareness and understanding. This process involves assessing one's strengths and weaknesses, identifying behavioral patterns, understanding motivations and emotional responses, thereby promoting personal growth and development. Self-reflection helps to strengthen self-awareness, improve decision-making abilities, enhance emotional management levels, and achieve better performance in personal and professional life. Through regular self-reflection, individuals can continuously adjust and optimize their behaviors and thinking patterns to achieve a higher level of self-actualization. Existing large language models do not have the function of "self-reflection" by themselves and are not easy to transplant.

[0091] In this embodiment, a self-reflection mechanism is designed to be directly incorporated into the large language model. Specifically, when the large language model is reasoning, it generates the next token one by one in an autoregressive (AR) manner. Specifically, the model contains multiple layers of encoders. The prompt content input into the model will be encoded into an initial token sequence and then input into the multiple layers of encoders. Each layer of encoder will process the input token sequence. Finally, the last layer of encoder in the multiple layers of encoders will output the first response token; then, the first response token will be added to the initial token sequence to form a new token sequence, which will be input into the multiple layers of encoders again for processing. The last layer of encoder will output the second response token, and the second response token will continue to be added to the token sequence and then input into the multiple layers of encoders again, and so on. The above process continues to loop until the model can obtain the response content corresponding to the prompt content.

[0092] The prompt content in this embodiment contains the thought chain prompt text, which is equivalent to inserting a specific token signal during the model reasoning process. In each reasoning of the model, it can capture the signal provided by the thought chain prompt text. Therefore, the thought chain prompt text in this embodiment can be called a self-reflection token, which provides the following information for the model:

[0093] When the model discriminates the relevance between the query and the doc, it provides branch control for the discrimination because the discrimination of the relevance between the query and the doc in this embodiment is a hierarchical and complex judgment logic;

[0094] For the relevance score itself, it provides specific scores for the relevance score, such as the three enumerated values 0, 1, and 2 in the aforementioned example.

[0095] As an example, the prompt content input into the language model can include the following parts:

[0096] ① Role information that instructs the model to reply as an expert in relevance judgment.

[0097] For example, it can be text such as "Please reply as an expert in relevance judgment" and similar content. In actual applications, it can be flexibly adjusted according to needs, and this embodiment does not limit it. In this way, the model can be instructed about the role it needs to play, activating the model's professional capabilities in the corresponding field.

[0098] ② Task information that instructs the task the model needs to complete.

[0099] This task information can include Query and doc. For example, it can be "Please determine the relevance between Query and doc" and similar content.

[0100] ③ Chain-of-thought prompt text;

[0101] As an example, this chain-of-thought prompt text can be text such as the following:

[0102] "Please use the following reasoning chain to identify the relevance score between Query and doc:

[0103] First, identify whether Query contains the query object name and query object modifier. If it does not contain the query object name, obtain a strong relevance score, and the value of the strong relevance score is 2;

[0104] If it contains the query object name, first identify whether the query object name matches the recommended object name. If it does not match, obtain an irrelevance score, and the value of the irrelevance score is 0;

[0105] If it matches, and in the case where Query does not contain the query object modifier, obtain a strong relevance score, and the value of the strong relevance score is 2;

[0106] In the case where it is determined to contain the query object modifier, identify whether the query object modifier matches the recommended object modifier; if it matches, obtain a strong relevance score, and if it does not match, obtain a weak relevance score, and the value of the weak relevance score is 1".

[0107] ④ Example information; one or more examples can be provided to the model for reference.

[0108] Optionally, in actual applications, the prompt content can also include other additional information, which is not limited in this embodiment. For example, it can be required that the model's reply includes the process of its reasoning chain, or the format of the score text between Query and doc, etc.

[0109] Based on the foregoing design of the relevance score between Query and doc, this embodiment also designs the corresponding types of scoring texts:

[0110] Scoring texts containing the following information: information indicating that the query term sample does not contain the name of the query object, and a strong relevance score indicating the relationship between the query term sample and the recommended object text;

[0111] Scoring texts containing the following information: information indicating that the query term sample contains the name of the query object, information indicating that the name of the query object does not match the name of the recommended object, and a non - relevance score indicating the relationship between the query term sample and the recommended object text;

[0112] Scoring texts containing the following information: information indicating that the query term sample contains the name of the query object, information indicating that the name of the query object matches the name of the recommended object, and a strong relevance score indicating the relationship between the query term sample and the recommended object text;

[0113] Scoring texts containing the following information: information indicating that the query term sample contains the name of the query object, information indicating that the name of the query object matches the name of the recommended object, information indicating that the query term sample contains a modifier of the query object, information indicating that the modifier of the query object matches the modifier of the recommended object, and a strong relevance score indicating the relationship between the query term sample and the recommended object text;

[0114] Scoring texts containing the following information: information indicating that the query term sample contains the name of the query object, information indicating that the name of the query object matches the name of the recommended object, information indicating that the query term sample contains a modifier of the query object, information indicating that the modifier of the query object does not match the modifier of the recommended object, and a weak relevance score indicating the relationship between the query term sample and the recommended object text.

[0115] As an example, the correct scoring texts included in the training samples adopt the above design, and the predicted scoring texts output by the model can also adopt the above design.

[0116] The structure of the above - mentioned scoring text can be understood as: "[Token = ENUM][[Score = VALUE]"; where

[0117]

[0118] As an example, taking the recommended object text as "Sweet Ice City Milk Tea, large cup" as an example, the training samples can be the following types:

[0119] ① The query is "Kintetsu Plaza", and the score is 2 points at this time

[0120] The training sample can be: "Kintetsu Square ## Sweet Ice City Milk Tea in large cup [isUK=YES][Score=0]"

[0121] ② The query is "ice cream in large cup": At this time, the score is 0

[0122] The training sample can be:

[0123] "Ice cream in large cup ## Sweet Ice City Milk Tea in large cup [isUK=NO][Score=][isSrel=NO][Score=0]"

[0124] ③ The query is "milk tea": At this time, the score is 2

[0125] The training sample can be:

[0126] "Milk tea ## Sweet Ice City Milk Tea in large cup [isUK=NO][Score=][isSrel=YES][Score=2]"

[0127] ④ The query is "milk tea in small cup": At this time, the score is 1

[0128] The training sample can be:

[0129] "Milk tea in small cup ## Sweet Ice City Milk Tea in large cup [isUK=NO][Score=][isSrel=YES][Score=][isMrel=NO]

[0130] [Score=1]"

[0131] ⑤ The query is "milk tea in large cup": At this time, the score is 2

[0132] The training sample can be:

[0133] "Milk tea in large cup ## Sweet Ice City Milk Tea in large cup [isUK=NO][Score=][isSrel=YES][Score=]

[0134] [isMrel=YES][Score=2]"

[0135] In the above training samples, the character "##" is used as the splicing character between the query and the doc. In actual applications, other characters can also be used, and this embodiment does not limit this.

[0136] In the above training samples, the text after the doc is the scoring text; for example, "[isUK=YES][Score=0]" or "[isUK=NO][Score=][isSrel=NO][Score=0]" etc. are the scoring texts.

[0137] It can be seen that the scoring text contains both the recognition steps in the chain of thought and the specific values of the relevance score.

[0138] Optionally, the model can generate a predicted scoring text based on the input prompt content, and the predicted scoring text can also adopt the format of the above-mentioned scoring text.

[0139] In this embodiment, after the model predicts the predicted scoring text of the relevance score between the query word sample and the recommended object text, the model can be trained based on the difference between the predicted scoring text and the correct scoring text. In practical applications, the loss function for model training can be designed as needed, and this embodiment does not limit it.

[0140] In practical applications, considering the different requirements for inference response speed in online and offline scenarios, this embodiment can also design and train two models, thus effectively taking into account the different requirements of model size, inference quality, online scenarios or offline scenarios.

[0141] In some examples, the method is used to first construct an offline evaluation model and then an online generation model;

[0142] The model parameters of the offline evaluation model are greater than those of the online generation model;

[0143] When constructing the offline evaluation model, the training sample set is: the real training sample set;

[0144] When constructing the online generation model, the training sample set is: the high-quality training sample set;

[0145] Among them, the number of samples in the real training sample set is less than that in the high-quality training sample set;

[0146] The high-quality training sample set is selected by using the prediction results of the offline evaluation model for the synthetic data in the preset synthetic dataset.

[0147] As an example, the training and inference of the offline evaluation model (Critic model) are carried out in the offline scenario, and a large large language model can be selected as needed to train a model with high inference quality; while the online evaluation model is used in online services and has a time-consuming requirement, and a model smaller than the offline evaluation model can be adopted as the online generation model (Generate model). In practical applications, existing language models can be selected as the basis for these two models as needed, and this embodiment does not limit it.

[0148] In practical applications, data can be pre-constructed in various ways.

[0149] For example, it is constructed based on the user's historical search behavior and historical click behavior. For example, in the user's historical search behavior, there are the user's historical queries. The service provider will display multiple historical recommended objects to the user based on the historical queries. These multiple historical recommended objects include the historical click objects and historical unclicked objects clicked by the user. The click objects can be set as docs with a strong relevance score. Optionally, the historical unclicked objects can be set as docs with an irrelevance score or a weak relevance score. Optionally, it is also optional to construct based on the user's historical orders.

[0150] As needed, it can also be constructed manually, or manually screen or label the automatically constructed data to obtain a batch of accurate and high-quality training sample sets, which are called real training sample sets in this embodiment. It can be seen that due to the labor cost, the number of real training sample sets is limited, and the real training sample sets are used to train the offline evaluation model.

[0151] After training, since the offline evaluation model is a large-scale model and is fine-tuned by the real training sample set, the offline evaluation model can have better inference quality.

[0152] The online evaluation model has a smaller model scale and requires more training samples. To reduce the labor cost, some sample generation techniques can be used to obtain a synthetic data set. Among them, the synthetic data set is generated by an automated technique, and its quality may not be high. Therefore, in this embodiment, it is designed that the offline evaluation model predicts the synthetic data in the synthetic data set, so as to screen out a high-quality training sample set from the synthetic data set. Thus, the high-quality training sample set can meet the data volume requirement for the training of the online evaluation model.

[0153] It can be seen that the training processes of the offline evaluation model and the online evaluation model can both be implemented by Figure 2A the embodiments shown, except that the training sample sets used by the two models can be different.

[0154] As an example, the synthetic data in the synthetic data set can be generated by prompting the target language model to rewrite the real training samples in the real training sample set.

[0155] The high-quality training samples in the high-quality training sample set can be obtained in the following way:

[0156] For each synthetic data, use the offline evaluation model to predict the predicted score text between the query word sample included in the synthetic data and the recommended object text. If the predicted score text matches the rewritten score text included in the synthetic data, use this synthetic data as a high-quality training sample.

[0157] In this embodiment, an existing language model can be used for generation, which is referred to as the target language model in this embodiment to distinguish it from the aforementioned other language models. Among them, the target language model can be the same as or different from the language model used to construct the offline evaluation model, and this embodiment does not limit this. The target language model can be prompted to automatically generate synthetic data, thereby reducing labor costs and also solving the problem of a small amount of real data.

[0158] As an example, it can be generated based on real training samples, and the real training samples include query word samples, recommended object texts, and correct scoring texts. A Prompt for prompting the target language model can be designed, for example, to guide the target language model to generate synthetic data that meets the requirements. Among them, the synthetic data also includes: query words, recommended object texts, and scoring texts.

[0159] Optionally, the Prompt for prompting the target language model can also include a chain of thought prompting text, so that in the synthetic data generated by the model based on the Prompt, the included scoring text can accurately represent the correlation score between the query word and the recommended object text in the synthetic data, so as to obtain high-quality synthetic data as much as possible.

[0160] For example, the Prompt can include the following content:

[0161] "You are a content generation expert and need to generate new training samples based on the real training samples I provide;

[0162] The composition of the training samples is as follows: query words, recommended object texts, and scoring texts; among them, the scoring text is used to represent the correlation score between the query word and the recommended object text;

[0163] The correlation score between the query word and the recommended object text can be determined by the following chain of thought; the specific chain of thought is *** (the asterisk here can refer to the embodiment of the chain of thought text described above and will not be expanded here), so as to generate an accurate correlation score;

[0164] Examples of real training samples are as follows: *** (the asterisk here represents one or more real training samples)".

[0165] After the online generation model is constructed in this embodiment, the online generation model can be applied to online inference. For example, in some cases, after the online generation model is constructed, the method may further include:

[0166] Use the online generation model to predict the correlation score between the current query term input by the user and the text of the object to be sorted, so as to obtain the first correlation score text output by the online generation model; wherein, the text of the object to be sorted is recalled based on the current query term.

[0167] For example, in this embodiment, a prompt content including the current query term, the text of the object to be sorted, and the thought chain prompt text can be constructed, so that the online generation model can output the first correlation score text between the current query term and the text of the object to be sorted. Therefore, in this embodiment, the online generation model can be used to give an accurate first correlation score text in real time during online inference.

[0168] In some examples, in a search engine or a recommendation system, there may be multiple texts of objects to be sorted, and the method may further include:

[0169] Sort the texts of the objects to be sorted according to the first correlation score text between the current query term and each text of the object to be sorted, and obtain the sorting results of each object to be sorted;

[0170] Recommend the sorting results to the user.

[0171] As an example, this embodiment can be applied to any search engine or recommendation system. It can obtain the current query term of the user, then recall a batch of objects to be sorted related to the current query term, and use the online generation model of this embodiment to obtain the first correlation score text between the current query term and each text of the object to be sorted. Based on this, an accurate sorting result of each text of the object to be sorted can be obtained, thereby improving the user experience.

[0172] In some examples, after the online evaluation model is applied, this embodiment can also use the offline evaluation model to supervise and diagnose the online evaluation model. For example, the method may further include:

[0173] Use the offline evaluation model to predict the correlation score between the current query term input by the user and the text of the object to be sorted, so as to obtain the second correlation score text output by the offline evaluation model;

[0174] Identify whether the first correlation score text is the same as the second correlation score text. If they are not the same, construct a new training sample using the current query term, the text of the object to be sorted, and the second correlation score text; the constructed new training sample is used to continue training the online generation model.

[0175] Optionally, the offline evaluation model can also predict the relevance score between the currently entered query term by the user and the text of the object to be sorted; among them, the prediction of the offline evaluation model and the prediction of the online generation model can be parallel, and the prediction of the offline evaluation model does not affect the prediction of the online generation model in the link of the search engine or recommendation system.

[0176] If the prediction result of the online generation model is different from the prediction result of the offline evaluation model, the prediction result of the offline evaluation model can be used as the standard, and a new sample for training the online generation model can be continuously constructed and the online generation model can be continuously trained.

[0177] Optionally, the process of the above offline evaluation model for supervised diagnosis can be continuously executed or intermittently executed. For example, the offline evaluation model can be started according to a set period to perform parallel prediction. In practical applications, it can be configured according to needs, and this embodiment does not limit it.

[0178] As Figure 2C shown, it is a schematic diagram of a training framework shown in this specification according to an exemplary embodiment, which shows the following training process:

[0179] Step 1: Construct a synthetic data set;

[0180] For example, data can be constructed through the positive and negative samples of the user's historical clicks.

[0181] Training samples can also be obtained through manual annotation to ensure the accuracy and quality of the data.

[0182] The COT (Chain of Thought) prompting technique can be used to generate prompt information. And a large language model is used to generate synthetic data based on the prompt information.

[0183] Step 2: Construct a Critic model;

[0184] An existing large-scale high-quality language model can be selected as the base, and the Critic model can be trained using the manually annotated real training sample set. After that, the Critic model can be used to predict the synthetic data set, so as to construct a relatively large number of high-quality training sample sets.

[0185] Step 3: Construct a Generate model;

[0186] A small-scale model that can meet the requirements of online inference speed can be used, and it is trained using the high-quality training sample set to obtain the Generate model.

[0187] During the online inference process, the Critic model can intermittently supervise the diagnosis to evaluate the relevance quality of the online Generate model. From this perspective, in the training framework designed in this embodiment, the Critic model and the Generate model form a "twin star" mode, which is also similar to the distillation learning in the teacher-student model.

[0188] As Figure 3A shown, it is a flowchart of a scoring method shown in this specification according to an exemplary embodiment. This embodiment can be applied to the server, and the method may include:

[0189] In step 302, obtain the current query term and the recommended object text corresponding to the current query term;

[0190] In step 304, obtain the prompt content including the current query term, the recommended object text, and the thought chain prompt text; the thought chain prompt text is used to represent the following thought chain: identify whether the current query term contains the query object name, whether the current query term contains the query object modifier, whether the query object name matches the recommended object name, and whether the query object modifier matches the recommended object modifier, and determine the corresponding relevance score according to the identification results;

[0191] In step 306, input the prompt content into a preset language model, so that the language model performs reasoning based on the thought chain and predicts the predicted score text of the relevance score between the query term sample and the recommended object text;

[0192] Among them, the language model is constructed by the steps of the foregoing model construction method embodiment.

[0193] The specific implementation process of this embodiment can refer to the foregoing embodiment and will not be elaborated here.

[0194] As Figure 3B shown, it is a flowchart of a search method shown in this specification according to an exemplary embodiment. This embodiment can be applied to the client, and the method includes:

[0195] In step 312, obtain the current query term input by the user.

[0196] In step 314, after sending the current query term to the server, receive the sorting result sent by the server and display it.

[0197] Among them, the server is used to recall each to-be-sorted object text corresponding to the current query term, and then use an online generation model to generate a first relevance scoring text between the current query term and each to-be-sorted object text, and then sort each to-be-sorted object text to obtain the sorting result of each to-be-sorted object; among them, the online generation model is a language model constructed by using the foregoing model construction method embodiments.

[0198] Corresponding to the embodiments of the foregoing model construction method / scoring method / search method, this specification also provides embodiments of a model construction device / a scoring device / a search device and a computer device to which they are applied.

[0199] The embodiments of the model construction device / scoring device / search device in this specification can be applied to a computer device, such as a server or a terminal device. The device embodiments can be implemented by software, or by hardware, or by a combination of software and hardware. Taking software implementation as an example, as a logically meaningful device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory by its corresponding processor and running. From the hardware level, as Figure 4 shown, it is a hardware structure diagram of the computer device where the model construction device / scoring device / search device in this specification is located. In addition to Figure 4 the shown processor 410, network interface 420, memory 430, and non-volatile memory 440, the computer device where the model construction device / scoring device / search device in the embodiment is located usually also includes other hardware according to the actual functions of the computer device, which will not be elaborated here.

[0200] As Figure 5 shown, Figure 5 is a block diagram of a model construction device shown according to an exemplary embodiment of this specification. The device is used to construct a language model capable of performing relevance scoring. The device includes:

[0201] A sample acquisition module 51, configured to: acquire a training sample set; where the training sample includes: a query term sample, a recommended object text including a recommended object name and a recommended object modifier, and a correct scoring text characterizing the relevance score between the query term sample and the recommended object text;

[0202] Content acquisition module 52, configured to: acquire prompt content including the query word sample, the recommended object text, and the thought chain prompt text; the thought chain prompt text is used to represent the following thought chain: identify whether the query word sample includes the query object name, whether the query word sample includes the query object modifier, whether the query object name matches the recommended object name, and whether the query object modifier matches the recommended object modifier, and determine the corresponding relevance score according to the identification result;

[0203] Training module 53, configured to: input the prompt content into a preset language model, so that the language model performs reasoning based on the thought chain, and after predicting a predicted score text of the relevance score between the query word sample and the recommended object text, perform training based on the difference between the predicted score text and the correct score text.

[0204] In some examples, the thought chain specifically includes:

[0205] First, identify whether the query word sample includes the query object name and the query object modifier. If the query object name is not included, obtain a strong relevance score;

[0206] If the query object name is included, first identify whether the query object name matches the recommended object name. If they do not match, obtain an irrelevance score;

[0207] If they match, and it is determined that the query word sample does not include the query object modifier, then obtain the strong relevance score;

[0208] In the case of determining that the query object modifier is included, identify whether the query object modifier matches the recommended object modifier; if they match, obtain the strong relevance score, if they do not match, obtain a weak relevance score.

[0209] In some examples, the correct score text has the following types:

[0210] The correct score text including the following information: information indicating that the query word sample does not include the query object name, and a strong relevance score indicating the relevance between the query word sample and the recommended object text;

[0211] The correct score text including the following information: information indicating that the query word sample includes the query object name, information indicating that the query object name does not match the recommended object name, and an irrelevance score indicating the relevance between the query word sample and the recommended object text;

[0212] Correct scoring text containing the following information: information indicating that the query term sample contains the name of the query object, information indicating that the name of the query object matches the name of the recommended object, and information indicating a strong correlation score between the query term sample and the recommended object text;

[0213] Correct scoring text containing the following information: information indicating that the query term sample contains the name of the query object, information indicating that the name of the query object matches the name of the recommended object, information indicating that the query term sample contains a modifier of the query object, information indicating that the modifier of the query object matches the modifier of the recommended object, and information indicating a strong correlation score between the query term sample and the recommended object text;

[0214] Correct scoring text containing the following information: information indicating that the query term sample contains the name of the query object, information indicating that the name of the query object matches the name of the recommended object, information indicating that the query term sample contains a modifier of the query object, information indicating that the modifier of the query object does not match the modifier of the recommended object, and information indicating a weak correlation score between the query term sample and the recommended object text.

[0215] In some examples, the device is used to first construct an offline evaluation model and then used to construct an online generation model;

[0216] The model parameters of the offline evaluation model are greater than the model parameters of the online generation model;

[0217] When constructing the offline evaluation model, the training sample set is: a real training sample set;

[0218] When constructing the online generation model, the training sample set is: a high-quality training sample set;

[0219] Wherein, the number of samples in the real training sample set is less than the number of samples in the high-quality training sample set;

[0220] The high-quality training sample set is selected based on the prediction results of the synthetic data in the preset synthetic dataset by the offline evaluation model.

[0221] In some examples, the synthetic data in the synthetic dataset is generated by prompting the target language model to rewrite the real training samples in the real training sample set;

[0222] The high-quality training samples in the high-quality training sample set are obtained through the following method:

[0223] For each piece of synthetic data, use the offline evaluation model to predict the predicted rating text between the query word sample included in the synthetic data and the recommended object text. If the predicted rating text matches the rewritten rating text included in the synthetic data, use this synthetic data as a high-quality training sample.

[0224] In some examples, the device further includes an online generation module, configured to: after constructing the online generation model, use the online generation model to predict the correlation score between the current query word input by the user and the text of the object to be sorted, so as to obtain the first correlation score text output by the online generation model; wherein, the text of the object to be sorted is recalled based on the current query word.

[0225] In some examples, there are multiple texts of the objects to be sorted, and the online generation module is further configured to:

[0226] Sort the texts of the objects to be sorted according to the first correlation score text between the current query word and each text of the object to be sorted, to obtain the sorting result of each object to be sorted;

[0227] Recommend the sorting result to the user.

[0228] In some examples, the training module 53 is further configured to:

[0229] Use the offline evaluation model to predict the correlation score between the correlation score between the current query word input by the user and the text of the object to be sorted, so as to obtain the second correlation score text output by the offline evaluation model;

[0230] Identify whether the first correlation score text is the same as the second correlation score text. If they are not the same, construct a new training sample using the current query word, the text of the object to be sorted, and the second correlation score text; the constructed new training sample is used to continue training the online generation model.

[0231] As Figure 6 shown, Figure 6 is a block diagram of a scoring device shown in accordance with an exemplary embodiment of the present specification. The device is applied to a client, and the device includes:

[0232] A first acquisition module 61, configured to: acquire a current query word and a recommended object text corresponding to the current query word;

[0233] A second acquisition module 62, configured to: acquire a prompt content including the current query term, the recommended object text, and a thought chain prompt text; the thought chain prompt text is used to represent the following thought chain: identify whether the current query term includes a query object name, whether the current query term includes a query object modifier, whether the query object name matches the recommended object name, and whether the query object modifier matches the recommended object modifier, and determine a corresponding relevance score according to the identification result;

[0234] A scoring module 63, configured to: input the prompt content into a preset language model, so that the language model performs reasoning based on the thought chain and predicts a predicted scoring text of the relevance score between the query term sample and the recommended object text;

[0235] Wherein, the language model is constructed by the steps of the foregoing model construction method embodiment.

[0236] The specific implementation process of this embodiment can refer to the foregoing embodiment and will not be elaborated herein.

[0237] As Figure 7 shown, Figure 7 is a block diagram of a search device shown in accordance with an exemplary embodiment of the present specification. The device is applied to a client and includes:

[0238] An acquisition module 71, configured to: acquire a current query term input by a user;

[0239] A display module 72, configured to: after sending the current query term to a server, receive a sorting result sent by the server and display it;

[0240] Wherein, the server is configured to recall each text of the objects to be sorted corresponding to the current query term, generate a first relevance scoring text between the current query term and each text of the objects to be sorted by using an online generation model, and then sort each text of the objects to be sorted to obtain a sorting result of each object to be sorted; wherein, the online generation model is a language model constructed by the foregoing model construction method embodiment.

[0241] The implementation processes of the functions and roles of each module in the foregoing model construction device / scoring device / search device are specifically described in detail in the corresponding steps of the foregoing model construction method / scoring method / search method and will not be elaborated herein.

[0242] Correspondingly, an embodiment of the present specification further provides a search system, which includes a server and a client. The server is configured to execute the steps of the foregoing model construction method / scoring method embodiment, and the client is configured to execute the steps of the foregoing search method embodiment.

[0243] Correspondingly, an embodiment of this specification further provides a computer program product, including a computer program, where when the computer program is executed by a processor, the steps of the foregoing method embodiments of the model building method / score calculation method / search method are implemented.

[0244] Correspondingly, an embodiment of this specification further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the program, the steps of the method embodiments of the model building method / score calculation method / search method are implemented.

[0245] Correspondingly, an embodiment of this specification further provides a computer-readable storage medium, on which a computer program is stored, where when the computer program is executed by a processor, the steps of the method embodiments of the model building method / score calculation method / search method are implemented.

[0246] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial descriptions of the method embodiments. The device embodiments described above are only illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this specification. A person of ordinary skill in the art can understand and implement it without creative efforts.

[0247] The foregoing embodiments can be applied to one or more computer devices. The computer device is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. The hardware of the computer device includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0248] The computer device can be any electronic product capable of performing human-computer interaction with a user. For example, a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an Internet protocol television (IPTV), a smart wearable device, etc.

[0249] The computer device may further include a network device and / or a user device. Among them, the network device includes, but is not limited to, a single network server, a server group composed of multiple network servers, or a cloud composed of a large number of hosts or network servers based on cloud computing (CloudComputing).

[0250] The network where the computer device is located includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, virtual private network (Virtual Private Network, VPN), etc.

[0251] The specific embodiments of this specification have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures do not necessarily require the particular order or sequential order shown to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0252] The step division of the above various methods is only for clear description. When implemented, they can be combined into one step or some steps can be split into multiple steps. As long as the same logical relationship is included, they are all within the protection scope of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs, but not changing the core design of its algorithm and process, are all within the protection scope of this application.

[0253] Although this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or the scope of what is claimed, but are mainly used to describe the features of specific embodiments of a particular invention. Certain features described in multiple embodiments in this specification may also be combined and implemented in a single embodiment. On the other hand, the various features described in a single embodiment may also be separately implemented in multiple embodiments or implemented in any suitable sub-combination. In addition, although the features may function in certain combinations as described above and are even initially claimed as such, one or more features from the claimed combination may in some cases be removed from the combination, and the claimed combination may refer to a sub-combination or a variant of the sub-combination.

[0254] Among them, the description of "specific examples", "some examples", etc. means that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of this specification. In this specification, the schematic description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0255] Those skilled in the art will readily conceive of other embodiments of this specification after considering the specification and practicing the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations of this specification, which follow the general principles of this specification and include common general knowledge or conventional technical means in the technical field not claimed in this application. The specification and examples are only to be considered as exemplary, and the true scope and spirit of this specification are pointed out by the following claims.

[0256] It should be understood that this specification is not limited to the exact structures already described and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is only limited by the appended claims.

[0257] The above are only the preferred embodiments of this specification and are not intended to limit this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this specification shall be included within the scope of protection of this specification.

Claims

1. A model construction method for constructing a language model capable of performing relevance scoring, the method comprising: Obtaining a training sample set; wherein, the training samples include: query word samples, recommended object texts including recommended object names and recommended object modifiers, and correct scoring texts representing the relevance scores between the query word samples and the recommended object texts; Obtaining prompt content including the query word samples, the recommended object texts, and thought chain prompt texts; the thought chain prompt texts are used to represent the following thought chain: first, identify whether the query word samples include a query object name and a query object modifier, if not, obtain a strong relevance score; if so, first identify whether the query object name matches the recommended object name, if not, obtain an irrelevance score; if they match, in the case where it is determined that the query word sample does not include the query object modifier, obtain the strong relevance score; in the case where it is determined that the query object modifier is included, identify whether the query object modifier matches the recommended object modifier; if they match, obtain the strong relevance score, if not, obtain a weak relevance score; Inputting the prompt content into a preset language model, so that the language model performs reasoning based on the thought chain, and after predicting a predicted scoring text of the relevance score between the query word sample and the recommended object text, trains based on the difference between the predicted scoring text and the correct scoring text; The model construction method is used to: first construct an offline evaluation model using a real training sample set; The offline evaluation model is used to predict, for each synthetic data in the synthetic data set, a predicted scoring text of the relevance score between the query word sample and the recommended object text included in the synthetic data, so that when the predicted scoring text matches the rewritten scoring text included in the synthetic data, the synthetic data is used as a high-quality training sample to obtain a high-quality training sample set; wherein, the synthetic data is generated by prompting a target language model to rewrite the real training samples in the real training sample set; The model construction method is then used to construct an online generation model using the high-quality training sample set; wherein, the model parameters of the offline evaluation model are greater than those of the online generation model; the number of samples in the real training sample set is less than the number of samples in the high-quality training sample set.

2. The method according to claim 1, wherein the scoring text has the following types: A scoring text including the following information: information indicating that the query word sample does not include a query object name, and a strong relevance score indicating the relevance between the query word sample and the recommended object text; A scoring text including the following information: information indicating that the query word sample includes a query object name, information indicating that the query object name does not match the recommended object name, and an irrelevance score indicating the relevance between the query word sample and the recommended object text; A scoring text containing the following information: information indicating that the query term sample contains the name of the query object, information indicating a match between the name of the query object and the name of the recommended object, and a strong correlation score indicating the correlation between the query term sample and the recommended object text; A scoring text containing the following information: information indicating that the query term sample contains the name of the query object, information indicating a match between the name of the query object and the name of the recommended object, information indicating that the query term sample contains a modifier of the query object, information indicating a match between the modifier of the query object and the modifier of the recommended object, and a strong correlation score indicating the correlation between the query term sample and the recommended object text; A scoring text containing the following information: information indicating that the query term sample contains the name of the query object, information indicating a match between the name of the query object and the name of the recommended object, information indicating that the query term sample contains a modifier of the query object, information indicating a non-match between the modifier of the query object and the modifier of the recommended object, and a weak correlation score indicating the correlation between the query term sample and the recommended object text.

3. The method according to claim 1, after constructing the online generation model, the method further comprises: Predicting the correlation score between the current query term input by the user and the text of the object to be sorted by using the online generation model, so as to obtain a first correlation scoring text output by the online generation model; wherein, the text of the object to be sorted is recalled based on the current query term.

4. The method according to claim 3, wherein there are multiple texts of the objects to be sorted, and the method further comprises: Sorting the texts of the objects to be sorted according to the first correlation scoring texts between the current query term and the texts of the objects to be sorted, so as to obtain a sorting result of each object to be sorted; Recommending the sorting result to the user.

5. The method according to claim 3, the method further comprises: Predicting the correlation score between the current query term input by the user and the text of the object to be sorted by using the offline evaluation model, so as to obtain a second correlation scoring text output by the offline evaluation model; Identifying whether the first correlation scoring text is the same as the second correlation scoring text, and if not, constructing a new training sample by using the current query term, the text of the object to be sorted and the second correlation scoring text; The constructed new training sample is used to continue training the online generation model.

6. A scoring method, the method is applied to a server, and the method comprises: Obtaining a current query term and a recommended object text corresponding to the current query term; Obtain a prompt content that includes the current query term, the recommended object text, and the thought chain prompt text; the thought chain prompt text is used to represent the following thought chain: first, identify whether the query term sample includes a query object name and a query object modifier. If the query object name is not included, obtain a strong correlation score. If the query object name is included, first identify whether the query object name matches the recommended object name. If they do not match, obtain an irrelevance score. If they match, in the case where it is determined that the query term sample does not include the query object modifier, obtain the strong correlation score. In the case where it is determined that the query object modifier is included, identify whether the query object modifier matches the recommended object modifier. If they match, obtain the strong correlation score. If they do not match, obtain a weak correlation score; Input the prompt content into a preset language model so that the language model performs reasoning based on the thought chain and predicts a predicted score text of the correlation score between the query term sample and the recommended object text; Wherein, the language model is constructed using the steps of the method according to any one of claims 1 to 5.

7. A search method, which is applied to a client, and the method includes: Obtain the current query term input by the user; After sending the current query term to the server, receive the sorting result sent by the server and display it; Wherein, the server is used to recall each text of the objects to be sorted corresponding to the current query term, generate a first correlation score text between the current query term and each text of the objects to be sorted using an online generation model, and then sort each text of the objects to be sorted to obtain the sorting result of each object to be sorted; wherein, the online generation model is a language model constructed using the steps of the method according to any one of claims 1 to 5.

8. A search system, the search system includes a server and a client, the server is used to execute the steps of the method according to any one of claims 1 to 6, and the client is used to execute the steps of the method according to claim 7.

9. A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor implements the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer program product, including a computer program, and the computer program implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer-readable storage medium, on which a computer program is stored, and the computer program implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

Images