A method and apparatus for determining the relevance of query-item pairs

Abstract

This specification provides a method and apparatus for determining the relevance of query-item pairs. The method includes: obtaining a neighbor set of a target query-target item pair based on historical user behavior data of a target search engine; the neighbor set including at least one of the following sets: a first set of first items corresponding to the target query, and a second set of first queries corresponding to the target items; constructing target hint information based on the target query-target item pair and the neighbor set; and inputting the target hint information into a large language model to obtain a relevance score of the target query-target item pair output by the large language model.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of large language models, and specifically to a method and apparatus for determining the relevance of query-item pairs. Background Technology

[0002] In today's information overload era, search engines have become a core tool for online content platforms, helping users quickly find items that match their queries. Relevance modeling is an important concept in information retrieval and natural language processing (NLP), and a key module for improving the user experience of search engines. Its main purpose in search engines is to measure the relevance of query-item pairs and identify items that match the user's input query.

[0003] A Large Language Model (LLM) is a model trained on a large corpus that can generate and understand natural language. These models are typically based on deep learning techniques, and the training process usually involves preprocessing and extracting features from large amounts of text data before using that data to train the model. When using a LLM for relevance modeling, the model learns the semantic relationships between queries and items during training, determining relevance based on the semantic consistency between queries and items.

[0004] However, semantics is only one aspect of relevance assessment, and current large language models cannot accurately determine the relevance between queries and items solely from a semantic perspective. Summary of the Invention

[0005] This specification provides a scheme for determining the relevance of query-item pairs by incorporating user behavior data into the prompt information to enhance the relevance modeling of the large language model, thereby effectively improving the accuracy of the model's relevance scoring of query-item pairs.

[0006] Firstly, embodiments of this specification provide a method for determining the relevance of query-item pairs, including:

[0007] Based on the historical behavior data of users of the target search engine, obtain the neighbor set of the target query-target item pair, the neighbor set including at least one of the following sets: a first set of the first item corresponding to the target query, and a second set of the first query corresponding to the target item;

[0008] Based on the target query-target item pair and the neighbor set, construct target prompt information;

[0009] The target prompt information is input into the large language model to obtain the relevance score of the target query-target item pair output by the large language model.

[0010] In some embodiments, the target query is a query input by a target user, the target item is an item in the candidate item set corresponding to the target query, and the relevance score of the target query-target item pair is used to determine the priority when the target item is displayed to the target user.

[0011] In some embodiments, after inputting the target suggestion information into a large language model to obtain the relevance score of the target query-target item pair output by the large language model, the method further includes:

[0012] The relevance scores of the target query-target item pairs are stored in an offline database, which is used for offline querying of relevance scores.

[0013] In some embodiments, after storing the relevance score of the target query-target item pair in an offline database, the method further includes:

[0014] In response to receiving a user query request, the system searches the offline database for the target query in the user query request.

[0015] In response to the existence of the target query in the offline database, the relevance score of the target query-item pair corresponding to the target query is obtained from the offline database.

[0016] In some embodiments, obtaining the neighbor set of the target query-target item pair based on the historical behavior data of the target search engine user includes:

[0017] Based on multiple query-item pairs in the historical behavior data of users of the target search engine, target query-target item pairs that meet the exposure and / or click-through rate conditions are filtered out.

[0018] From the multiple query-item pairs, determine the neighbor set of the target query-target item pair.

[0019] In some embodiments, determining the neighbor set of the target query-target item pair from among the plurality of query-item pairs includes:

[0020] From multiple query-item pairs, the first item that meets the first click-through rate requirement corresponding to the target query is selected to obtain the first set; and / or,

[0021] From the multiple query-item pairs, the first query that corresponds to the target item and meets the second click-through rate requirement is selected to obtain the second set.

[0022] In some embodiments, after inputting the target suggestion information into a large language model to obtain the relevance score of the target query-target item pair output by the large language model, the method further includes:

[0023] The large language model is trained based on the relevance score and the relevance labels of the target query-target item pair.

[0024] In some embodiments, the method further includes:

[0025] Based on a preset knowledge base, obtain the query attribute information corresponding to the target query and / or the project attribute information corresponding to the target project;

[0026] The target suggestion information is constructed based on the target query-target item pair and the neighbor set of the target query-target item pair, including:

[0027] Based on the target query-target item pair, the neighbor set of the target query-target item pair, the query attribute information and / or the item attribute information, target prompt information is constructed.

[0028] In some embodiments, constructing target suggestion information based on the target query-target item pair, the neighbor set of the target query-target item pair, the query attribute information, and / or the item attribute information includes:

[0029] Based on the neighbor set of the target query-target item pair, a first target prompt information is constructed. The first target prompt information is used to query the large language model: whether the first set and the second set in the neighbor set are related.

[0030] Based on the neighbor set of the target query-target item pair, the query attribute information and the item attribute information, a second target prompt information is constructed. The second target prompt information is used to query the large language model: given the first set and the second set, are the query attribute information and the item attribute information related?

[0031] Based on the target query-target item pair, the neighbor set of the target query-target item pair, the query attribute information, and the item attribute information, a third target hint information is constructed. The third target hint information is used to query the large language model: given the first set, the second set, the query attribute information, and the item attribute information, whether the target query and the target item are related.

[0032] The step of inputting the target prompt information into the large language model to obtain the relevance score of the target query-target item pair output by the large language model includes:

[0033] The first target prompt information is input into the large language model to obtain the first relevance score of the target query-target item pair output by the large language model;

[0034] The second target prompt information is input into the large language model to obtain the second relevance score of the target query-target item pair output by the large language model;

[0035] The third target prompt information is input into the large language model to obtain the third relevance score of the target query-target item pair output by the large language model.

[0036] In some embodiments, the method further includes:

[0037] A first loss function is determined based on the first relevance score and the relevance label of the target query-target item pair;

[0038] The second loss function is determined based on the second relevance score and the relevance label of the target query-target item pair;

[0039] The third loss function is determined based on the third relevance score and the relevance label of the target query-target item pair;

[0040] Based on the first relevance score, the second relevance score, and the third relevance score, an aggregate relevance score is determined, and based on the aggregate relevance score and the relevance label of the target query-target item pair, a fourth loss function is determined;

[0041] The large language model is trained based on the first loss function, the second loss function, the third loss function, and the fourth loss function.

[0042] In some embodiments, determining the aggregated correlation score based on the first correlation score, the second correlation score, and the third correlation score includes:

[0043] A target function containing learnable parameters is determined, the learnable parameters being adjusted during the training of the large language model, and the value of the target function corresponding to each target prompt is related to the amount of information contained in each target prompt;

[0044] Using the values ​​of the objective functions corresponding to each target prompt as weights, the first relevance score, the second relevance score, and the third relevance score are aggregated to obtain the aggregated relevance score.

[0045] In some embodiments, the objective function is an exponential kernel function, and the learnable parameter is located at the exponential position of the exponential kernel function.

[0046] In some embodiments, obtaining the neighbor set of the target query-target item pair based on the historical behavior data of the target search engine user includes:

[0047] In response to an update to the historical behavior data, the neighbor set of the target query-target item pair is obtained using the updated user search behavior data.

[0048] Secondly, embodiments of this specification provide an apparatus for determining the relevance of query-item pairs, comprising:

[0049] The data acquisition unit is configured to acquire a neighbor set of target query-target item pairs based on the historical behavior data of users of the target search engine. The neighbor set includes at least one of the following sets: a first set of first items corresponding to the target query, and a second set of first queries corresponding to the target item.

[0050] The prompt building unit is configured to construct target prompt information based on the target query-target item pair and the neighbor set;

[0051] The relevance scoring unit is configured to input the target prompt information into a large language model and obtain the relevance score of the target query-target item pair output by the large language model.

[0052] Thirdly, embodiments of this specification provide a computing device including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements the method described in any of the implementations in the first aspect.

[0053] In the solutions provided by the above embodiments of this specification, a set of neighbors for the target query-target item pair is obtained by performing neighbor retrieval on the historical behavior data of the target search engine user. This introduces domain-specific knowledge into the relevance modeling of the large language model. Based on this knowledge, target prompt information is constructed and input into the large language model to obtain a relevance score. This allows the large language model to determine relevance not only from a semantic perspective but also from user behavior data, thereby enabling it to understand the user's implicit search intent and more accurately determine the relevance of the query-item pair. Attached Figure Description

[0054] To more clearly illustrate the technical solutions of the various embodiments disclosed in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only a few embodiments disclosed in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart of the process of training a large language model in the embodiments of this specification;

[0056] Figure 2 This is a schematic diagram of the process of obtaining the neighbor set in the embodiments of this specification;

[0057] Figure 3 This is a schematic diagram of the progressive prompting and progressive aggregation process in the embodiments of this specification;

[0058] Figure 4 This is a schematic diagram illustrating the performance of the model trained under different parameters in the embodiments of this specification;

[0059] Figure 5 This is a schematic diagram illustrating the online and offline collaborative service scenarios in the embodiments of this specification;

[0060] Figure 6 This is another flowchart of the method for determining the relevance of query-item pairs in the embodiments of this specification;

[0061] Figure 7 This is a schematic diagram of the device for determining the relevance of query-item pairs in the embodiments of this specification. Detailed Implementation

[0062] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0063] As mentioned earlier, to ensure a good user experience when using a search engine, it's crucial to guarantee a strong correlation between the user's query and the resulting search results (i.e., items). This makes relevance modeling in search engines essential. When using LLMs for relevance modeling, LLMs can infer the degree of relevance between the query and the items based on clues. Assuming we have a target query q and a target item i, relevance modeling can be simply described as follows:

[0064] By designing a cue message τ(q,i), LLMs can determine, based on the probability P(v|τ(q,i)), which tag word mapping v (i.e., "relevant" or "irrelevant") is most likely to fill the mask. This cue message τ(q,i) can be expressed as:

[0065] Are [Query q] and [Item i] related? [Mask]

[0066] (τ(q,i)=Is[q]and[i]related?[mask])

[0067] Among them, the relevance label y qi ∈{0,1} corresponds to the tag word mapping v (i.e., "relevant" or "irrelevant") in the LLMs vocabulary, and is used to represent the degree of relevance between query q and item i.

[0068] For ease of description, the tag mapping v will be set to relevant in the following text. When the tag mapping v is relevant, the prompt message τ(q,i) can be represented as: Are [Query q] and [Item i] relevant? In this case, the relevant tag y qi The higher the value of , the stronger the relevance between query q and item i. Inputting the hint information τ(q,i) into the model, the probability P(v|τ(q,i)) output by the model can be considered as the relevance score of the query-item pair. To enable general LLMs to adapt to relevance modeling tasks, we can use P(v|τ(q,i)) and y... qi The difference between them determines the loss function, for example, choosing to use the cross-entropy loss function L. cePerform supervised fine-tuning on the model to make the difference between the two smaller. In this way, the trained LLMs can provide relevance scores for the queries entered by users and the candidate items in the search scenario.

[0069] However, most queries and items in the search scenario are usually very short and ambiguous. Although LLMs have rich world knowledge through pre-training on a large amount of text corpora, due to the lack of domain-specific knowledge, LLMs still face difficulties in understanding short and ambiguous query-item pairs from search engines. For example, given a query "Zheyi", it is difficult for the model to understand its actual semantics and accurately give the relevance score between this query and its corresponding item based on semantic understanding. The historical click items for this query include "The First Affiliated Hospital of Zhejiang University", indicating a strong correlation between them.

[0070] Based on the above analysis, considering the historical behavior data of users in the search engine, which helps to identify users' search intentions, embodiments of this specification propose a method for determining the relevance of query-item pairs. By using user behavior data for neighbor retrieval to obtain a neighbor set to assist in constructing prompt information in the relevance modeling of LLMs, the world knowledge in LLMs and the domain-specific knowledge introduced by user behavior data can be effectively combined, making the relevance of query-item pairs determined by LLMs more accurate.

[0071] First, the training process of the large language model used in the embodiments of this specification will be described below. This training process can be executed by any device, platform, or cluster of devices with computing and processing capabilities, including steps S101 - S104 as shown below.

[0072] As Figure 1 shown, in step S101, based on the historical behavior data of users of the target search engine, obtain a neighbor set of target query-target item pairs. The neighbor set includes at least one of the following sets: the first set of the first items corresponding to the target query, and the second set of the first queries corresponding to the target item.

[0073] In practice, the historical behavior data of users can be the user search logs recorded by the search engine or the user behavior data captured through data tracking in applications or websites. The historical behavior data includes the queries entered by users in the search engine and exposure click data such as the browsed and clicked items corresponding to the queries. It should be noted that the historical behavior data of users in this embodiment is not for a specific user, but a large amount of users' historical behavior data is used. In other embodiments, the historical behavior data of a specific user or group of users can also be used.

[0074] In historical behavior data, the item corresponding to the query can be considered the behavior neighbor of that query, and the query corresponding to that item can be considered the behavior neighbor of that item. The neighbor set is obtained by retrieving behavior neighbors from historical behavior data and can reflect the user's search intent. Next, we will combine... Figure 2 The process of obtaining the neighbor set is explained.

[0075] In a search scenario, after a user enters a query, they see multiple candidate items displayed on the page. Users typically click on items that match their search intent. Historical behavioral data, such as search logs, records click data for these query-item pairs. This click data can be considered to reflect the user's satisfaction with the items corresponding to the query. Therefore, for a target query, the first item corresponding to the target query can be determined based on the click data of the target query and its corresponding different items (i.e., target query-item pairs) in historical behavioral data, resulting in a first set of the first items corresponding to the target query. Similarly, for a target item, the first query corresponding to the target item can be determined based on the click data of the target item and its corresponding different queries (i.e., query-target item pairs) in historical behavioral data, resulting in a second set of the first queries corresponding to the target item.

[0076] While domain-specific knowledge in historical behavioral data can enhance the relevance modeling capabilities of LLMs in search scenarios, not all knowledge is beneficial. Noise in historical behavioral data may mislead LLMs into making incorrect judgments. To mitigate the impact of noise, query-item pairs in historical behavioral data can be filtered to obtain a high-confidence log.

[0077] One approach is to filter out target query-target item pairs that meet exposure and / or click-through rate criteria from multiple query-item pairs in the historical behavior data of users of the target search engine, and then determine the neighbor set of the target query-target item pair from the multiple query-item pairs.

[0078] Page Views (PV) are the number of times a query-item pair is shown to a user. When a user enters a query, and an item appears in the page's results, that query-item pair receives one exposure. For example, an exposure threshold can be used. When the exposure threshold is 100, query-item pairs with fewer than 100 PVs can be filtered out, retaining high-confidence target query-item pairs.

[0079] Click-through rate (CTR) is the ratio of clicks to impressions. For multiple items displayed to a user, when a user clicks on an item, the click count for that query-item pair is incremented by 1. For example, a CTR condition could be a CTR threshold. If the CTR threshold is 0.2, query-item pairs with CTRs higher than 0.2 are selected as target query-target item pairs to mitigate the impact of noise.

[0080] In other implementations, neighbor retrieval can be performed on historical behavior data for certain time periods, such as historical behavior data from the past month, to ensure the timeliness of domain-specific knowledge in the historical behavior data.

[0081] After filtering out the target query-target item pairs, we can obtain the behavior neighbors of the target query and the behavior neighbors of the target item to obtain a first set and a second set. The behavior neighbors of the target query are the items corresponding to the target query, and the behavior neighbors of the target item are the queries corresponding to the target item. Considering that in historical behavior data, a target query corresponds to multiple items, and not every item helps identify user intent, and that if the first set contains too many items, the amount of prompt information will be very large, consuming excessive computing resources and affecting computing speed, we can filter the multiple items corresponding to the target query to determine the first item corresponding to the target query. Similarly, we can filter the multiple queries corresponding to the target item to determine the first query corresponding to the target item.

[0082] One implementation method is to filter out the first item that meets the first click-through rate requirement corresponding to the target query from multiple query-item pairs to obtain a first set; and / or, to filter out the first query that meets the second click-through rate requirement corresponding to the target item from multiple query-item pairs to obtain a second set.

[0083] For example, the behavioral neighbors of the target query can be sorted or an inverted index can be used based on click-through rate (CTR) to select the top-K (e.g., 20) behavioral neighbors. These behavioral neighbors can be understood as a set of related items retrieved under the target query, resulting in the first set. Similarly, the behavioral neighbors of the target item can be sorted or an inverted index can be used based on CTR to select the top-K (e.g., 20) behavioral neighbors. These behavioral neighbors can be understood as a set of related queries that can retrieve the target item, resulting in the second set. The first and second CTR requirements can be the same or different.

[0084] like Figure 2 As shown, the first set of the first items corresponding to the target query q is obtained. The second set of the first query corresponding to target item i Then, indexes can be created in the database index for the target query q and the target item i, pointing to respectively. and For each query-item pair, the set of neighbors for that query-item pair can be retrieved from the database.

[0085] Considering that domain knowledge in search scenarios changes rapidly every day, and LLMs lack the ability to quickly adapt to these changes, this affects the model's ability to make accurate relevance judgments. One approach is to respond to updates in historical behavior data by using the updated user search behavior data to obtain the neighbor set of the target query-target item pair. For example, ... Figure 2 As shown, the above operations can be repeated daily using updated search logs, with indexes built daily for the target query and target item respectively, to ensure the timeliness of knowledge in a specific domain.

[0086] It should be clarified that although the behavioral neighbors of most queries and projects can be obtained, the behavioral neighbors of a small number of queries or projects cannot be obtained. In this case, we will set it to empty in the subsequent target prompt information.

[0087] Next, in step S102, target prompt information is constructed based on the target query - target item pair and neighbor set.

[0088] The target prompt information can be a single target prompt information or multiple target prompt information.

[0089] For example, a single target hint could be constructed as: Given a [neighbor set], are [target query q] and [target item i] related?

[0090] While LLMs can infer relevance levels from individual cues, their insensitivity to input suggests a lack of awareness required to identify relevance inferences; they cannot explicitly perceive from which aspects relevance is inferred based on a single cue. Furthermore, reliance on independent cues places high demands on the quality of these cues in relevance modeling; even slight deficiencies in the design of these independent cues can lead to performance degradation. Additionally, the constructed target cues can be multiple.

[0091] For example, the constructed multiple target prompts can be Prompt 1 and Prompt 2, where Prompt 1 is: [First set] 】 and

Second Set

[0092] Next, in step S103, the target prompt information is input into the large language model to obtain the relevance score of the target query-target item pair output by the large language model.

[0093] Specifically, when the target prompt is a single target prompt, the single target prompt is input into the large language model to obtain the relevance score of the target query-target item pair; when the target prompt is multiple target prompts, the multiple target prompts are input into the large language model separately to obtain multiple relevance scores of the target query-target item pair. For example, inputting Prompt 1 and Prompt 2 into the large language model separately will yield the relevance scores corresponding to Prompt 1 and Prompt 2, respectively. The input order of Prompt 1 and Prompt 2 is not required.

[0094] Finally, in step S104, the large language model is trained based on the relevance score and the relevance labels of the obtained target query-target item pairs.

[0095] Specifically, the loss function can be determined by the difference between the relevance score and the relevance label, and the large language model can be trained using the loss function.

[0096] For example, when there is a single target suggestion, the loss function is determined based on the difference between the relevance score corresponding to that target suggestion and the relevance label of the target query-target item pair. When there are multiple target suggestions, all suggestions can share the same relevance label. The loss function can be determined separately based on the difference between the relevance scores corresponding to each of the multiple target suggestions and the relevance label of the target query-target item pair, or the relevance scores corresponding to the multiple target suggestions can be weighted and summed into an overall relevance score, and the loss function can be determined based on the difference between the overall relevance score and the relevance label of the target query-target item pair.

[0097] This embodiment does not restrict the specific loss function used. After determining the loss function, the parameters of the large language model can be adjusted through backpropagation based on it. Network training ends when the network iterations reach the training termination condition. This embodiment does not restrict the training termination condition. This condition can be reaching a certain number of iterations or the loss function value falling below a certain threshold.

[0098] During the training process of the aforementioned large language model, a neighbor set of the target query-target item pair is obtained by performing neighbor retrieval on the historical behavior data of users in the target search engine. This introduces domain-specific knowledge into the training of the large language model. Based on this knowledge, target prompt information is constructed to train the large language model, enabling the large language model to perform relevance modeling not only from a semantic perspective but also from historical behavior data. The trained large language model can understand the user's implicit search intent and make more accurate relevance scores.

[0099] As an implementation approach, in order to make LLMs more sensitive to the relevance judgment of different aspects and improve the robustness of model performance, a progressive prompting approach can be adopted in step S102, that is, to construct prompting information from simple to complex to multiple targets, wherein the amount of information in the prompting information gradually increases.

[0100] For example, the constructed multiple target prompts can be Prompt 1 and Prompt 2, where Prompt 1 is: [First set] 】 and

Second Set

Second Set

[0101] This allows LLMs to infer the probability that the tag word mapping v is "relevant" by gradually using cue information from simple to complex. In subsequent training, different weights can be assigned to target cue information with different amounts of information, thereby enhancing the model's sensitivity to relevance judgment and improving the stability of prediction results.

[0102] In one implementation, in addition to the neighbor set obtained in step S101, knowledge construction prompts for related fields can be obtained from a preset knowledge base to further supplement the specific field knowledge in the prompts. For example, based on the preset knowledge base, query attribute information corresponding to the target query and / or project attribute information corresponding to the target project can be obtained, and then target prompt information can be constructed based on the target query-target project pair, the neighbor set of the target query-target project pair, the query attribute information, and / or the project attribute information.

[0103] A pre-defined knowledge base is a repository containing knowledge, information, and data related to a specific field, such as a technical knowledge base, a scientific knowledge base, or a financial knowledge base. For example, a pre-defined knowledge base could be an industrial knowledge base, which is a collection of knowledge, information, and data related to an industrial field. An industrial knowledge base is created by collecting, organizing, storing, and distributing industry-related knowledge and information. Attribute information for a target query q can be obtained from the industrial knowledge base. and attribute information of target project i Attributes can be descriptive information about a query or item, such as brand, keywords, intent, and category. This information can supplement domain-specific information. For example, when a user enters the query "Beijing Housing Provident Fund," the query attributes could be: keywords "Beijing," "Housing Provident Fund," and category "Public Services." Candidate items could be "Beijing Housing Provident Fund Withdrawal," with attributes of: keywords "Beijing," "Housing Provident Fund," "Withdrawal," intent "Withdraw Housing Provident Fund," and category "Public Services." Candidate items could also be "Beijing Housing Provident Fund Inquiry," with attributes of: keywords "Beijing," "Housing Provident Fund," "Inquiry," intent "Inquire about Housing Provident Fund," and category "Public Services." Similarly, when a user enters the query "McDonald's," the query attributes could be: keywords "McDonald's," brand "McDonald's," and category "Food and Beverage." Candidate items could be "McDonald's Ordering," with attributes of: keywords "McDonald's," "Ordering," brand "McDonald's," intent "Ordering," and category "Food and Beverage."

[0104] Based on the information obtained above, one or more target prompts can be constructed.

[0105] Next, combined Figure 3 This paper explains the process of training a large language model using progressive suggestions and progressive aggregation. The aim is to guide the model to consider different aspects of information step by step through progressive suggestions and aggregation, so that the trained model can accurately evaluate the relevance of query-item pairs.

[0106] First, based on the target query-target item pair, the neighbor set of the target query-target item pair, the query attribute information, and the item attribute information, three target hint messages are constructed from simple to complex:

[0107]

[0108] Based on the neighbor set of the target query-target item pair, construct the first target prompt information. The first objective prompt is used to query the large language model: Are the [first set] and [second set] related? and (Related?)

[0109] Based on the neighbor set of the target query-target item pair, query attribute information and item attribute information to construct a second target hint message. The second objective prompt is used to query the large language model: given the first set and the second set, are the query attribute information and the item attribute information related? (Given) and are and (Related?)

[0110] Based on the target query-target item pair and the neighbor set of the target query-target item pair, query attribute information and item attribute information to construct third target hint information. The third target prompt is used to query the large language model: given the [first set], [second set], [query attribute information], and [item attribute information], are the [target query] and [target item] related? (Given) and Is [q] and [i] related?

[0111] Among them, the amount of information in the target prompts gradually increases from the first target prompt to the third target prompt.

[0112] Next, the constructed target prompts are input into the large language model to obtain multiple relevance scores of the target query-target item pairs output by the large language model.

[0113] Specifically, the first target hint information is input into the large language model to obtain the first relevance score of the target query-target item pair output by the large language model; the second target hint information is input into the large language model to obtain the second relevance score of the target query-target item pair output by the large language model; and the third target hint information is input into the large language model to obtain the third relevance score of the target query-target item pair output by the large language model.

[0114] Then, using the same relevance label, for each relevance score, the corresponding loss function is determined based on the difference between the relevance score and the relevance label.

[0115] Specifically, a first loss function is determined based on the first relevance score and the relevance label of the target query-target item pair; a second loss function is determined based on the second relevance score and the relevance label of the target query-target item pair; and a third loss function is determined based on the third relevance score and the relevance label of the target query-target item pair.

[0116] For example, the loss function can be the cross-entropy loss function, and the l-th loss function corresponding to the l-th target prompt information can be expressed as:

[0117] Next, based on the first relevance score, the second relevance score, and the third relevance score, the aggregate relevance score is determined, and based on the aggregate relevance score and the relevance label of the target query-target item pair, the fourth loss function is determined.

[0118] Specifically, when aggregating the various correlation scores, different weight coefficients can be used to sum the various correlation scores to obtain the aggregated correlation score. In this embodiment, there is no restriction on the selection of the weight coefficients corresponding to the correlation scores. For example, different fixed weights can be assigned to different correlation scores.

[0119] Considering that the amount of information in target prompts gradually increases from simple to complex, the weight coefficients can be set to be related to the amount of information in the target prompts. This is intended to help the model learn the progressive relationships between target prompts from simple to complex during training, thereby improving the model's ability to judge relevance.

[0120] One approach is to use a target function with learnable parameters as the weighting coefficients for the relevance scores.

[0121] Specifically, a target function containing learnable parameters is determined, and the values ​​of the target function corresponding to each target prompt information are used as weights to aggregate the first relevance score, the second relevance score, and the third relevance score to obtain the aggregated relevance score.

[0122] The learnable parameter is used for adjustment during the training of the large language model. That is, when adjusting the network parameters of the large language model according to the loss function, the learnable parameter is also adjusted to obtain optimal weights. The value of the objective function corresponding to each target cue is related to the amount of information contained in each target cue. For example, when it is desired that the importance of the target cue increases with the increase of information content, a kernel function can be used as the objective function to model the incremental trend, so that the value of the objective function increases with the increase of information content. The kernel function chosen can be a Gaussian kernel function, an exponential kernel function, a logarithmic decay kernel function, etc.

[0123] As one implementation, the objective function is an exponential kernel function, and the learnable parameters are located at the exponential position of the exponential kernel function. For example, when the information content of the l-th objective prompt is positively correlated with the value of l, the exponential kernel function corresponding to the l-th objective prompt can be as follows:

[0124] K(Δ l )=Exp(Δ l |λ) (2)

[0125] Where λ is a learnable parameter, Δ l It is the attenuation level of the defined l-th target prompt information, Δ lIt is negatively correlated with l, such as Figure 3 The K-value curve shown indicates that the learned parameter λ has a negative value, and the K value is related to Δ. l It shows a negative correlation.

[0126] The aggregated relevance score is obtained by progressively aggregating the relevance scores corresponding to the above multiple target prompts:

[0127]

[0128] In this embodiment, L is 3, and P(v|τ) l ) is the prompt information for the l-th target τ l The corresponding correlation score.

[0129] Then, based on the aggregate relevance score and the relevance label of the target query-target item pair, the fourth loss function L is determined. main .

[0130] Finally, the large language model is trained based on the first loss function, the second loss function, the third loss function, and the fourth loss function.

[0131] For example, the first, second, third, and fourth loss functions can be weighted and summed to obtain a mixed objective function. Then, the large language model can be trained with the goal of minimizing the loss value of the mixed objective function.

[0132] For example, when iteratively training a large language model, the aggregated relevance score can be used as the output of the overall training task for the main supervision task, while the first, second, and third loss functions are used as secondary supervision tasks. The constructed hybrid objective function is as follows:

[0133] L = L main +αL auxi (4)

[0134]

[0135] Here, α is a hyperparameter used to adjust the weights of the sub-supervisory tasks. In other implementations, it can also be L. auxi Different Configure different weight coefficients to further adjust the impact of different target prompts on the model.

[0136] For example, Figure 4 The diagram illustrates the model performance trained under different parameters. AUC (Area Under Curve) is a model performance evaluation metric used to measure the core performance of the model; a higher AUC value indicates better model performance. Figure 4As shown, the model's performance continuously improves with the increase in the number of behavioral neighbors. Among the different objective functions, the mean-pool kernel, the exponential kernel, the log kernel, and the Gaussian kernel, the exponential kernel performs best. The hyperparameter α performs well when set around 0.2, while values ​​that are too large (e.g., 1.0) or too small (e.g., 0.01) will lead to a decrease in model performance.

[0137] The training method described above employs advanced progressive prompting technology to gradually optimize the output of the large language model. By comprehensively considering multiple dimensions and different amounts of information, the method progressively aggregates the target information, which can effectively enhance the relevance judgment ability of the large language model.

[0138] The following describes the application stages of the large language model, which can be divided into online application and offline application.

[0139] In one embodiment, in the case of an online application, the target query is the query entered by the target user, and the target items are the items in the candidate item set corresponding to the target query. The relevance score of the target query-target item pair is used to determine the priority when the target item is displayed to the target user. When the target user enters the target query into the target search engine, the user device used by the target user sends a user query request containing the target query to the server side corresponding to the target search engine. The server can obtain the neighbor set of the target query-target item pair based on the historical behavior data of users of the target search engine.

[0140] Specifically, after receiving a user query request containing a target query, the server can first determine multiple target items from the candidate item set corresponding to the target query, resulting in multiple target query-target item pairs. For example, for the target query "housing provident fund", its corresponding candidate set could be {"housing provident fund query", "housing provident fund withdrawal", "national housing provident fund", "housing provident fund loan query"}, and the resulting multiple target query-target item pairs could be housing provident fund-housing provident fund query pair, housing provident fund-housing provident fund withdrawal pair, housing provident fund-national housing provident fund pair, and housing provident fund-housing provident fund loan query.

[0141] Then, for each target query-target item pair, obtain the neighbor set of the target query-target item pair. For a more detailed explanation of the process of obtaining the neighbor set, please refer to the relevant explanation in the previous text, which will not be repeated here.

[0142] As one implementation method, the system built during the training phase, such as Figure 2 If the database shown contains a set of neighbors for the target query-target item pair, the set of neighbors can also be retrieved from that database.

[0143] Next, based on the target query-target item pair and the neighbor set, target hint information is constructed. For a more detailed explanation of the process of constructing target hint information, please refer to the relevant instructions in the training phase above, which will not be repeated here.

[0144] Finally, the target suggestion information is input into the large language model to obtain the relevance score of the target query-target item pair output by the large language model.

[0145] Specifically, when the target prompt is a single target prompt, the single target prompt is input into the large language model to obtain the relevance score of the target query-target item pair; when the target prompt is multiple target prompts, the multiple target prompts are input into the large language model separately to obtain multiple relevance scores of the target query-target item pair. The multiple relevance scores can be aggregated to obtain the aggregated relevance score as the final relevance score. The aggregation method can be found in the relevant explanation of the training phase above, and will not be repeated here.

[0146] In yet another embodiment, in the case of an offline application, it may be possible to perform, as follows: Figure 1 In steps S101-S103, the relevance score of each target query-target item pair in the historical behavior data is obtained, and the relevance score of the target query-target item pair is stored in an offline database. When a target query-target item pair has multiple relevance scores due to multiple target hints, these multiple relevance scores can be aggregated to obtain the aggregated relevance score as the final relevance score. The aggregation method can be found in the relevant explanation of the training phase above, and will not be repeated here. The offline database is used for offline querying of relevance scores, and this offline database can be updated in response to updates to the historical behavior data.

[0147] Then, in response to receiving a user query request, the system searches for the target query in the user query request in the offline database. If the target query exists in the offline database, the system retrieves the relevance score of the target query-item pair corresponding to the target query from the offline database.

[0148] Specifically, after receiving a user query request containing a target query, the server can retrieve the target query from the offline database. If the target query is retrieved, the server determines the relevance score of at least one item corresponding to the target query.

[0149] The relevance score obtained during the application phase is used to determine the priority of displaying target items to target users. For example, the priority of displaying target items to target users can be determined based on the relevance score and user preferences. Alternatively, the relevance score can be returned to the user's device or other servers so that the user's device or other servers can determine the priority of displaying target items to target users on the search engine interface based on the relevance score.

[0150] The method for determining the relevance of query-item pairs provided in the above embodiments obtains the neighbor set of the target query-target item pair by performing neighbor retrieval on the historical behavior data of users of the target search engine. This introduces domain-specific knowledge into the relevance modeling of the large language model. Based on this knowledge, target prompt information is constructed and input into the large language model to obtain a relevance score. This allows the large language model to determine relevance not only from a semantic perspective but also from user behavior data. As a result, it can gain insight into the user's implicit search intent, more accurately determine the relevance of query-item pairs, and then accurately determine the priority of displaying target items to target users based on relevance, thereby improving the user experience.

[0151] Considering that in real-world search scenarios, it is difficult to deploy an online LLM to handle all search traffic at an acceptable cost and latency, and given that relevance assessment is objective and non-personalized—meaning that the relevance score of the same query-item pair can be consistent across all users, unlike recommendation algorithms which personalize scores for each user—online services can be provided to all users as long as the relevance score of a query-item pair is available, significantly reducing the number of online requests. Furthermore, considering that the semantic information of specific queries or items is relatively stable and does not change much, the relevance score of query-item pairs does not need to be frequently updated over a period of time. Based on this, embodiments of this specification propose a method for determining the relevance of query-item pairs through online and offline collaborative services, which can provide users with a better search experience at an acceptable cost and latency.

[0152] Below, in conjunction with Figure 5 The diagram illustrates a scenario of online and offline collaborative services, showcasing the process of determining the relevance of query-item pairs using the pre-trained large language model. Figure 6 This is a flowchart of a method for determining the relevance of query-item pairs in embodiments of this specification. This method can be executed by any device, platform, or cluster of devices with computing and processing capabilities, including steps S601-S603 as shown below.

[0153] In step S601, in response to receiving a user query request, the target query in the user query request is searched from the offline database.

[0154] like Figure 6 As shown, the user query request is first sent to the offline database DB in the offline service.

[0155] Next, in step S602, in response to the existence of a target query in the offline database, the relevance score of the target query-item pair corresponding to the target query is obtained from the offline database.

[0156] If the target query exists in the database, the corresponding relevance score is returned.

[0157] For a more detailed explanation of the offline service process, please refer to the relevant instructions above, which will not be repeated here.

[0158] Then, in step S603, in response to the absence of the target query in the offline database, the neighbor set of the target query-target item pair is obtained based on the historical behavior data of the target search engine user. Based on the target query-target item pair and the neighbor set, target hint information is constructed. The target hint information is input into the large language model to obtain the relevance score of the target query-target item pair output by the large language model.

[0159] If a non-target query exists in the database, a request is sent to the online service, and the relevance score is obtained based on the trained LLM. Considering the acceptable cost and latency of the online service, and the very low cost of the offline service, it is possible to choose an LLM trained with more training data for the offline service, such as historical behavioral data with 10 parameters, to improve the accuracy of the relevance score calculation. Conversely, it is possible to choose an LLM trained with less training data for the online service, such as historical behavioral data with 2 parameters, to reduce the latency and deployment cost of the online service.

[0160] As one implementation approach, the large language model used by online services can be obtained by distilling the large language model used by offline services.

[0161] For a more detailed explanation of the online service process, please refer to the relevant instructions above, which will not be repeated here.

[0162] For example, an LLM trained with 10 parameters could be used to perform offline inference using daily updated search logs and update relevance scores stored in an offline database. Simultaneously, a smaller LLM (e.g., an LLM trained with 2 parameters) could be distilled from this LLM and offered online. When a user enters a query into a target search engine, the target search engine first queries the offline database from the offline service and obtains relevance scores based on the query results and candidate items. If the relevance score for the query cannot be obtained from the offline database, the online service will be required to predict the relevance score in a timely manner. Approximately 95% of the traffic will be handled by the offline service, while the remaining traffic will be handled by the online service. This approach significantly improves the efficiency of using large language models for relevance judgment in industrial scenarios and reduces computational resource requirements.

[0163] Figure 7 This is a schematic diagram of a device for determining the relevance of query-item pairs according to an embodiment of this specification. This device can be applied to any device, platform, or cluster of devices with computing and processing capabilities. The device includes:

[0164] The data acquisition unit 71 is configured to acquire a neighbor set of the target query-target item pair based on the historical behavior data of the target search engine. The neighbor set includes at least one of the following sets: a first set of the first item corresponding to the target query, and a second set of the first query corresponding to the target item.

[0165] Hint building unit 72 is configured to build target hint information based on target query-target item pairs and neighbor sets;

[0166] The relevance scoring unit 73 is configured to input target hint information into the large language model and obtain the relevance score of the target query-target item pair output by the large language model.

[0167] In one implementation, the target query is a query entered by the target user, the target item is an item in the candidate item set corresponding to the target query, and the relevance score of the target query-target item pair is used to determine the priority when the target item is displayed to the target user.

[0168] In one embodiment, the apparatus further includes an offline service unit 74 configured to store the relevance scores of target query-target item pairs into an offline database, the offline database being used for offline querying of relevance scores.

[0169] In one implementation, the offline service unit 74 is further configured to, in response to receiving a user query request, search for the target query in the user query request from the offline database; and, in response to the existence of the target query in the offline database, obtain the relevance score of the target query-item pair corresponding to the target query from the offline database.

[0170] In one implementation, the data acquisition unit 71 is specifically configured to filter out target query-target item pairs that meet exposure and / or click-through rate conditions from multiple query-item pairs in the historical behavior data of users of the target search engine; and determine the neighbor set of the target query-target item pairs from the multiple query-item pairs.

[0171] In one implementation, when the data acquisition unit 71 is configured to determine the neighbor set of a target query-target item pair from multiple query-item pairs, it is specifically configured to filter out the first item corresponding to the target query that meets the first click-through rate requirement from multiple query-item pairs to obtain a first set; and / or, filter out the first query corresponding to the target item that meets the second click-through rate requirement from multiple query-item pairs to obtain a second set.

[0172] In one embodiment, the device further includes: an attribute information unit 75, configured to acquire query attribute information corresponding to the target query and / or project attribute information corresponding to the target project based on a preset knowledge base;

[0173] The prompt building unit 72 is specifically configured to build target prompt information based on the target query-target item pair, the neighbor set of the target query-target item pair, query attribute information and / or item attribute information.

[0174] In one implementation, the prompt construction unit 72 is specifically configured to: construct a first target prompt based on the neighbor set of the target query-target item pair, the first target prompt being used to query the large language model whether the first set and the second set in the neighbor set are related; construct a second target prompt based on the neighbor set of the target query-target item pair, querying attribute information and item attribute information, the second target prompt being used to query the large language model whether the attribute information and item attribute information are related given the first set and the second set; and construct a third target prompt based on the target query-target item pair and the neighbor set of the target query-target item pair, querying attribute information and item attribute information, the third target prompt being used to query the large language model whether the target query and the target item are related given the first set, the second set, the query attribute information, and the item attribute information.

[0175] The relevance scoring unit 73 is specifically configured to: input the first target prompt information into the large language model to obtain the first relevance score of the target query-target item pair output by the large language model; input the second target prompt information into the large language model to obtain the second relevance score of the target query-target item pair output by the large language model; and input the third target prompt information into the large language model to obtain the third relevance score of the target query-target item pair output by the large language model.

[0176] In one embodiment, the apparatus further includes: a model training unit 76 configured to determine a first loss function based on a first relevance score and a relevance label of a target query-target item pair; determine a second loss function based on a second relevance score and a relevance label of the target query-target item pair; determine a third loss function based on a third relevance score and a relevance label of the target query-target item pair; determine an aggregated relevance score based on the first, second, and third relevance scores, and determine a fourth loss function based on the aggregated relevance score and the relevance label of the target query-target item pair; and train a large language model based on the first, second, third, and fourth loss functions.

[0177] In one implementation, the model training unit 76 is configured to determine the aggregated relevance score based on the first relevance score, the second relevance score, and the third relevance score. Specifically, this is configured to: determine an objective function containing learnable parameters, which are used to adjust during the training of the large language model, wherein the value of the objective function corresponding to each target prompt is related to the amount of information contained in each target prompt; and aggregate the first relevance score, the second relevance score, and the third relevance score using the value of the objective function corresponding to each target prompt as a weight to obtain the aggregated relevance score.

[0178] In one implementation, the objective function is an exponential kernel function, and the learnable parameters are located at the exponential position of the exponential kernel function.

[0179] In one implementation, the data acquisition unit 71 is specifically configured to acquire a set of neighbors for the target query-target item pair using the updated user search behavior data in response to an update in the historical behavior data.

[0180] This specification also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform actions such as... Figure 1 and Figure 6 The method described.

[0181] This specification also provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements, as shown in the embodiment. Figure 1 and Figure 6 The method described.

[0182] This specification also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implements... Figure 1 and Figure 6 The steps of the described method.

[0183] Those skilled in the art will recognize that the functions described in the various embodiments disclosed in this specification in one or more of the examples above can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0184] In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0185] The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of the multiple embodiments disclosed in this specification. It should be understood that the above descriptions are merely specific implementations of the multiple embodiments disclosed in this specification and are not intended to limit the protection scope of the multiple embodiments disclosed in this specification. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of the multiple embodiments disclosed in this specification should be included within the protection scope of the multiple embodiments disclosed in this specification.

Claims

1. A method for determining relevance of a query-item pair, the method comprising: obtaining a neighbor set of a target query-target item pair based on historical behavior data of a user of a target search engine, the neighbor set comprising at least one of a first set of first items corresponding to the target query and a second set of first queries corresponding to the target item; obtaining query attribute information corresponding to the target query and item attribute information corresponding to the target item based on a preset knowledge base; constructing at least two progressive target prompt information based on the target query-target item pair, the neighbor set, the query attribute information, and the item attribute information: a first target prompt information for asking a large language model whether the first set and the second set in the neighbor set are relevant; a second target prompt information for asking the large language model whether the query attribute information and the item attribute information are relevant given the first set and the second set; a third target prompt information for asking the large language model whether the target query and the target item are relevant given the first set, the second set, the query attribute information, and the item attribute information; inputting the target prompt information into a large language model respectively to obtain a relevance score of the target query-target item pair output by the large language model respectively.

2. The method of claim 1, wherein, The target query is a query input by a target user, the target item is an item in a candidate item set corresponding to the target query, and the relevance score of the target query-target item pair is used to determine a priority of the target item when displayed to the target user.

3. The method of claim 1, wherein, After the target prompt information is input into the large language model respectively to obtain the relevance score of the target query-target item pair output by the large language model respectively, the method further comprises: storing the relevance score of the target query-target item pair into an offline database, the offline database being used for offline query of the relevance score.

4. The method of claim 3, wherein, After the relevance score of the target query-target item pair is stored into the offline database, the method further comprises: in response to receiving a user query request, searching the target query in the user query request from the offline database; in response to the target query existing in the offline database, obtaining a relevance score of a target query-item pair corresponding to the target query from the offline database.

5. The method of claim 1, wherein, The obtaining of the neighbor set of the target query-target item pair based on the historical behavior data of the user of the target search engine comprises: screening a target query-target item pair satisfying an exposure condition and / or a click-through rate condition from a plurality of query-item pairs in the historical behavior data of the user of the target search engine; determining the neighbor set of the target query-target item pair from the plurality of query-item pairs.

6. The method of claim 5, wherein, The determining of the neighbor set of the target query-target item pair from the plurality of query-item pairs comprises: screening, from the plurality of query-item pairs, the first item corresponding to the target query and satisfying a first click rate requirement to obtain the first set; and / or, screening, from the plurality of query-item pairs, the first query corresponding to the target item and satisfying a second click rate requirement to obtain the second set.

7. The method of claim 1, wherein, After the target prompt information is respectively input into the large language model to obtain the relevance scores of the target query-target item pairs output by the large language model, the method further comprises: training the large language model based on the relevance scores and the relevance labels of the target query-target item pairs obtained.

8. The method of claim 1, wherein, the target prompt information is respectively input into the large language model to obtain the relevance scores of the target query-target item pairs output by the large language model, comprising: inputting the first target prompt information into the large language model to obtain the first relevance score of the target query-target item pair output by the large language model; inputting the second target prompt information into the large language model to obtain the second relevance score of the target query-target item pair output by the large language model; inputting the third target prompt information into the large language model to obtain the third relevance score of the target query-target item pair output by the large language model.

9. The method of claim 8, wherein, The method further comprises: determining a first loss function according to the first relevance score and the relevance label of the target query-target item pair; determining a second loss function according to the second relevance score and the relevance label of the target query-target item pair; determining a third loss function according to the third relevance score and the relevance label of the target query-target item pair; determining an aggregated relevance score according to the first relevance score, the second relevance score and the third relevance score, and determining a fourth loss function according to the aggregated relevance score and the relevance label of the target query-target item pair; training the large language model according to the first loss function, the second loss function, the third loss function and the fourth loss function.

10. The method of claim 9, wherein, The determination of the aggregated relevance score according to the first relevance score, the second relevance score and the third relevance score comprises: determining a target function containing learnable parameters, the learnable parameters being used to adjust the target function corresponding to each target prompt information in the training process of the large language model, the value of the target function corresponding to each target prompt information being related to the information amount contained in the target prompt information; aggregating the first relevance score, the second relevance score and the third relevance score by taking the value of the target function corresponding to each target prompt information as a weight to obtain the aggregated relevance score.

11. The method of claim 10, wherein, The target function is an exponential kernel function, and the learnable parameter is located at the exponential position of the exponential kernel function.

12. The method of any one of claims 1-11, wherein, The target function is an exponential kernel function, and the learnable parameter is located at the exponential position of the exponential kernel function. The target function is an exponential kernel function, and the learnable parameter is located at the exponential position of the exponential kernel function. In response to the historical behavior data being updated, a neighbor set of the target query-target item pair is obtained using the updated user search behavior data.

13. An apparatus for determining relevance of a query-item pair, the apparatus comprising: a data obtaining unit configured to obtain, based on historical behavior data of a user of a target search engine, a neighbor set of a target query-target item pair, the neighbor set comprising at least one of a first set of first items corresponding to the target query and a second set of first queries corresponding to the target item; obtain, based on a preset knowledge base, query attribute information corresponding to the target query and item attribute information corresponding to the target item; a prompt constructing unit configured to construct, based on the target query-target item pair, the neighbor set, the query attribute information, and the item attribute information, at least two progressive target prompt information, including: first target prompt information for asking a large language model whether the first set and the second set in the neighbor set are relevant; second target prompt information for asking the large language model whether the query attribute information and the item attribute information are relevant given the first set and the second set; third target prompt information for asking the large language model whether the target query and the target item are relevant given the first set, the second set, the query attribute information, and the item attribute information; a relevance score unit configured to input the target prompt information into a large language model respectively to obtain a relevance score of the target query-target item pair output by the large language model respectively.

14. A computing device comprising a memory and a processor, wherein, The memory stores executable code, and the processor executes the executable code to implement the method of any one of claims 1-12.

Images