Method and device for determining correlation of query-item pair

Abstract

The embodiment of the invention provides a method and device for determining the correlation of a query-item pair. The method comprises the steps that on the basis of historical behavior data of a user of a target search engine, a neighbor set of a target query-target item pair is obtained, and the neighbor set comprises at least one of the following sets: a first set of a first item corresponding to a target query and a second set of the first query corresponding to a target item; based on the target query-target item pair and the neighbor set, constructing target prompt information; and inputting the target prompt information into a large language model to obtain a correlation score of the target query-target item pair output by the large language model.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of large language models, and in particular, to a method and apparatus for determining the relevance of a query-item pair. Background Art

[0002] In the current era of information overload, search engines have become the core tool of online content platforms to help users quickly find items that match their queries. Relevance Modeling is an important concept in information retrieval and natural language processing (NLP), and is also a key module for improving search engine user experience. Its main purpose in search engines is to measure the relevance of query-item pairs and identify items that match the query entered by the user.

[0003] A Large Language Model (LLM) is a model that is trained with a large corpus and can generate and understand natural language. This model is usually based on deep learning technology, and the training process usually involves preprocessing and feature extraction of a large amount of text data, and then using this data to train the model. When using a large language model for relevance modeling, the model can learn the semantic relationship between queries and items during training, and determine relevance based on the semantic consistency between queries and items.

[0004] However, the semantic perspective is only one aspect of relevance judgment, and current large language models cannot accurately judge the relevance between queries and items from a semantic perspective alone. Summary of the invention

[0005] The embodiments of this specification provide a solution for determining the relevance of a query-item pair, which introduces user behavior data in prompt information to enhance the relevance modeling of a large language model, and effectively improves the accuracy of the model's score for the relevance of a query-item pair.

[0006] In a first aspect, an embodiment of the present specification provides a method for determining the relevance of a query-item pair, comprising:

[0007] Based on historical behavior data of users of the target search engine, a neighbor set of a target query-target item pair is obtained, wherein the neighbor set includes at least one of the following sets: a first set of first items corresponding to the target query, and a second set of first queries corresponding to the target item;

[0008] constructing target prompt information based on the target query-target item pair and the neighbor set;

[0009] The target prompt information is input into a large language model to obtain a relevance score of the target query-target item pair output by the large language model.

[0010] In some embodiments, the target query is a query input by a target user, the target item is an item in a candidate item set corresponding to the target query, and the relevance score of the target query-target item pair is used to determine the priority of the target item when it is displayed to the target user.

[0011] In some embodiments, after inputting the target prompt information into a large language model to obtain a relevance score of the target query-target item pair output by the large language model, the method further includes:

[0012] The relevance score of the target query-target item pair is stored in an offline database, and the offline database is used for offline query relevance scores.

[0013] In some embodiments, after storing the relevance score of the target query-target item pair in an offline database, the method further comprises:

[0014] In response to receiving a user query request, searching the offline database for a target query in the user query request;

[0015] In response to the target query existing in the offline database, a relevance score of a target query-item pair corresponding to the target query is obtained from the offline database.

[0016] In some embodiments, obtaining a neighbor set of a target query-target item pair based on historical behavior data of users of the target search engine includes:

[0017] Based on the plurality of query-item pairs in the historical behavior data of users of the target search engine, filter out target query-target item pairs that meet the exposure condition and / or click-through rate condition;

[0018] A neighbor set of the target query-target item pair is determined from the plurality of query-item pairs.

[0019] In some embodiments, determining a neighbor set of the target query-target item pair from the plurality of query-item pairs comprises:

[0020] Filtering the first items that meet the first click-through rate requirement and correspond to the target query from the plurality of query-item pairs to obtain the first set; and / or,

[0021] The first queries corresponding to the target items and meeting the second click rate requirement are screened out from the plurality of query-item pairs to obtain the second set.

[0022] In some embodiments, after inputting the target prompt information into a large language model to obtain a relevance score of the target query-target item pair output by the large language model, the method further includes:

[0023] The large language model is trained based on the relevance score and the obtained relevance label of the target query-target item pair.

[0024] In some embodiments, the method further comprises:

[0025] Based on a preset knowledge base, obtaining query attribute information corresponding to the target query and / or project attribute information corresponding to the target project;

[0026] The constructing target prompt information based on the target query-target item pair and the neighbor set of the target query-target item pair includes:

[0027] Target prompt information is constructed based on the target query-target item pair, a neighbor set of the target query-target item pair, the query attribute information and / or the item attribute information.

[0028] In some embodiments, constructing target prompt information based on the target query-target item pair, the neighbor set of the target query-target item pair, the query attribute information and / or the item attribute information comprises:

[0029] Constructing first target prompt information according to the neighbor set of the target query-target item pair, wherein the first target prompt information is used to query the large language model: whether the first set and the second set in the neighbor set are related;

[0030] Constructing second target prompt information according to the neighbor set of the target query-target item pair, the query attribute information and the item attribute information, wherein the second target prompt information is used to inquire the large language model: given the first set and the second set, whether the query attribute information and the item attribute information are related;

[0031] Constructing third target prompt information according to the target query-target item pair, the neighbor set of the target query-target item pair, the query attribute information and the item attribute information, wherein the third target prompt information is used to inquire the large language model: given the first set, the second set, the query attribute information and the item attribute information, whether the target query and the target item are related;

[0032] The step of inputting the target prompt information into a large language model to obtain a relevance score of the target query-target item pair output by the large language model includes:

[0033] Inputting the first target prompt information into a large language model to obtain a first relevance score of the target query-target item pair output by the large language model;

[0034] Inputting the second target prompt information into a large language model to obtain a second relevance score of the target query-target item pair output by the large language model;

[0035] The third target prompt information is input into a large language model to obtain a third relevance score of the target query-target item pair output by the large language model.

[0036] In some embodiments, the method further comprises:

[0037] determining a first loss function based on the first relevance score and the relevance label of the target query-target item pair;

[0038] determining a second loss function based on the second relevance score and the relevance label of the target query-target item pair;

[0039] determining a third loss function based on the third relevance score and the relevance label of the target query-target item pair;

[0040] determining an aggregated relevance score based on the first relevance score, the second relevance score, and the third relevance score, and determining a fourth loss function based on the aggregated relevance score and a relevance label of the target query-target item pair;

[0041] The large language model is trained according to the first loss function, the second loss function, the third loss function and the fourth loss function.

[0042] In some embodiments, determining an aggregate relevance score according to the first relevance score, the second relevance score, and the third relevance score comprises:

[0043] Determining an objective function including learnable parameters, wherein the learnable parameters are used to adjust during the training of the large language model, and the value of the objective function corresponding to each target prompt information is related to the amount of information contained in the each target prompt information;

[0044] The first relevance score, the second relevance score, and the third relevance score are aggregated using the values ​​of the objective function corresponding to the respective objective prompt information as weights to obtain the aggregated relevance score.

[0045] In some embodiments, the objective function is an exponential kernel function, and the learnable parameter is located at an exponential position of the exponential kernel function.

[0046] In some embodiments, obtaining a neighbor set of a target query-target item pair based on historical behavior data of users of the target search engine includes:

[0047] In response to the historical behavior data being updated, a neighbor set of the target query-target item pair is obtained using the updated user search behavior data.

[0048] In a second aspect, an embodiment of the present specification provides a device for determining the relevance of a query-item pair, including:

[0049] A data acquisition unit is configured to acquire a neighbor set of a target query-target item pair based on historical behavior data of a user of a target search engine, wherein the neighbor set includes at least one of the following sets: a first set of first items corresponding to the target query, and a second set of first queries corresponding to the target item;

[0050] a prompt construction unit configured to construct target prompt information based on the target query-target item pair and the neighbor set;

[0051] The relevance scoring unit is configured to input the target prompt information into a large language model to obtain a relevance score of the target query-target item pair output by the large language model.

[0052] In a third aspect, an embodiment of the present specification provides a computing device, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method described in any implementation manner in the first aspect is implemented.

[0053] In the solution provided by the above-mentioned embodiments of the present specification, a neighbor set of a target query-target item pair is obtained by performing a neighbor search on the historical behavior data of users of the target search engine, thereby introducing knowledge in a specific field for the relevance modeling of a large language model, and constructing target prompt information based on this knowledge and inputting it into the large language model to obtain a relevance score, so that the large language model determines relevance not only from a semantic perspective but also based on user behavior data, thereby being able to gain insight into the user's implicit search intent and more accurately determine the relevance of the query-item pair. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the multiple embodiments disclosed in this specification, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only the multiple embodiments disclosed in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Figure 1 is a flow chart of the process of training a large language model in the embodiment of this specification;

[0056] Figure 2 is a schematic diagram of a process of obtaining a neighbor set in an embodiment of this specification;

[0057] Figure 3 It is a schematic diagram of the progressive prompting and progressive aggregation process in the embodiments of this specification;

[0058] Figure 4 is a schematic diagram of the model performance obtained by training under different parameters in the embodiments of this specification;

[0059] Figure 5 is a schematic diagram of a scenario of online and offline collaborative services in an embodiment of this specification;

[0060] Figure 6 is another flow chart of a method for determining the relevance of a query-item pair according to an embodiment of the present specification;

[0061] Figure 7 Schematic diagram of the structure of an apparatus for determining the relevance of a query-item pair in an embodiment of the present specification. DETAILED DESCRIPTION

[0062] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0063] As mentioned above, in order to ensure that users have a good experience when using search engines, it is necessary to ensure that there is an effective correlation between the query entered by the user and the search results (i.e., the item), which makes relevance modeling in search engines crucial. When using LLMs for relevance modeling, LLMs can infer the degree of relevance between the query and the item through prompt information. Assuming that we have a target query q and a target item i, relevance modeling can be simply expressed as follows:

[0064] By designing the hint information τ(q,i), LLMs can determine which label word mapping v (i.e., “relevant” or “irrelevant”) is most likely to fill the mask according to the probability P(v|τ(q,i)). The hint information τ(q,i) can be expressed as:

[0065] Are [query q] and [item i] related? [mask]

[0066] (τ(q,i)=Is[q]and[i]related?[mask])

[0067] Among them, the relevance label y qi ∈{0,1} corresponds to the label word mapping v (i.e., “relevant” or “irrelevant”) in the LLMs vocabulary, which is used to indicate the degree of relevance between the query q and the item i.

[0068] In order to facilitate the description of this solution, the label word mapping v is set to be relevant in the following text. When the label word mapping v is relevant, the prompt information τ(q,i) can be expressed as: Are [query q] and [item i] relevant? At this time, the relevance label y qi The higher the value of , the higher the correlation between query q and item i. The hint information τ(q,i) is input into the model, and the probability P(v|τ(q,i)) output by the model can be regarded as the correlation score of the query-item pair. In order to make general LLMs adapt to the correlation modeling task, we can use P(v|τ(q,i)) and y qi The difference between them determines the loss function, such as choosing to use the cross entropy loss function L ce, supervised fine-tuning operations are performed on the model to make the difference between the two smaller and smaller. In this way, the trained LLMs can provide relevance scores for user-entered queries and candidate items in search scenarios.

[0069] However, most queries and items in search scenarios are usually very short and vague. Although LLMs have rich world knowledge through pre-training on a large amount of text corpus, they still face difficulties in understanding short and vague query-item pairs from search engines due to the lack of knowledge in specific fields. For example, given a query "Zhejiang First", it is difficult for the model to understand its actual semantics and cannot accurately give the relevance score of the query and its corresponding item based on semantic understanding, while the historical click items of the query include "The First Affiliated Hospital of Zhejiang University", which shows that there is a strong correlation between them.

[0070] Based on the above analysis, considering that the historical behavior data of users in the search engine is helpful to identify the user's search intention, the embodiment of this specification proposes a method for determining the relevance of query-item pairs, which uses user behavior data to perform neighbor retrieval to obtain a neighbor set to assist in the construction of prompt information in the relevance modeling of LLMs, thereby effectively combining the world knowledge possessed by LLMs and the domain-specific knowledge introduced by user behavior data, so that the relevance of query-item pairs determined by LLMs is more accurate.

[0071] The following first describes the training process of the large language model used in the embodiments of this specification. The training process can be performed by any device, platform or device cluster with computing and processing capabilities, including steps S101-S104 shown below.

[0072] like Figure 1 As shown, in step S101, based on the historical behavior data of users of the target search engine, a neighbor set of the target query-target item pair is obtained, and the neighbor set includes at least one of the following sets: a first set of first items corresponding to the target query, and a second set of first queries corresponding to the target items.

[0073] In practice, the user's historical behavior data can be the user search log recorded by the search engine, or the user behavior data captured by embedding in the application or website. The historical behavior data includes the query entered by the user in the search engine, and the exposure and click data such as the browsed and clicked items corresponding to the query. It should be noted that the user's historical behavior data in this embodiment is not for a certain user, but uses the historical behavior data of a large number of users. In other embodiments, the historical behavior data of a certain or certain specific user groups may also be used.

[0074] In the historical behavior data, the item corresponding to the query can be regarded as the behavioral neighbor of the query, and the query corresponding to the item can be regarded as the behavioral neighbor of the item. The neighbor set is obtained by performing behavioral neighbor retrieval on the historical behavior data, which can reflect the user's search intention. Next, we will combine Figure 2 The process of obtaining the neighbor set is described.

[0075] In a search scenario, after a user enters a query, they can see multiple candidate items displayed on the page. Users usually click on items that match their search intent. Historical behavior data, such as the search log, records the click data of the query-item pair. It can be considered that the click data reflects the user's satisfaction with the item corresponding to the query. Therefore, for a target query, the first item corresponding to the target query can be determined based on the click data of the target query and its corresponding different items (i.e., the target query-item pair) in the historical behavior data, and the first set of first items corresponding to the target query can be obtained; similarly, for a target item, the first query corresponding to the target item can be determined based on the click data of the target item and its corresponding different queries (i.e., the query-target item pair) in the historical behavior data, and the second set of first queries corresponding to the target item can be obtained.

[0076] Considering that although the domain-specific knowledge in historical behavior data can improve the relevance modeling ability of LLMs in search scenarios, not all knowledge is beneficial, the noise in historical behavior data may mislead LLMs to make wrong judgments. In order to reduce the impact of noise, the query-item pairs in the historical behavior data can be filtered to obtain high-confidence logs.

[0077] As an implementation method, based on multiple query-item pairs in the historical behavior data of users of the target search engine, target query-target item pairs that meet exposure conditions and / or click-through rate conditions can be screened out, and then a neighbor set of the target query-target item pair can be determined from the multiple query-item pairs.

[0078] Page View (PV) is the number of times a query-item pair is displayed to a user. When a user enters a query, an item appears in the displayed results of the page, that is, the query-item pair is exposed once. Exemplarily, the exposure condition can be an exposure threshold. When the exposure threshold is 100, query-item pairs with less than 100 PVs can be filtered out, and high-confidence target query-target item pairs are retained.

[0079] The click-through rate is the ratio of the number of clicks to the number of exposures. For multiple items displayed to the user, when the user clicks on an item, the number of clicks on the query-item pair increases by 1. Exemplarily, the click-through rate condition may be a click-through rate threshold. When the click-through rate threshold is 0.2, the query-item pair with a click-through rate higher than 0.2 is selected as the target query-target item pair to reduce the impact of noise.

[0080] In other implementations, neighbor retrieval may be performed on historical behavior data of a certain time period, for example, historical behavior data of the past month, so as to ensure the timeliness of specific domain knowledge in the historical behavior data.

[0081] After filtering out the target query-target item pair, the behavior neighbors of the target query and the behavior neighbors of the target item can be obtained respectively to obtain the first set and the second set, wherein the behavior neighbors of the target query are the items corresponding to the target query, and the behavior neighbors of the target item are the queries corresponding to the target item. Considering that in the historical behavior data, the target query corresponds to multiple items, and not every item is helpful in identifying the user's intention, and if there are too many items in the first set, the amount of prompt information data will be very large, occupying too many computing resources and affecting the computing speed. Therefore, the multiple items corresponding to the target query can be screened to determine the first item corresponding to the target query. Similarly, the multiple queries corresponding to the target item are screened to determine the first query corresponding to the target item.

[0082] As an implementation method, the first item corresponding to the target query and meeting the first click-through rate requirement can be screened out from multiple query-item pairs to obtain the first set; and / or the first query corresponding to the target item and meeting the second click-through rate requirement can be screened out from multiple query-item pairs to obtain the second set.

[0083] For example, the behavioral neighbors of the target query can be sorted or inverted indexed according to the click-through rate, so as to select the top-K (for example, 20) behavioral neighbors, which can be understood as the set of related items retrieved under the target query, to obtain the first set. Similarly, the behavioral neighbors of the target item can be sorted or inverted indexed according to the click-through rate, so as to select the top-K (for example, 20) behavioral neighbors, which can be understood as the set of related queries that can retrieve the target item, to obtain the second set. The first click-through rate requirement and the second click-through rate requirement can be the same or different.

[0084] like Figure 2 As shown, after obtaining the first set of the first items corresponding to the target query q The second set of the first query corresponding to the target item i After that, you can create indexes for the target query q and the target item i in the database Index, pointing to and For each target query-item pair, a neighbor set of the target query-item pair may be retrieved from the database.

[0085] Considering that professional domain knowledge in search scenarios changes rapidly every day, and LLMs lack the ability to quickly adapt to these changes, which affects the model's ability to make accurate relevance judgments, as an implementation method, in response to updates to historical behavior data, the updated user search behavior data can be used to obtain the neighbor set of the target query-target item pair. For example, Figure 2 As shown, the above operation can be repeated every day using updated search logs, and indexes can be built for target queries and target items respectively every day to ensure the timeliness of knowledge in a specific field.

[0086] It should be clear that although the behavioral neighbors of most queries and items can be obtained, there are still a small number of behavioral neighbors of queries or items that cannot be obtained. For this case, we will set it to empty in the subsequent target prompt information.

[0087] Next, in step S102, target prompt information is constructed based on the target query-target item pair and the neighbor set.

[0088] The target prompt information may be single target prompt information or multiple target prompt information.

[0089] Exemplarily, the constructed single target prompt information can be: given [neighborhood set], are [target query q] and [target item i] related?

[0090] Considering that although LLMs can infer the degree of relevance through separate prompt information, the insensitivity of LLMs to the input indicates that they lack the awareness required to identify aspects of relevance reasoning, and cannot clearly perceive which aspects to infer relevance from for a single prompt information. Moreover, the reliance on independent prompt information makes relevance modeling require high quality of prompt information. A slight insufficiency in the design of the independent prompt may lead to a decline in model performance. The target prompt information constructed can also be multiple target prompt information.

[0091] Exemplarily, the multiple target prompt information constructed may be Prompt 1 and Prompt 2, wherein Prompt 1 is: [the first set 】 and

【Second set ] Are they related? Prompt 2 is: Are [target query q] and [target item i] related?

[0092] Next, in step S103, the target prompt information is input into the large language model to obtain the relevance score of the target query-target item pair output by the large language model.

[0093] Specifically, when the target prompt information is a single target prompt information, the single target prompt information is input into the large language model to obtain the relevance score of the target query-target item pair; when the target prompt information is multiple target prompt information, the multiple target prompt information are respectively input into the large language model to obtain multiple relevance scores of the target query-target item pair, for example, Prompt 1 and Prompt 2 are respectively input into the large language model to obtain the relevance score corresponding to Prompt 1 and the relevance score corresponding to Prompt 2, and there is no requirement for the input order of Prompt 1 and Prompt 2.

[0094] Finally, in step S104, the large language model is trained based on the relevance scores and the obtained relevance labels of the target query-target item pairs.

[0095] Specifically, the loss function can be determined by the difference between the relevance score and the relevance label, and the large language model can be trained by the loss function.

[0096] For example, when the target prompt information is a single target prompt information, the loss function is determined according to the difference between the relevance score corresponding to the target prompt information and the relevance label of the target query-target item pair; when the target prompt information is multiple target prompt information, all prompts can share the same relevance label, and the loss function can be determined according to the difference between the relevance scores corresponding to the multiple target prompt information and the relevance label of the target query-target item pair, or the relevance scores corresponding to the multiple target prompt information are weighted and summed into an overall relevance score, and the loss function is determined according to the difference between the overall relevance score and the relevance label of the target query-target item pair.

[0097] This embodiment does not restrict the specific loss function to be used. After determining the loss function, the parameters of the large language model can be adjusted by back propagation according to the loss function. When the network iteration reaches the training end condition, the network training is terminated. This embodiment does not restrict the training end condition. Among them, the condition can be that the iteration reaches a certain number of times, or the value of the loss function is less than a certain threshold.

[0098] During the training process of the above-mentioned large language model, a neighbor set of the target query-target item pair is obtained by performing neighbor retrieval on the user's historical behavior data in the target search engine, thereby introducing domain-specific knowledge for the training of the large language model. Based on this knowledge, target prompt information is constructed to train the large language model, so that the large language model performs relevance modeling not only based on the semantic perspective but also based on the historical behavior data. The trained large language model can gain insight into the user's implicit search intention and perform more accurate relevance scoring.

[0099] As an implementation method, in order to make LLMs more sensitive to correlation judgments in different aspects and improve the robustness of model performance, a progressive prompting method can be adopted in step S102, that is, constructing prompt information from simple to complex to multiple targets, wherein the amount of information in the prompt information gradually increases.

[0100] Exemplarily, the multiple target prompt information constructed may be Prompt 1 and Prompt 2, wherein Prompt 1 is: [the first set 】 and

Second set

Second set

[0101] This allows LLMs to infer the possibility that the label word mapping v is "relevant" by gradually using prompt information from simple to complex. In the subsequent training process, different weights can be given to target prompt information with different amounts of information, thereby enhancing the model's sensitivity to relevance judgment and improving the stability of the prediction results.

[0102] In one embodiment, in addition to the neighbor set obtained in step S101, knowledge in related fields can also be obtained from a preset knowledge base to construct prompt information, so as to further supplement the specific field knowledge in the prompt information. For example, based on the preset knowledge base, query attribute information corresponding to the target query and / or project attribute information corresponding to the target project are obtained, and then the target prompt information is constructed based on the target query-target project pair, the neighbor set of the target query-target project pair, the query attribute information and / or the project attribute information.

[0103] The preset knowledge base is a library containing knowledge, information and data in related fields, such as a technical knowledge base, a scientific knowledge base and a financial knowledge base. For example, the preset knowledge base can be an industrial knowledge base, which is a collection of knowledge, information and data related to the industrial field. The industrial knowledge base is obtained by collecting, organizing, storing and distributing industrial-related knowledge and information. The attribute information of the target query q can be obtained from the industrial knowledge base. and the attribute information of the target item i Attribute information (Attributes) can be descriptive information about the query or project such as brand, keyword, intent, category, etc., which can be used as a supplement to the field-specific information. For example, when the query entered by the user is "Beijing Provident Fund", the query attribute information can be: keywords "Beijing" "Provident Fund", category "People's Livelihood Services"; the candidate project can be "Beijing Provident Fund Withdrawal", the project attribute information is: keywords "Beijing" "Provident Fund" "Withdrawal", intent "Withdrawal of Provident Fund", category "People's Livelihood Services"; the candidate project can also be "Beijing Provident Fund Query", the project attribute information is: keywords "Beijing" "Provident Fund" "Query", intent "Query Provident Fund", category "People's Livelihood Services". When the query entered by the user is "McDonald's", the query attribute information is: keywords "McDonald's", brand "McDonald's", category "Catering"; the candidate project can be "McDonald's Order", the project attribute information is: keywords "McDonald's" "Order", brand "McDonald's", intent "Order", category "Catering".

[0104] Based on the above information obtained, single or multiple target prompt information can be constructed.

[0105] Next, combine Figure 3 , the process of training a large language model using progressive hints and progressive aggregation is explained. It is expected that by using progressive hints and aggregation process, the model can be gradually guided to consider different aspects of information, so that the trained model can accurately evaluate the relevance of query-item pairs.

[0106] First, three target prompt information from simple to complex are constructed using the target query-target item pair, the neighbor set of the target query-target item pair, the query attribute information, and the item attribute information:

[0107]

[0108] Construct the first target prompt information based on the neighbor set of the target query-target item pair The first target prompt is used to ask the large language model: Are [the first set] and [the second set] related? and related? )

[0109] According to the neighbor set of the target query-target item pair, query attribute information and item attribute information to construct the second target prompt information The second target prompt information is used to ask the large language model: Given the [first set] and [second set], are [query attribute information] and [item attribute information] related? and are and related? )

[0110] According to the target query-target item pair and the neighbor set of the target query-target item pair, query the attribute information and item attribute information, and construct the third target prompt information The third target prompt information is used to ask the large language model: Given the [first set], [second set], [query attribute information], and [item attribute information], are the [target query] and [target item] related? (Given and is[q]and[i]related? )

[0111] Among them, from the first target prompt information to the third target prompt information, the amount of information in the target prompt information gradually increases.

[0112] Next, the constructed multiple target prompt information is respectively input into the large language model to obtain multiple relevance scores of the target query-target item pairs output by the large language model.

[0113] Specifically, the first target prompt information is input into the large language model to obtain a first relevance score of the target query-target item pair output by the large language model; the second target prompt information is input into the large language model to obtain a second relevance score of the target query-target item pair output by the large language model; the third target prompt information is input into the large language model to obtain a third relevance score of the target query-target item pair output by the large language model.

[0114] Then, using the same relevance label, for each relevance score, its corresponding loss function is determined according to the difference between the relevance score and the relevance label.

[0115] Specifically, a first loss function is determined based on the first relevance score and the relevance label of the target query-target item pair; a second loss function is determined based on the second relevance score and the relevance label of the target query-target item pair; and a third loss function is determined based on the third relevance score and the relevance label of the target query-target item pair.

[0116] For example, the loss function can use the cross entropy loss function, and the lth loss function corresponding to the lth target prompt information can be expressed as

[0117] Next, an aggregated relevance score is determined based on the first relevance score, the second relevance score, and the third relevance score, and a fourth loss function is determined based on the aggregated relevance score and a relevance label of the target query-target item pair.

[0118] Specifically, when aggregating the various correlation scores, different weight coefficients may be used to perform weighted summation on the various correlation scores to obtain an aggregated correlation score. This embodiment does not restrict the selection of weight coefficients corresponding to the correlation scores. For example, different fixed weights may be assigned to different correlation scores.

[0119] Considering that the amount of information in target prompt information increases gradually from simple to complex, the weight coefficient can be set to be related to the amount of information in the target prompt information, in the expectation that the model will learn the progressive relationship in target prompt information from simple to complex during training, thereby improving the model's relevance judgment ability.

[0120] As an implementation approach, an objective function including learnable parameters may be used as weight coefficients for the relevance score.

[0121] Specifically, an objective function including learnable parameters is determined, and the values ​​of the objective function corresponding to each target prompt information are used as weights to aggregate the first relevance score, the second relevance score, and the third relevance score to obtain an aggregated relevance score.

[0122] Among them, the learnable parameters are used to adjust during the training process of the large language model, that is, when the network parameters of the large language model are adjusted according to the loss function, the learnable parameters are also adjusted to obtain a better weight. The value of the objective function corresponding to each target prompt information is related to the amount of information contained in each target prompt information. For example, when it is expected that the importance of the target prompt information will increase as the amount of information increases, a kernel function can be used as the objective function to model the incremental trend so that the value of the objective function increases as the amount of information increases. The selected kernel function can be a Gaussian kernel function, an exponential kernel function, a logarithmic decay kernel function, etc.

[0123] As an implementation method, the objective function is an exponential kernel function, and the learnable parameter is located at the exponential position of the exponential kernel function. For example, when the amount of information of the lth target prompt information is positively correlated with the value of l, the exponential kernel function corresponding to the lth target prompt information can be as follows:

[0124] K(Δ l )=Exp(Δ l |λ) (2)

[0125] Among them, λ is a learnable parameter, Δ l is the attenuation degree of the defined l-th target prompt information, Δ lIt is negatively correlated with l, such as Figure 3 The K value curve shown in the figure shows that the value learned by the learnable parameter λ is negative, and the K value is related to Δ l Negatively correlated.

[0126] The relevance scores corresponding to the above multiple target prompt information are progressively aggregated to obtain the aggregated relevance score:

[0127]

[0128] In this embodiment, L is 3, P(v|τ l ) is the lth target prompt information τ l The corresponding relevance score.

[0129] Then, the fourth loss function L is determined based on the aggregated relevance score and the relevance label of the target query-target item pair. main .

[0130] Finally, the large language model is trained according to the first loss function, the second loss function, the third loss function and the fourth loss function.

[0131] For example, the first loss function, the second loss function, the third loss function and the fourth loss function can be weightedly summed to obtain a mixed objective function, and then the large language model can be trained with the goal of minimizing the loss value of the mixed objective function.

[0132] For example, when iteratively training a large language model, the aggregated relevance score can be used as the output of the overall training task to perform the main supervision task, and the first loss function, the second loss function, and the third loss function can be used as secondary supervision tasks. The constructed hybrid objective function is as follows:

[0133] L=L main +αL auxi (4)

[0134]

[0135] Among them, α is a hyperparameter used to adjust the weight of the sub-supervision task. In other implementations, L auxi Different Configure different weight coefficients to further adjust the impact of different target prompt information on the model.

[0136] For example, Figure 4 The figure shows the model performance obtained by training under different parameters, where AUC (Area Under Curve) is a model performance evaluation indicator used to measure the core performance of the model. The larger the AUC value, the better the performance of the model. Figure 4As shown in the figure, as the number of behavioral neighbors increases, the performance of the model continues to improve; for different objective functions, the mean pool kernel function (Mean), exponential kernel function (Exp), logarithmic kernel function (Log), and Gaussian kernel function (Norm), the exponential kernel function works best; the hyperparameter α works best when set near 0.2, and values ​​that are too large (such as 1.0) or too small (such as 0.01) will lead to a decrease in model performance.

[0137] The above training method uses advanced progressive prompting technology to gradually optimize the output of the large language model, and gradually aggregates the target improvement information by comprehensively considering multiple dimensions and different amounts of information, which can effectively enhance the relevance judgment ability of the large language model.

[0138] The application stages of the large language model are explained below. The application stages can be divided into online application and offline application.

[0139] In one embodiment, in the case of an online application, the target query is a query input by a target user, the target item is an item in a candidate item set corresponding to the target query, and the relevance score of the target query-target item pair is used to determine the priority of the target item when it is displayed to the target user. After the target user inputs the target query in the target search engine, the user device used by the target user sends a user query request containing the target query to the server side corresponding to the target search engine, and the server can obtain a neighbor set of the target query-target item pair based on the historical behavior data of the user of the target search engine.

[0140] Specifically, after receiving a user query request containing a target query, the server can first determine multiple target items from the candidate item set corresponding to the target query, and obtain multiple target query-target item pairs. For example, for the target query "provident fund", the corresponding candidate set can be {"provident fund query", "provident fund withdrawal", "national housing provident fund", "provident fund housing loan query"}, and the multiple target query-target item pairs obtained can be provident fund-provident fund query pairs, provident fund-provident fund withdrawal pairs, provident fund-national housing provident fund pairs and provident fund-provident fund housing loan queries.

[0141] Then, for each target query-target item pair, a neighbor set of the target query-target item pair is obtained. For a more detailed explanation of the process of obtaining the neighbor set, please refer to the relevant description in the previous article, which will not be repeated here.

[0142] As an implementation method, the training phase constructs Figure 2 When a neighbor set of the target query-target item pair exists in the database shown, the neighbor set may also be retrieved from the database.

[0143] Next, based on the target query-target item pair and the neighbor set, the target prompt information is constructed. For a more detailed explanation of the process of constructing the target prompt information, please refer to the relevant description of the training stage in the previous article, which will not be repeated here.

[0144] Finally, the target prompt information is input into the large language model to obtain the relevance score of the target query-target item pair output by the large language model.

[0145] Specifically, when the target prompt information is a single target prompt information, the single target prompt information is input into the large language model to obtain the relevance score of the target query-target item pair; when the target prompt information is multiple target prompt information, the multiple target prompt information are respectively input into the large language model to obtain multiple relevance scores of the target query-target item pair. The multiple relevance scores can be aggregated to obtain an aggregated relevance score as the final relevance score. The aggregation method can refer to the relevant instructions in the previous training stage, which will not be repeated here.

[0146] In another embodiment, in the case of offline application, it may be performed as follows Figure 1 In the process of step S101-step S103, the relevance score of each target query-target item pair in the historical behavior data is obtained, and the relevance score of the target query-target item pair is stored in the offline database, wherein, when a target query-target item pair has multiple relevance scores due to multiple target prompt information, the multiple relevance scores can be aggregated to obtain the aggregated relevance score as the final relevance score. The aggregation method can refer to the relevant description of the training stage above, which will not be repeated here. The offline database is used for offline query relevance scores, and the offline database can be updated in response to the update of historical behavior data.

[0147] Then, in response to receiving a user query request, searching the offline database for a target query in the user query request, and in response to the target query existing in the offline database, obtaining a relevance score of a target query-item pair corresponding to the target query from the offline database.

[0148] Specifically, after receiving a user query request including a target query, the server may retrieve the target query from the offline database, and when the target query is retrieved, determine a relevance score of at least one item corresponding to the target query.

[0149] The relevance score obtained in the application phase is used to determine the priority of the target project when it is displayed to the target user. For example, the priority of the target project when it is displayed to the target user can be determined based on the relevance score and information such as user preferences, or the relevance score can be returned to the user device or other server so that the user device or other server determines the priority of the target project when it is displayed to the target user on the search engine interface based on the relevance score.

[0150] The method for determining the relevance of a query-item pair provided in the above embodiment obtains a neighbor set of a target query-target item pair by performing a neighbor search on the historical behavior data of users of a target search engine, thereby introducing knowledge in a specific field for the relevance modeling of a large language model, and constructs target prompt information based on the knowledge and inputs it into the large language model to obtain a relevance score, so that the large language model determines relevance not only from a semantic perspective but also based on user behavior data, thereby being able to gain insight into the user's implicit search intent, more accurately determine the relevance of a query-item pair, and then accurately determine the priority of displaying the target item to the target user based on the relevance, thereby improving the user experience.

[0151] Considering that in real-world search scenarios, it is difficult to deploy LLM online to handle all search traffic at an acceptable cost and delay, and considering that the judgment of relevance is objective and non-personalized, this means that the relevance score of the same query-item pair can be consistent for all users, rather than personalized for each user like the recommendation algorithm. Therefore, as long as the relevance score of a query-item pair can be obtained, the relevance score can be used to provide offline services to all users, which can greatly reduce the number of online requests. In addition, it is also considered that for specific queries or items, their semantic information is relatively stable and does not change much, and the relevance score of the query-item pair does not need to be frequently updated over a period of time. Based on this, the embodiment of this specification proposes a method for determining the relevance of query-item pairs for online and offline collaborative services, which can provide users with a better search experience at an acceptable cost and delay.

[0152] Next, combine Figure 5 A diagram of the online and offline collaborative service scenario, showing the process of determining the relevance of query-item pairs using the large language model trained above. Figure 6 Flow chart of a method for determining the relevance of a query-item pair in an embodiment of the present specification. The method can be executed by any device, platform or device cluster with computing and processing capabilities, including steps S601-S603 as shown below.

[0153] In step S601, in response to receiving a user query request, a target query in the user query request is searched from an offline database.

[0154] like Figure 6 As shown, the user query request request is first sent to the offline database DB in the offline service.

[0155] Next, in step S602, in response to the existence of the target query in the offline database, a relevance score of a target query-item pair corresponding to the target query is obtained from the offline database.

[0156] If the target query exists in the DB, the corresponding relevance score is returned.

[0157] For a more detailed explanation of the offline service process, please refer to the relevant instructions in the previous article, which will not be repeated here.

[0158] Then, in step S603, in response to the absence of the target query in the offline database, a neighbor set of the target query-target item pair is obtained based on the historical behavior data of users of the target search engine, target prompt information is constructed based on the target query-target item pair and the neighbor set, the target prompt information is input into the large language model, and a relevance score of the target query-target item pair output by the large language model is obtained.

[0159] If there is a non-target query in the DB, the request will continue to be sent to the online service, and then the relevance score will be obtained based on the trained LLM. Considering the affordable cost and delay of online services and the very low cost of offline services, LLMs trained with more training data can be selected for offline services, such as historical behavior data with 10B parameters, to improve the accuracy of relevance score calculation, and LLMs trained with less training data can be selected for online services, such as historical behavior data with 2B parameters, to reduce the delay and deployment cost of online services.

[0160] As an implementation method, the large language model used by the online service can be obtained by distilling the large language model used by the offline service.

[0161] For a more detailed explanation of the online service process, please refer to the relevant instructions in the previous article, which will not be repeated here.

[0162] Exemplarily, an LLM obtained by training with 10B parameters can be used to perform offline reasoning using search logs updated daily, and to update the relevance scores stored in the offline database. At the same time, a smaller LLM (for example, an LLM obtained by training with 2B parameters) obtained by distilling the LLM can be used for online services. When a user enters a query in the target search engine, the target search engine will first query the offline database from the offline service, and obtain the relevance score based on the query results and candidate items. If the relevance score pair corresponding to the query cannot be obtained from the offline database, the online service will be required to predict the relevance score in a timely manner. Approximately 95% of the traffic will be handled by the offline service, and the remaining traffic will be handled by the online service. In this way, the efficiency of using large language models to process relevance judgments in industrial scenarios can be greatly improved, and the demand for computing resources can be reduced.

[0163] Figure 7 1 is a schematic diagram of a device for determining the relevance of a query-item pair in an embodiment of the present specification. The device can be applied to any device, platform or device cluster with computing and processing capabilities. The device includes:

[0164] The data acquisition unit 71 is configured to acquire a neighbor set of a target query-target item pair based on historical behavior data of users of the target search engine, wherein the neighbor set includes at least one of the following sets: a first set of first items corresponding to the target query, and a second set of first queries corresponding to the target items;

[0165] a prompt construction unit 72 configured to construct target prompt information based on the target query-target item pair and the neighbor set;

[0166] The relevance scoring unit 73 is configured to input the target prompt information into the large language model to obtain the relevance score of the target query-target item pair output by the large language model.

[0167] In one implementation, the target query is a query input by a target user, the target item is an item in a candidate item set corresponding to the target query, and the relevance score of the target query-target item pair is used to determine the priority of the target item when it is displayed to the target user.

[0168] In one embodiment, the apparatus further includes: an offline service unit 74 configured to store the relevance score of the target query-target item pair into an offline database, where the offline database is used for offline query relevance scores.

[0169] In one embodiment, the offline service unit 74 is further configured to, in response to receiving a user query request, search for a target query in the user query request from an offline database; in response to the existence of the target query in the offline database, obtain a relevance score of a target query-item pair corresponding to the target query from the offline database.

[0170] In one embodiment, the data acquisition unit 71 is specifically configured to screen out target query-target item pairs that meet exposure conditions and / or click-through rate conditions based on multiple query-item pairs in the historical behavior data of users of the target search engine; and determine a neighbor set of the target query-target item pair from the multiple query-item pairs.

[0171] In one embodiment, when the data acquisition unit 71 is configured to determine a neighbor set of a target query-target item pair from multiple query-item pairs, it is specifically configured to filter out the first item corresponding to the target query and meeting the first click-through rate requirement from the multiple query-item pairs to obtain a first set; and / or, to filter out the first query corresponding to the target item and meeting the second click-through rate requirement from the multiple query-item pairs to obtain a second set.

[0172] In one embodiment, the apparatus further includes: an attribute information unit 75 configured to obtain query attribute information corresponding to the target query and / or item attribute information corresponding to the target item based on a preset knowledge base;

[0173] The prompt construction unit 72 is specifically configured to construct target prompt information based on the target query-target item pair, the neighbor set of the target query-target item pair, the query attribute information and / or the item attribute information.

[0174] In one embodiment, the prompt construction unit 72 is specifically configured to construct first target prompt information according to the neighbor set of the target query-target item pair, the first target prompt information is used to query the large language model: whether the first set and the second set in the neighbor set are related; query attribute information and item attribute information according to the neighbor set of the target query-target item pair, and construct second target prompt information, the second target prompt information is used to query the large language model: given the first set and the second set, whether the query attribute information and the item attribute information are related; query attribute information and item attribute information according to the target query-target item pair and the neighbor set of the target query-target item pair, and construct third target prompt information, the third target prompt information is used to query the large language model: given the first set, the second set, the query attribute information and the item attribute information, whether the target query and the target item are related;

[0175] The relevance scoring unit 73 is specifically configured to: input the first target prompt information into the large language model to obtain a first relevance score of the target query-target item pair output by the large language model; input the second target prompt information into the large language model to obtain a second relevance score of the target query-target item pair output by the large language model; input the third target prompt information into the large language model to obtain a third relevance score of the target query-target item pair output by the large language model.

[0176] In one embodiment, the device also includes: a model training unit 76, which is configured to determine a first loss function based on the first relevance score and the relevance label of the target query-target item pair; determine a second loss function based on the second relevance score and the relevance label of the target query-target item pair; determine a third loss function based on the third relevance score and the relevance label of the target query-target item pair; determine an aggregated relevance score based on the first relevance score, the second relevance score, and the third relevance score, and determine a fourth loss function based on the aggregated relevance score and the relevance label of the target query-target item pair; train a large language model based on the first loss function, the second loss function, the third loss function, and the fourth loss function.

[0177] In one embodiment, the model training unit 76 is configured to determine the aggregated relevance score based on the first relevance score, the second relevance score and the third relevance score, and is specifically configured to: determine an objective function including learnable parameters, the learnable parameters are used to adjust during the training process of the large language model, and the value of the objective function corresponding to each target prompt information is related to the amount of information contained in each target prompt information; use the value of the objective function corresponding to each target prompt information as a weight, aggregate the first relevance score, the second relevance score and the third relevance score to obtain an aggregated relevance score.

[0178] In one embodiment, the objective function is an exponential kernel function, and the learnable parameters are located at the exponential position of the exponential kernel function.

[0179] In one embodiment, the data acquisition unit 71 is specifically configured to acquire a neighbor set of a target query-target item pair using updated user search behavior data in response to an update of the historical behavior data.

[0180] The present specification also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the following Figure 1 and Figure 6 Describe the method.

[0181] The embodiment of the present specification also provides a computing device, including a memory and a processor, wherein the memory stores an executable code, and when the processor executes the executable code, the following is implemented: Figure 1 and Figure 6 Describe the method.

[0182] The embodiments of the present specification also provide a computer program product, including a computer program / instruction, which is executed by a processor to implement the following Figure 1 and Figure 6 Describe the steps of the method.

[0183] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the multiple embodiments disclosed in this specification can be implemented by hardware, software, firmware, or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0184] In some cases, the actions or steps described in the claims may be performed in a different order than in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0185] The specific implementation methods described above further illustrate in detail the purposes, technical solutions and beneficial effects of the multiple embodiments disclosed in this specification. It should be understood that the above description is only the specific implementation methods of the multiple embodiments disclosed in this specification, and is not used to limit the protection scope of the multiple embodiments disclosed in this specification. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the multiple embodiments disclosed in this specification should be included in the protection scope of the multiple embodiments disclosed in this specification.

Claims

1. A method for determining the relevance of a query-item pair, the method comprising: Based on historical behavior data of users of the target search engine, a neighbor set of a target query-target item pair is obtained, wherein the neighbor set includes at least one of the following sets: a first set of first items corresponding to the target query, and a second set of first queries corresponding to the target item; constructing target prompt information based on the target query-target item pair and the neighbor set; The target prompt information is input into a large language model to obtain a relevance score of the target query-target item pair output by the large language model.

2. The method according to claim 1, wherein: The target query is a query input by a target user, the target item is an item in a candidate item set corresponding to the target query, and the relevance score of the target query-target item pair is used to determine the priority of the target item when it is displayed to the target user.

3. The method according to claim 1, wherein: After inputting the target prompt information into a large language model and obtaining a relevance score of the target query-target item pair output by the large language model, the method further includes: The relevance score of the target query-target item pair is stored in an offline database, and the offline database is used for offline query relevance scores.

4. The method according to claim 3, wherein: After storing the relevance score of the target query-target item pair in an offline database, the method further includes: In response to receiving a user query request, searching the offline database for a target query in the user query request; In response to the target query existing in the offline database, a relevance score of a target query-item pair corresponding to the target query is obtained from the offline database.

5. The method according to claim 1, wherein: The step of obtaining a neighbor set of a target query-target item pair based on historical behavior data of users of the target search engine includes: Based on the plurality of query-item pairs in the historical behavior data of users of the target search engine, filter out target query-target item pairs that meet the exposure condition and / or click-through rate condition; A neighbor set of the target query-target item pair is determined from the plurality of query-item pairs.

6. The method according to claim 5, wherein: The step of determining a neighbor set of the target query-target item pair from the plurality of query-item pairs includes: Filtering the first items that meet the first click-through rate requirement and correspond to the target query from the plurality of query-item pairs to obtain the first set; and / or, The first queries corresponding to the target items and meeting the second click rate requirement are screened out from the plurality of query-item pairs to obtain the second set.

7. The method according to claim 1, wherein: After inputting the target prompt information into a large language model and obtaining a relevance score of the target query-target item pair output by the large language model, the method further includes: The large language model is trained based on the relevance score and the obtained relevance label of the target query-target item pair.

8. The method according to claim 1, wherein: The method further comprises: Based on a preset knowledge base, obtaining query attribute information corresponding to the target query and / or project attribute information corresponding to the target project; The constructing target prompt information based on the target query-target item pair and the neighbor set of the target query-target item pair includes: Target prompt information is constructed based on the target query-target item pair, a neighbor set of the target query-target item pair, the query attribute information and / or the item attribute information.

9. The method according to claim 8, wherein: The constructing target prompt information based on the target query-target item pair, the neighbor set of the target query-target item pair, the query attribute information and / or the item attribute information includes: Constructing first target prompt information according to the neighbor set of the target query-target item pair, wherein the first target prompt information is used to query the large language model: whether the first set and the second set in the neighbor set are related; Constructing second target prompt information according to the neighbor set of the target query-target item pair, the query attribute information and the item attribute information, wherein the second target prompt information is used to inquire the large language model: given the first set and the second set, whether the query attribute information and the item attribute information are related; Constructing third target prompt information according to the target query-target item pair, the neighbor set of the target query-target item pair, the query attribute information and the item attribute information, wherein the third target prompt information is used to inquire the large language model: given the first set, the second set, the query attribute information and the item attribute information, whether the target query and the target item are related; The step of inputting the target prompt information into a large language model to obtain a relevance score of the target query-target item pair output by the large language model includes: Inputting the first target prompt information into a large language model to obtain a first relevance score of the target query-target item pair output by the large language model; Inputting the second target prompt information into a large language model to obtain a second relevance score of the target query-target item pair output by the large language model; The third target prompt information is input into a large language model to obtain a third relevance score of the target query-target item pair output by the large language model.

10. The method according to claim 9, wherein: The method further comprises: determining a first loss function based on the first relevance score and the relevance label of the target query-target item pair; determining a second loss function based on the second relevance score and the relevance label of the target query-target item pair; determining a third loss function based on the third relevance score and the relevance label of the target query-target item pair; determining an aggregated relevance score based on the first relevance score, the second relevance score, and the third relevance score, and determining a fourth loss function based on the aggregated relevance score and a relevance label of the target query-target item pair; The large language model is trained according to the first loss function, the second loss function, the third loss function and the fourth loss function.

11. The method according to claim 10, wherein: The determining an aggregate relevance score according to the first relevance score, the second relevance score, and the third relevance score comprises: Determining an objective function including learnable parameters, wherein the learnable parameters are used to adjust during the training of the large language model, and the value of the objective function corresponding to each target prompt information is related to the amount of information contained in the each target prompt information; The first relevance score, the second relevance score, and the third relevance score are aggregated using the values ​​of the objective function corresponding to the respective objective prompt information as weights to obtain the aggregated relevance score.

12. The method according to claim 11, wherein: The objective function is an exponential kernel function, and the learnable parameter is located at an exponential position of the exponential kernel function.

13. The method according to any one of claims 1 to 12, wherein: The step of obtaining a neighbor set of a target query-target item pair based on historical behavior data of users of the target search engine includes: In response to the historical behavior data being updated, a neighbor set of the target query-target item pair is obtained using the updated user search behavior data.

14. An apparatus for determining the relevance of a query-item pair, the apparatus comprising: A data acquisition unit is configured to acquire a neighbor set of a target query-target item pair based on historical behavior data of a user of a target search engine, wherein the neighbor set includes at least one of the following sets: a first set of first items corresponding to the target query, and a second set of first queries corresponding to the target item; a prompt construction unit configured to construct target prompt information based on the target query-target item pair and the neighbor set; The relevance scoring unit is configured to input the target prompt information into a large language model to obtain a relevance score of the target query-target item pair output by the large language model.

15. A computing device comprising a memory and a processor, wherein: The memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 13 is implemented.

Images