Search quality optimization method and apparatus, computer device, storage medium, and computer program product

Abstract

The application relates to a search quality optimization method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: obtaining at least one target search record, the target search record comprising a target search word and search result content features, the correlation degree between the search result content features and the target search word satisfying a first correlation condition, the correlation degree being evaluated by a preset correlation scoring model; selecting at least one to-be-reviewed search record satisfying a second correlation condition from the target search records; for any to-be-reviewed search record, performing correlation review on the target search word and the search result content features to obtain a correlation review result; in the case that the correlation review result does not satisfy the first correlation condition, constructing a negative sample based on the target search record and the correlation review result, the negative sample being used for optimizing the preset correlation scoring model. The method can improve the search quality optimization effect.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a search quality optimization method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] With the explosive growth of internet information, information search technology has become a core tool for people to accurately locate the content they need from massive amounts of data. The degree of matching between search results and user query intent directly affects search efficiency and user experience; the higher the degree of matching, the more efficient the user is in obtaining information, and the better the overall search experience.

[0003] In related technologies, the degree of matching between search results and users' true query intent can be inferred by statistically analyzing metrics such as click-through rate and browsing time of search results.

[0004] However, low-quality content such as clickbait headlines can easily induce users to make invalid clicks, causing click-through rates and other metrics to deviate from their true nature and fail to accurately reflect the degree of match between search results and genuine query intent. This reduces the reliability of search quality assessments based on these metrics, making it difficult to effectively improve search quality. Summary of the Invention

[0005] Therefore, it is necessary to provide a search quality optimization method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the search quality optimization effect in response to the above-mentioned technical problems.

[0006] Firstly, this application provides a method for optimizing search quality. The method includes:

[0007] Obtain at least one target search record, which includes target search terms and search result content features, and the relevance between the search result content features and the target search terms satisfies the first relevance condition. The relevance is evaluated by a preset relevance scoring model.

[0008] From each target search record, select at least one search record that meets the second relevance condition and needs to be reviewed;

[0009] For any search record to be reviewed, the relevance between the target search term in the search record and the content features of the search results is reviewed to obtain the relevance review result; the evaluation method of relevance is different from the evaluation method of relevance review.

[0010] If the relevance review results do not meet the first relevance condition, negative samples are constructed based on the search records to be reviewed and the relevance review results. These negative samples are used to optimize the preset relevance scoring model.

[0011] Secondly, this application also provides a search quality optimization apparatus. The apparatus includes:

[0012] The acquisition module is used to acquire at least one target search record. The target search record includes target search terms and search result content features. The relevance between the search result content features and the target search terms satisfies the first relevance condition. The relevance is evaluated by a preset relevance scoring model.

[0013] The filtering module is used to filter out at least one search record to be reviewed from each target search record that meets the second relevance condition;

[0014] The relevance verification module is used to verify the relevance between the target search terms and the content features of the search results in any search record to be verified, and to obtain the relevance verification result; the evaluation method of relevance is different from that of relevance verification.

[0015] The negative sample construction module is used to construct negative samples based on the search records to be reviewed and the relevance review results when the relevance review results do not meet the first relevance condition. The negative samples are used to optimize the preset relevance scoring model.

[0016] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0017] At least one target search record is obtained, which includes target search terms and search result content features. The relevance between the search result content features and the target search terms satisfies the first relevance condition, and the relevance is evaluated by a preset relevance scoring model. From each target search record, at least one search record to be reviewed is selected that satisfies the second relevance condition. For any search record to be reviewed, the relevance between the target search terms and search result content features in the search record to be reviewed is reviewed to obtain a relevance review result. The evaluation method for relevance is different from the evaluation method for relevance review. If the relevance review result does not satisfy the first relevance condition, a negative sample is constructed based on the search record to be reviewed and the relevance review result. The negative sample is used to optimize the preset relevance scoring model.

[0018] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0019] At least one target search record is obtained, which includes target search terms and search result content features. The relevance between the search result content features and the target search terms satisfies the first relevance condition, and the relevance is evaluated by a preset relevance scoring model. From each target search record, at least one search record to be reviewed is selected that satisfies the second relevance condition. For any search record to be reviewed, the relevance between the target search terms and search result content features in the search record to be reviewed is reviewed to obtain a relevance review result. The evaluation method for relevance is different from the evaluation method for relevance review. If the relevance review result does not satisfy the first relevance condition, a negative sample is constructed based on the search record to be reviewed and the relevance review result. The negative sample is used to optimize the preset relevance scoring model.

[0020] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0021] At least one target search record is obtained, which includes target search terms and search result content features. The relevance between the search result content features and the target search terms satisfies the first relevance condition, and the relevance is evaluated by a preset relevance scoring model. From each target search record, at least one search record to be reviewed is selected that satisfies the second relevance condition. For any search record to be reviewed, the relevance between the target search terms and search result content features in the search record to be reviewed is reviewed to obtain a relevance review result. The evaluation method for relevance is different from the evaluation method for relevance review. If the relevance review result does not satisfy the first relevance condition, a negative sample is constructed based on the search record to be reviewed and the relevance review result. The negative sample is used to optimize the preset relevance scoring model.

[0022] The aforementioned search quality optimization method, apparatus, computer equipment, storage medium, and computer program product first sample and collect search records generated during the search process to obtain at least one target search record. This target search record contains target search terms and search result content features. During the search process, a preset relevance scoring model has been used to evaluate the relevance between the search result content features and the target search terms in real time. The real-time evaluation results show that the relevance between the two meets the first relevance condition. Subsequently, at least one search record to be reviewed that meets the second relevance condition is selected from each target search record. For any selected search record to be reviewed, the relevance between the target search terms and the search result content features is reviewed to complete the review of the real-time evaluation result of the preset relevance scoring model. If the relevance review result obtained does not meet the first relevance condition, it indicates that there is a deviation between the real-time evaluation result of the online model and the review result. In this case, a negative sample is constructed based on the target search record and the relevance review result, and this negative sample is used for the optimization training of the preset relevance scoring model.

[0023] The aforementioned search quality optimization method can capture typical cases where the preset relevance scoring model fails to perform accurately, without interfering with the normal operation of the online search service, and transform them into negative samples required for the optimization of the relevance scoring model. With the continuous accumulation of negative samples and the iterative optimization of the preset relevance scoring model, the accuracy of the model gradually improves, and its evaluation of the matching degree between search results and query intent becomes more reliable, thereby effectively promoting a substantial improvement in the online search quality optimization effect. Compared with quality assessment methods that rely on easily distorted indicators such as click-through rate, the search terms used for relevance verification in this application can directly reflect the user's true query intent, and the content features of the search results can accurately and comprehensively reflect the content information of the search results. Relevance analysis of the two can effectively reduce the impact of external interference factors on the quality assessment results, thereby improving the reliability of search quality assessment. Attached Figure Description

[0024] Figure 1 This is a diagram illustrating the application environment of a search quality optimization method in one embodiment;

[0025] Figure 2 This is a schematic diagram illustrating a scenario where search results are displayed on a comprehensive search webpage, as shown in one embodiment.

[0026] Figure 3 This is a flowchart illustrating a search quality optimization method in one embodiment;

[0027] Figure 4 This is a flowchart illustrating the process of filtering search records to be reviewed from target search records in one embodiment.

[0028] Figure 5 This is a flowchart illustrating the process of filtering search records to be reviewed from target search records in another embodiment;

[0029] Figure 6 This is a schematic diagram of the process for triggering an alarm in one embodiment;

[0030] Figure 7 This is a schematic diagram illustrating a scenario where search results are displayed in sorted order, as shown in one embodiment.

[0031] Figure 8 This is a flowchart illustrating the search quality optimization method in another embodiment;

[0032] Figure 9 This is a structural block diagram of a search quality optimization device in one embodiment;

[0033] Figure 10 This is an internal structural diagram of a computer device in one embodiment;

[0034] Figure 11 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0036] The search quality optimization method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another server.

[0037] During the search process, users can enter search terms through the search interface displayed on terminal 102. In response to the search term input operation, terminal 102 obtains the search term entered by the user, retrieves at least one search result matching the search term from the data source, and displays each search result to the user through the search results page displayed on terminal 102.

[0038] Taking the search process as an example, which involves the interaction of terminal 102, server 104, and data source, the process of obtaining at least one search result matching the search term from the data source includes: terminal 102 sending the search term to server 104, server 104 further sending the search term to the data source, which stores a large amount of candidate data. The data source uses the search term as a query condition and, through similarity matching or data source-defined filtering conditions, searches for candidate search results that match the term from the candidate data, and returns the candidate search results to server 104. After receiving the candidate search results, server 104 calls a preset relevance scoring model to evaluate the relevance between the content features of the candidate search results and the search term, and determines the candidate search results that meet the first relevance condition as the final search results. Subsequently, server 104 sends the final search results to terminal 102.

[0039] Optionally, the method of obtaining at least one search result matching the search term from the data source may include: the terminal 102 may send the search term to the data source, which stores a large amount of candidate data. The data source uses the search term as a query condition, searches for a matching search result from the candidate data, and returns the search result to the terminal 102. During the search result query process, the data source may invoke a preset relevance scoring model to evaluate the relevance between the content features of the candidate data and the search term, and determine the candidate data whose relevance meets the first relevance condition as the search result.

[0040] Optionally, the method of obtaining at least one search result matching the search term from the data source may further include: the terminal 102 may send the search term to the data source, which stores a large amount of alternative data. The data source uses the search term as a query condition, and through similarity matching or data source-defined filtering conditions, searches for alternative search results that match the search term from the alternative data, and returns the alternative search results to the terminal 102. After receiving the alternative search results, the terminal 102 calls a preset relevance scoring model to evaluate the relevance between the content features of the alternative search results and the search term, and determines the alternative search results whose relevance meets the first relevance condition as the final search results.

[0041] Optionally, in the search results process, in addition to evaluating relevance, other filtering conditions can be combined for comprehensive filtering to obtain search results that meet multiple filtering conditions. These other filtering conditions may include at least one of the following: timeliness, geographic location, source authority, content type, and compliance. This embodiment does not impose any limitations on these conditions.

[0042] Without interfering with the normal operation of the online search service, search quality optimization methods can be triggered in real time or at regular intervals. By reviewing the search results, typical cases of inaccurate judgments by the preset relevance scoring model can be captured and transformed into negative samples required for the optimization of the relevance scoring model. By using negative samples to optimize the relevance scoring model, continuous optimization of search quality can be achieved.

[0043] Taking the search quality optimization method implemented by server 104 as an example, server 104 can sample at least a portion of search records from the search records stored in the data storage system or from the search logs stored in terminal 102 in real time, periodically, or triggered, as target search records, and further filter out search records to be reviewed from the target search records. For any search record to be reviewed, server 104 performs a relevance review on the target search terms and search result content features in the search record to be reviewed, obtaining a relevance review result. If the relevance review result does not meet the first relevance condition, server 104 further constructs negative samples based on the search record to be reviewed and the relevance review result.

[0044] After constructing negative samples, server 104 can use the constructed negative samples in real time, periodically, or triggered to optimize the preset relevance scoring model, thereby achieving continuous optimization of search quality.

[0045] After constructing negative samples, server 104 can also perform statistical analysis based on the negative samples, and perform one or more operations such as generating search quality reports and triggering quality alarms according to the results of the statistical analysis.

[0046] Optionally, when the search process is implemented through the interaction of terminal 102, server 104 and data source, before obtaining the target search record, server 104 may extract the search result content features from one of the search results during the search process or after determining the final search result, and then store the search result content features of the search result and the corresponding search terms as a search record in the data storage system.

[0047] Optionally, when the search process is implemented through interaction between terminal 102 and the data source, before obtaining the target search record, terminal 102 may extract the search result content features from each search result during the search process or after determining the final search result, and store the search result content features of each search result and the corresponding search terms as a search record in the search log. Terminal 102 may also send the search records in the search log to server 104 in real time, periodically, or triggered. Server 104 may store the received search records in the data storage system.

[0048] Optionally, if the data processing capability of terminal 102 meets the usage requirements, the search quality optimization method can also be implemented by terminal 102.

[0049] Alternatively, the search quality optimization method can also be implemented by a system including terminal 102 and server 104 through the interaction between terminal 102 and server 104.

[0050] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0051] In one possible implementation, search quality optimization methods can be applied to streaming media search scenarios, such as when a user searches for relevant audio and video content using the search function of a streaming media platform on a computer. During the streaming media search process, a pre-set relevance scoring model assesses the relevance between the user's search terms and the content features of the search results. However, because streaming media content contains multimodal information such as video, audio, and text, and the user's search intent is often implicit in the multiple dimensions of the content, the accuracy of the relevance assessment between the search results and the search terms decreases. Therefore, continuous optimization of search quality is necessary.

[0052] Specifically, during streaming media search, the following can be extracted as search result content features: video title, video tags, video cover image, video keyframe image, text extracted from the video using OCR (Optical Character Recognition) technology, and text summaries generated by converting the audio content of the video using ASR (Automatic Speech Recognition) technology. These multimodal search result content features are stored together with the user-input search terms to form a complete search record.

[0053] If a search record is sampled and selected as a target search record, and meets the second relevance condition, thus being filtered as a search record to be reviewed, the relevance between the search term and the aforementioned search result content features can be re-verified to obtain a relevance review result. If the relevance review result indicates that the relevance between the two does not meet the first relevance condition, then a negative sample is constructed based on the search record to be reviewed and its corresponding relevance review result. Subsequently, based on one or more of the constructed negative samples, the preset relevance scoring model can be iteratively optimized to improve its accuracy in relevance assessment in streaming media search scenarios.

[0054] In one possible implementation, search quality optimization methods can be applied to short video search scenarios, such as when users search for relevant short video content using the search function of a short video app on their mobile phones. During the short video search process, a pre-set relevance scoring model assesses the relevance between the user's search terms and the search results' content features. However, because short video content integrates multi-dimensional information such as visuals, audio, subtitles, and hashtags, and often requires a comprehensive understanding of multimodal content to accurately match the user's search intent, the accuracy of the relevance assessment between search results and search terms decreases. Therefore, continuous optimization of search quality is necessary.

[0055] Specifically, during short video searches, the following content can be extracted as search result features: short video title, short video hashtags, video cover image, video keyframe images, accompanying text descriptions, text overlaid within the video extracted using OCR technology, and text summaries generated from the audio content of the video using ASR technology. These multimodal search result features are stored together with the user-input search terms to form a complete search history.

[0056] If a search record is sampled and selected as a target search record, and meets the second relevance condition, thus being filtered as a search record to be reviewed, the relevance between the search term and the aforementioned search result content features can be re-verified to obtain a relevance review result. If the relevance review result indicates that the relevance between the two does not meet the first relevance condition, a negative sample is constructed based on the search record to be reviewed and its corresponding relevance review result. Subsequently, based on one or more of the constructed negative samples, the preset relevance scoring model can be iteratively optimized to improve its accuracy in relevance assessment in short video search scenarios.

[0057] In one possible implementation, search quality optimization methods can be applied to mixed-index search scenarios. Mixed-index search refers to a search method that presents users with search results of different content types or from different data sources, arranged according to a unified sorting strategy. During mixed-index search, a pre-set relevance scoring model assesses the relevance of the user's search terms to the content characteristics of each type of search result. However, because mixed-index search results include various content types such as videos, images and text, accounts, and services, and these contents are not arranged by type but rather by a comprehensive sorting based on relevance, timeliness, and other multi-dimensional indicators, the significant differences in characteristics between different content types lead to a decrease in the accuracy of the relevance assessment between search results and search terms. For example, ... Figure 2As shown, a user opens a comprehensive search webpage 201 in a computer browser and enters the search term "Game A competition finals held in location B" in the search box 202. The search results page displays different types of content arranged according to their overall relevance, appearing in the following order: news and information related to the Game A competition finals being held in location B 203 (text and images); news and information related to teams from location B entering the Game A competition finals 204; a press conference video of the Game A competition finals being held in location B 205 (video); a comparative analysis video of the participating teams in the Game A competition finals 206 (video); and Weibo posts from well-known game bloggers who traveled to location B to watch the competition 207 (social media). While news and information related to teams from location B entering the Game A competition finals 204 partially matches the search term literally, its core content refers to the event of teams from location B qualifying for the finals, not the event itself being held in location B. This indicates a relevance drift problem, and the preset relevance scoring model misjudges the relevance of this search result, causing the output relevance to meet the preset first relevance adjustment and thus being identified as one of the search results. Therefore, for mixed search scenarios, continuous optimization of search quality is necessary.

[0058] Specifically, during the mixed search process, corresponding content features can be extracted as search result content features based on the characteristics of different types of search results: For video search results, video titles, video tags, video cover images, video keyframe images, text extracted from the video using OCR technology, and text summaries generated by converting audio content from the video using ASR technology can be extracted; for image and text search results, article titles, text summaries, and accompanying image features can be extracted; for account search results, account names, account descriptions, authentication information, and historical content tags can be extracted; for service search results, service names, service descriptions, service categories, and user review tags can be extracted. These multi-type, multi-modal search result content features are stored together with the user-input search terms to form a complete search record.

[0059] If a search record is sampled and selected as a target search record, and meets the second relevance condition, thus being filtered as a search record to be reviewed, the relevance between the search term and the aforementioned search result content features can be re-verified to obtain a relevance review result. If the relevance review result indicates that the relevance between the two does not meet the first relevance condition, a negative sample is constructed based on the search record to be reviewed and its corresponding relevance review result. Subsequently, based on one or more constructed negative samples, the preset relevance scoring model can be iteratively optimized to improve its accuracy in relevance assessment under mixed search scenarios.

[0060] In one possible implementation, the search quality optimization method can be applied to voice search scenarios such as smart speakers and voice assistants. In voice interaction devices like smart speakers, search terms are presented in voice form. Users issue commands via natural language, and the search system uses a pre-set relevance scoring model to quickly locate search results that meet the preset first relevance condition from a massive amount of content. By extracting search records for relevance verification and constructing negative samples, and then iteratively optimizing the pre-set relevance scoring model based on the negative samples, the accuracy of relevance assessment in voice search scenarios can be effectively improved, thereby achieving the goal of optimizing voice search quality.

[0061] In one possible implementation, search quality optimization methods can be applied to autonomous driving scenarios. In autonomous driving scenarios, the search task mainly involves the rapid matching and retrieval of historical trajectory data, environmental perception information, and cloud-based image resources. For example, during operation, a vehicle needs to search a cloud database for historical trajectories similar to the current road environment to assist in path planning and behavioral decisions; or it can collect real-time images through onboard cameras and search for matching results containing specific objects (such as traffic signs, obstacles, and pedestrians) in an encrypted image library. This search process can leverage a pre-defined relevance scoring model to evaluate the relevance between image features, trajectory features, and query conditions, and filter out search results that meet the pre-defined first relevance condition. By extracting search records for relevance verification, constructing negative samples, and iteratively optimizing the pre-defined relevance scoring model based on these negative samples, the accuracy of image matching and trajectory retrieval in autonomous driving scenarios can be effectively improved, thereby providing more reliable search support for the decision-making system and enhancing the safety and reliability of autonomous driving.

[0062] In one embodiment, such as Figure 3 As shown, a search quality optimization method is provided, which is applied to... Figure 1 Taking server 104 as an example, the explanation includes steps 302 to 308, where:

[0063] Step 302: Obtain at least one target search record. The target search record includes the target search term and the content features of the search results. The relevance between the content features of the search results and the target search term satisfies the first relevance condition. The relevance is evaluated by a preset relevance scoring model.

[0064] In this context, a search record can refer to a data unit containing the content characteristics of a search result, which includes a search term and a corresponding search result.

[0065] In response to a search term input operation on the search interface, the terminal obtains the user-input search term and displays at least one search result corresponding to that search term on the search results page. Subsequently, the terminal or server can combine the search term and one of the corresponding search results into a single search record. If multiple search results are found based on a single search term, the search term and each search result can be combined to form multiple distinct search records.

[0066] After triggering search quality optimization, the server can sample at least a portion of the pre-assembled and stored search records as target search records. A target search record can refer to one of the search records obtained through sampling from a set of search records.

[0067] The sampling methods can include at least one of random sampling, scenario-based sampling, and traffic entry point sampling. Scenario-based sampling refers to setting corresponding sampling ratios for different pre-defined search scenarios, such as general search, vertical search, private domain search, and community search, and then extracting target search records from the corresponding search records for each search scenario according to the sampling ratio. Scenario-based sampling can increase the scope of search scenarios covered by the target search records and improve their representativeness. Traffic entry point sampling refers to setting corresponding sampling ratios for different traffic entry points, such as the general search box, vertical search tabs, and in-app search modules, and then extracting target search records from the corresponding search records for each traffic entry point according to the sampling ratio. Traffic entry point sampling helps cover search operations under different user access paths and improves the representativeness of the target search records.

[0068] Search terms refer to the query text entered by the user during the search process. Target search terms refer to the search terms contained in the target search records. Search results refer to the specific content items returned after a user initiates a search, such as web page links, video cards, product entries, etc., which belong to structured data. Search result content features refer to information used to characterize the attributes of the search result content, which can take various forms such as text, image features, and video cover features.

[0069] Taking video search results as an example, the content features of search results may include at least one of textual features and visual features. Textual features may include at least one of the following: video title, video tags, text extracted from the video via OCR, and text summaries generated by converting the audio content in the video using ASR technology. Visual features may include at least one of the following: video cover image, video keyframe images, etc.

[0070] Relevance refers to the degree of association between the content features of search results and the target search terms. Depending on the application scenario and evaluation granularity, relevance can have various representations.

[0071] Optionally, the relevance can be a relevance score expressed as a continuous numerical value, such as 0.85, 0.63, etc. The higher the value, the stronger the correlation, which is suitable for search scenarios that require fine-grained ranking.

[0072] Optionally, the relevance can be expressed as a discrete level of relevance, such as "highly correlated", "generally correlated", "lowly correlated", "no correlation", etc., which is suitable for qualitative analysis scenarios.

[0073] Optionally, the relevance can be a binary judgment label represented by a binary classification result, such as "relevant" and "irrelevant", which is suitable for preliminary screening or simple judgment scenarios.

[0074] The first relevance criterion can refer to a preset threshold or logical rule used to filter content items that meet specific relevance requirements as search results. For example, assuming the first relevance criterion is "relevance score ≥ 0.7", during the user's search process, a preset relevance scoring model can be used to evaluate the relevance between candidate content items and search terms, allowing only candidate content items with a relevance score greater than or equal to 0.7 to be included as the final search results.

[0075] A relevance scoring model can refer to a pre-trained machine learning model used to evaluate the relevance between the content features of search results and search terms.

[0076] Optionally, the preset relevance scoring model can be implemented using a language model. This language model can be a small-scale or large-scale model, depending on factors such as computing resources and response speed requirements in the actual application scenario; this embodiment does not impose any restrictions. Compared to classification models with structures such as BERT (Bidirectional Encoder Representations from Transformers), language models possess stronger semantic understanding capabilities and can more effectively process search results containing multimodal information such as text and video covers. Simultaneously, language models exhibit better generalization performance, learning superior inference logic when faced with complex and varied negative samples, thereby continuously improving their accuracy in evaluating the relevance between search results and search terms.

[0077] Specifically, in the process of responding to search requests in real time, the terminal, server, or data source can use a preset relevance scoring model to evaluate the relevance between the search result features of the candidate content items and the user's input search terms. Only content items whose relevance meets the preset first relevance condition will be displayed to the user as search results, and corresponding search records will be formed based on this.

[0078] The server can initiate the search quality optimization process in real time, on a schedule, or triggered by a specific event. Once initiated, the search quality optimization process first samples at least a portion of the search records from the data storage system or from the search logs stored on the terminal, using these samples as the target search records for this search quality optimization.

[0079] Step 304: Select at least one search record to be reviewed from each target search record that meets the second relevance condition.

[0080] The second relevance condition can refer to a preset judgment threshold or logical rule, which is used to filter target search records that meet specific requirements as search records to be reviewed.

[0081] Optionally, the second relevance condition can be set for the relevance obtained from the evaluation of a preset relevance scoring model. In this case, the screening dimensions of the second relevance condition are different from those of the first relevance condition. The first relevance condition is used to directly determine the numerical value of the relevance, for example, setting the relevance score to be greater than 0.8; the second relevance condition is used to determine the confidence level when the model outputs the relevance, for example, setting the confidence level to be less than 0.85, which is calculated by the preset relevance scoring model while evaluating the relevance.

[0082] Optionally, the second relevance condition can be set for at least one of the relevance and confidence scores obtained from the large-scale relevance analysis model. After obtaining the target search records, the relevance between the search results content features in each target search record and the target search terms can be preliminarily verified using the large-scale relevance analysis model. If at least one of the relevance and confidence scores output by the large-scale relevance analysis model satisfies the second relevance condition, then the target search record is identified as a search record to be reviewed. For example, for a specific target search record, the relevance score output by the preset relevance scoring model is 0.82, which meets the requirement of ≥0.8 in the first preset relevance condition. Therefore, it is used as a search result and forms a search record. The relevance analysis model outputs a relevance score of 0.75 for the same target search record, which meets the requirement of ≤0.8 in the second preset relevance condition. Therefore, this target search record can be considered as a search record to be reviewed. Alternatively, the relevance analysis model outputs a relevance score of 0.81 and a confidence score of 0.6 for the same target search record. Although the relevance score of 0.81 does not meet the requirement of ≤0.8 in the second preset relevance condition, the confidence score of 0.6 meets the requirement of ≤0.85 in the second preset relevance condition. Therefore, this target search record can also be considered as a search record to be reviewed.

[0083] Search records pending review can refer to search records that have been further filtered from the target search records and whose relevance needs to be reviewed.

[0084] Specifically, after obtaining the target search records, the search records that need to be reviewed can be further filtered out based on the second relevance condition.

[0085] Step 306: For any search record to be reviewed, the relevance between the target search term and the search result content features in the search record to be reviewed is reviewed to obtain the relevance review result; the evaluation method of relevance is different from the evaluation method of relevance review.

[0086] Relevance review refers to the process of re-evaluating the target search terms and search results in the search records to be reviewed, using a method different from the preset relevance scoring model. Relevance review can be conducted using more accurate methods such as manual evaluation or a large-scale relevance arbitration model.

[0087] Compared to the method of having a professional evaluation team manually evaluate the target search records directly after sampling, screening the target search records based on the second relevance condition can significantly reduce the sample size that requires manual intervention, thereby greatly shortening the execution cycle of search quality optimization and enabling more timely detection and response to sudden fluctuations in online search quality.

[0088] Compared to manual evaluation, using a relevance arbitration model can further improve processing efficiency, shorten the execution cycle of search quality optimization, and thus enhance the timeliness of response to fluctuations in online search quality.

[0089] The relevance review result refers to the analytical conclusion formed after performing a relevance review on the target search terms and search result content features in the search records to be reviewed. The relevance review result can include at least one of the following: the relevance determined by the relevance review, and thought chain data. For example, the matching between search result content features and target search terms can be analyzed item by item to identify key matching points or mismatches, and the logical explanation for judging these key matching points or mismatches can be recorded. By recording the complete reasoning process and forming thought chain data, the relevance review result can provide traceable judgment reasons for subsequent negative sample construction, and can also enable the pre-set relevance scoring model to be trained based on richer feedback information.

[0090] Specifically, for each selected search record to be reviewed, a relevance review is conducted on the target search terms and search result content features, which differs from the preset relevance scoring model. During the relevance review process, the evaluators or the relevance arbitration model, through a more comprehensive and in-depth understanding of the query intent of the search terms and the content essence of the search result content features, gradually reason and ultimately determine whether the two match, thus forming the relevance review result.

[0091] Step 308: If the relevance review result does not meet the first relevance condition, construct a negative sample based on the search record to be reviewed and the relevance review result. The negative sample is used to optimize the preset relevance scoring model.

[0092] Negative samples can refer to search record samples that, when verified, do not actually meet the first relevance condition, but are misjudged as meeting the first relevance condition by the preset relevance scoring model.

[0093] Specifically, after obtaining the relevance verification results, it is determined whether the obtained relevance verification results meet the first relevance condition. If the relevance verification results do not meet the first relevance condition, it indicates that the preset relevance scoring model has a bias in judging the relevance between the content features of the search results and the search terms in the search record to be verified. In this case, the search record to be verified and the relevance verification results are jointly constructed into a negative sample. This negative sample will be used for subsequent optimization training of the preset relevance scoring model, so that the model can better identify such cases after updates and iterations, and avoid similar judgment errors from occurring again.

[0094] In the aforementioned search quality optimization method, firstly, search records generated during the search process are sampled to obtain at least one target search record. This target search record contains target search terms and search result content features. During the search process, a preset relevance scoring model has been used to evaluate the relevance between the search result content features and the target search terms in real time. The real-time evaluation results show that the relevance between the two meets the first relevance condition. Subsequently, at least one search record that meets the second relevance condition is selected from each target search record for review. For any selected search record to be reviewed, the relevance between the target search terms and search result content features is reviewed to complete the review of the real-time evaluation results of the preset relevance scoring model. If the relevance review result does not meet the first relevance condition, it indicates that there is a deviation between the real-time evaluation results of the online model and the review results. In this case, a negative sample is constructed based on the target search record and the relevance review result, and this negative sample is used for the optimization training of the preset relevance scoring model.

[0095] The aforementioned search quality optimization method can capture typical cases where the preset relevance scoring model fails to perform accurately, without interfering with the normal operation of the online search service, and transform them into negative samples required for the optimization of the relevance scoring model. With the continuous accumulation of negative samples and the iterative optimization of the preset relevance scoring model, the accuracy of the model gradually improves, and its evaluation of the matching degree between search results and query intent becomes more reliable, thereby effectively promoting a substantial improvement in the online search quality optimization effect. Compared with quality assessment methods that rely on easily distorted indicators such as click-through rate, the search terms used for relevance verification in this application can directly reflect the user's true query intent, and the content features of the search results can accurately and comprehensively reflect the content information of the search results. Relevance analysis of the two can effectively reduce the impact of external interference factors on the quality assessment results, thereby improving the reliability of search quality assessment.

[0096] Analysis of search scenarios frequently resulting in misjudgments reveals that the primary reason for the need for search quality optimization lies in the inherent error in the pre-defined relevance scoring model. Furthermore, this error may not be reflected in the model's own confidence level of the output. In other words, it's possible for the pre-defined relevance scoring model to output an incorrect result with a high confidence level. Therefore, relying solely on the confidence level of the pre-defined relevance scoring model to filter search records for review might miss cases where the model assigns high confidence but is actually incorrect, thus limiting the potential for search quality optimization.

[0097] In one embodiment, selecting at least one search record to be reviewed from each target search record that meets the second relevance condition includes:

[0098] Using at least one large-scale relevance analysis model, preliminary relevance checks are performed on the target search terms and search result content features in the target search records, resulting in preliminary relevance check results output by each large-scale relevance analysis model. The large-scale relevance analysis model is obtained by fine-tuning a pre-trained large language model using target search record training samples. The model parameters of the large-scale relevance analysis model are larger than those of the preset relevance scoring model, and the evaluation methods for relevance differ between the preliminary relevance check and the relevance review. From each target search record, at least one search record to be reviewed is selected whose preliminary relevance check results satisfy the second relevance condition.

[0099] Among them, a large-scale relevance analysis model can refer to a vertical-category large language model that is fine-tuned using training samples relevant to the relevance analysis task. It is used to analyze and evaluate the relevance between target search terms and the content features of search results. For example, an open-source large language model can be used as a base, and labeled search record training samples can be used to fine-tune it to enable it to perform relevance analysis, thus forming a large-scale relevance analysis model.

[0100] To meet the real-time requirements of the search process, relevance scoring models can employ a relatively small number of parameters. This limits their model capacity and expressive power, resulting in insufficient semantic capacity and reasoning depth. When faced with complex scenarios such as semantic ambiguity, multimodal content, or long-tail queries, they remain at a shallow semantic similarity matching level, struggling with complex logical reasoning and deep intent analysis. This can easily lead to judgment bias due to a failure to understand deeper intents. In contrast, large-scale relevance analysis models possess a much larger number of parameters, deeper network layers, and are trained on massive amounts of data. This results in stronger semantic understanding and complex case recognition capabilities. They can analyze the logical connections and intent relevance between search result content features and search terms from a deeper semantic dimension, thus more accurately identifying cases that the pre-defined relevance scoring model might misjudge. This preliminary verification can filter out potentially biased search records from a large number of target search records for further review.

[0101] Optionally, considering that the evaluation results of a single model are limited by its training data, model capacity, and generalization ability, it is prone to judgment bias when facing complex scenarios such as semantic ambiguity, multimodal content, or long-tail queries. Therefore, relying solely on the output of a single model to screen search records to be reviewed may, on the one hand, miss some cases with judgment bias, and on the other hand, make it difficult to comprehensively identify the risk of misjudgment of search records to be reviewed from multiple dimensions, thus affecting the comprehensiveness and accuracy of subsequent negative sample construction. Therefore, it is possible to synthesize the preliminary relevance assessment results output by multiple large relevance analysis models to evaluate the target search records from multiple dimensions and screen the search records to be reviewed accordingly.

[0102] The preliminary relevance check result can refer to the result output by the large relevance analysis model after performing a preliminary relevance check on the content features of the target search term and the search results. It can include at least one of relevance, thought chain data and confidence.

[0103] The second relevance condition can be set based on the initial relevance check results. This second relevance condition does not rely on the initial relevance check results of a single large-scale relevance analysis model, but rather is a comprehensive judgment of the initial relevance check results output by various large-scale relevance analysis models. It filters out target search records that the preset relevance scoring model may misjudge, or that differ between the various large-scale relevance analysis models, as search records to be reviewed, providing accurate target objects for subsequent review and negative sample construction.

[0104] Optionally, the second relevance condition may include: in the initial relevance assessment results, the number of relevance assessment results with a relevance score lower than a preset relevance score threshold exceeds a preset low relevance score threshold. For example, five large relevance analysis models are used to analyze a target search record, with a preset relevance score threshold of 0.6 and a preset low relevance score threshold of 0. If one or more large relevance analysis models output a relevance score lower than 0.6 for the target search record, the second relevance condition is met, and the target search record is selected as a search record to be reviewed. The second relevance condition in this embodiment is applicable to identifying cases where most models agree that the relevance is low; such target search records may have significant relevance bias.

[0105] Optionally, the second relevance condition may include: in the initial relevance assessment results, the number of models with confidence scores lower than a preset confidence threshold exceeds a preset low-confidence threshold. For example, five large relevance analysis models are used to analyze a target search record, with a preset confidence threshold of 0.7 and a preset low-confidence threshold of 1. If for a target search record, two or more large relevance analysis models output confidence scores lower than 0.7, the second relevance condition is met, and the target search record is selected as a search record to be reviewed. The second relevance condition in this embodiment is applicable to identifying cases where the model's own judgment has high uncertainty. Such target search records may have semantic ambiguity or insufficient information.

[0106] Optionally, the second relevance condition may include: the dispersion or difference between the relevance scores in the initial relevance assessment results exceeds a preset disagreement threshold. For example, if three large relevance analysis models are used to analyze a target search record, and the output relevance scores are 0.8, 0.3, and 0.7 respectively; by calculating their variance or range, if their variance or range exceeds a preset disagreement threshold, such as a range > 0.4, it indicates that there is a significant disagreement among the large relevance analysis models regarding the relevance of the target search record, and the target search record meets the second relevance condition and is selected as a search record to be reviewed. The second relevance condition in this embodiment is applicable to identifying cases where there is a large disagreement among the models. Such target search records may belong to complex scenarios with ambiguous boundaries and difficult to determine.

[0107] Optionally, the second relevance condition may satisfy at least one of the following: In each preliminary relevance check result, the number of preliminary relevance check results with a relevance lower than a preset relevance threshold exceeds a preset low relevance threshold; in each preliminary relevance check result, the number of models with a confidence level lower than a preset confidence threshold exceeds a preset low confidence threshold; the dispersion or difference between the relevance levels in each preliminary relevance check result exceeds a preset divergence threshold. If a target search record satisfies any one of these conditions, the target search record is selected as a search record to be reviewed.

[0108] Specifically, a certain number of search record training samples can be used in advance to fine-tune at least one pre-trained large language model, enabling it to analyze the correlation between target search terms and search result content features, thus forming a large correlation analysis model. After obtaining the target search records, the target search terms in each target search record, along with the multimodal search result content features, can be assembled into suggestion text. Subsequently, for each target search record's corresponding suggestion text, such as... Figure 4 As shown, at least one large-scale relevance analysis model is invoked to perform an initial relevance check on the target search terms and search result content features in the target search record based on the prompt text. Each large-scale relevance analysis model performs the analysis independently and outputs its corresponding initial relevance check results. After obtaining the initial relevance check results output by each large-scale relevance analysis model, the results are combined to evaluate whether the target search record meets the preset second relevance condition. Finally, the target search records that meet the second relevance condition are identified as search records to be reviewed and enter the subsequent review and negative sample construction process.

[0109] In this embodiment, by introducing multiple large-scale relevance analysis models to analyze target search records in parallel, it is possible to evaluate the same target search record from multiple dimensions from different model perspectives and data distributions. This effectively identifies judgment biases and misjudgment risks that a single model might overlook. By setting a second relevance condition, target search records with discrepancies between the large-scale relevance analysis models, low confidence levels, or questionable evaluation results can be filtered out, making subsequent verification and negative sample construction more targeted and comprehensive. This helps to more systematically capture cases of inaccurate judgments by the preset relevance scoring model, providing richer feedback information for model optimization and thus improving the overall effect of search quality optimization.

[0110] In one embodiment, the relevance of the target search terms in the search records to be reviewed to the content features of the search results is reviewed to obtain the relevance review results, including:

[0111] The relevance arbitration model is used to verify the relevance between the target search terms and the content features of the search results in the search records to be reviewed, and the relevance verification results are obtained. The relevance arbitration model is obtained by reinforcement learning training on a pre-trained large language model.

[0112] Among them, the relevance arbitration model refers to a large language model used to review and evaluate the relevance between the target search term and the content features of the search results. The relevance arbitration model can be a closed-source large language model or an open-source large language model trained with reinforcement learning.

[0113] Reinforcement learning training is a machine learning method based on the Markov decision process framework. It allows an agent to learn optimal policies through trial and error in its interactions with the environment. The agent performs actions in the environment and receives reward signals based on the outcomes of those actions. These reward signals guide the agent to adjust its policy to maximize long-term cumulative rewards.

[0114] Large-scale relevance analysis models can analyze the logical connections and intent relevance between search result content features and search terms from a deeper semantic dimension, thereby identifying cases where pre-defined relevance scoring models may misjudge and filtering out search records for review from a large number of target search records. However, when cases are in ambiguous boundaries, or when multiple large-scale relevance analysis models give conflicting judgments on the same search result, a higher-level relevance arbitration model is needed. Compared to large-scale relevance analysis models, relevance arbitration models are mainly used to resolve disputes and establish standards. Relevance arbitration models have stronger metacognitive reasoning capabilities, that is, the ability to examine and critique their own and others' judgment processes. Closed-source large language models or open-source large language models trained with reinforcement learning are significantly superior in terms of human preference alignment, complex instruction compliance, and logical consistency. They can comprehensively consider query intent, search result content features, the judgment basis of existing analysis models, and historical arbitration cases to make judgments on disputed cases and output detailed explanations of the criteria.

[0115] In practical applications, large-scale relevance arbitration models can be implemented in various ways. As one option, they can be implemented using closed-source large language models, such as GPT-5.2 and Google-Gemini-3 Pro. These models acquire powerful reasoning capabilities during the pre-training phase through methods like RLHF (Reinforcement Learning from Human Feedback), possessing strong general semantic understanding and logical reasoning abilities. They can perform layer-by-layer reasoning and comprehensive analysis on the relevance between search result content features and target search terms, thus obtaining a more accurate relevance output compared to pre-defined relevance scoring models.

[0116] As an alternative approach, the relevance arbitration model can also be obtained by reinforcement training based on an open-source large language model. Specifically, a suitable open-source large language model can be selected as the base, and reinforcement learning techniques can be used to improve the model's general reasoning ability and complex semantic understanding level, forming a relevance arbitration model with review and evaluation capabilities.

[0117] Specifically, after obtaining the search records to be reviewed, the relevance arbitration model is invoked to review the relevance between the target search terms and the content features of the search results in each search record. Based on its deep reasoning ability learned during reinforcement learning training, the relevance arbitration model performs multi-dimensional understanding and comprehensive analysis of the degree of association between the target search terms and the content features of the search results, and outputs the corresponding relevance review results.

[0118] In this embodiment, a large-scale relevance arbitration model trained through reinforcement learning is introduced to evaluate the search records to be reviewed. Compared to methods relying on manual evaluation, the large-scale relevance arbitration model can process large-scale samples automatically, significantly improving review efficiency. Furthermore, the large-scale relevance arbitration model possesses strong semantic understanding and complex scene recognition capabilities, enabling it to output more accurate and interpretable relevance review results. By using these high-quality review results for subsequent negative sample construction and model optimization, more reliable feedback signals can be provided for the iteration of the preset relevance scoring model, thereby improving the overall effect of search quality optimization.

[0119] In one embodiment, the initial relevance check result includes at least one of a second relevance degree and a confidence level of the second relevance degree; from each target search record, at least one search record to be checked is selected whose initial relevance check result satisfies the second relevance condition, including:

[0120] For any target search record, extract the second relevance and the confidence level of the second relevance from the initial relevance verification results corresponding to the target search record; determine the confidence level that is less than a preset confidence threshold as the target confidence level, and determine the second relevance that is less than a preset relevance threshold as the target second relevance level; and determine the maximum relevance difference value based on the maximum value of the second relevance level and the minimum value of the second relevance level; if at least one second relevance condition is met in each initial relevance verification result, the target search record is determined as a search record to be verified; wherein, the second relevance condition includes: the number of target confidence levels exceeds a preset first quantity threshold, the number of target second relevance levels exceeds a preset second quantity threshold, and the maximum relevance difference value exceeds a preset difference threshold.

[0121] The second relevance can refer to the relevance evaluation result output by the large relevance analysis model after performing a preliminary relevance check on the target search records. It can be expressed in the form of relevance score, relevance level, etc.

[0122] Optionally, the second relevance can be a relevance level expressed in discrete rank form, such as "highly relevant," "moderately relevant," "lowly relevant," and "irrelevant." The initial relevance assessment result is mainly used to filter out search records to be reviewed from the target search records, rather than to perform fine-grained ranking of relevance. Therefore, compared to outputting specific relevance scores, using relevance levels can distinguish the relevance of search results at a more concise granularity. While meeting the filtering requirements, it reduces the requirements for the accuracy of the model output and makes the setting of filtering conditions more flexible and interpretable.

[0123] The target second relevance can refer to the second relevance whose value is less than the preset relevance threshold.

[0124] Confidence level refers to the degree of confidence a model has in the relevance of its output, typically represented by a value between 0 and 1. A higher value indicates greater confidence in the model's output. In this embodiment, confidence level refers to the degree of confidence a large correlation analysis model has in the second degree of relevance of its output.

[0125] The target confidence level can refer to the confidence level whose value is less than a preset confidence threshold.

[0126] The maximum relevance difference value can refer to the difference between the maximum and minimum values ​​of the second relevance values ​​output by each large relevance analysis model. It is used to measure the degree of disagreement among the large relevance analysis models in judging the relevance of the same target search record.

[0127] In this embodiment, the second relevance condition includes: the number of target confidence scores exceeding a preset first threshold, the number of target second relevance scores exceeding a preset second threshold, and the maximum relevance difference exceeding a preset difference threshold. Specifically, the number of target second relevance scores exceeding the preset second threshold is applicable to identifying cases where most models agree that the relevance is low; these target search records may have significant relevance bias. The number of target confidence scores exceeding the preset first threshold is applicable to identifying cases where the model's own judgment is highly uncertain; these target search records may have semantic ambiguity or insufficient information. The maximum relevance difference exceeding the preset difference threshold is applicable to identifying cases where there is significant disagreement among models; these target search records may belong to complex scenarios with ambiguous boundaries and difficult to determine.

[0128] Specifically, such as Figure 5As shown, for each target search record, multiple large-scale relevance analysis models are used to perform preliminary relevance checks on the target search record, and the output preliminary relevance check results are obtained. Each preliminary relevance check result includes the second relevance score output by the large-scale relevance analysis model, and the confidence score of the large-scale relevance analysis model for its own output results. Subsequently, each confidence score is compared with a preset confidence threshold one by one. Confidence scores lower than the preset confidence threshold are determined as target confidence scores, and their number is counted. Simultaneously, each second relevance score is compared with a preset relevance threshold one by one. Second relevance scores lower than the preset relevance threshold are determined as target second relevance scores, and their number is counted. Simultaneously, the maximum and minimum values ​​of each second relevance score are found, and the difference between the two is calculated as the maximum relevance difference value. Next, based on the determined number of target confidence scores, the number of target second relevance scores, and the maximum relevance difference value, it is determined whether the target search record meets at least one second relevance condition. The second relevance conditions include the number of target second relevance scores exceeding a preset second quantity threshold, the maximum relevance difference value exceeding a preset difference threshold, and the number of target confidence scores exceeding a preset first quantity threshold. Target search records that meet at least one of the second relevance criteria are identified as search records to be reviewed and will proceed to the subsequent review process.

[0129] In this embodiment, when screening search records to be reviewed, information from multiple dimensions, including relevance, confidence, and the degree of disagreement between models, is introduced to construct a comprehensive screening mechanism. Compared to screening methods that rely on a single model or a single indicator, this approach considers both the relevance judgment results of the large relevance analysis model and the model's confidence in its own judgment, enabling a more comprehensive identification of potentially problematic cases. Furthermore, by introducing the maximum relevance difference value as an indicator, it effectively captures situations where multiple large relevance analysis models show significant disagreement in their relevance judgments of the same target search record. Disagreement between models often indicates a certain degree of complexity or ambiguity in the case, requiring further review and confirmation. Through this multi-dimensional, multi-model comprehensive screening mechanism, target search records that may have judgment biases or are controversial can be identified more comprehensively and accurately as search records to be reviewed, providing high-quality sample input for subsequent review and negative sample construction.

[0130] After collecting one or more negative samples, the collected negative samples can be used to optimize and train the preset correlation scoring model.

[0131] In one embodiment, the relevance verification result includes a first relevance degree and thought chain data, wherein the thought chain data is used to characterize the reasoning process of obtaining the first relevance degree from the large-scale relevance arbitration model; the method further includes:

[0132] Based on the search records to be reviewed, the first relevance score, and the thought chain data, the preset relevance scoring model is optimized.

[0133] Among them, the first relevance can refer to the relevance evaluation result output by the relevance arbitration model after reviewing the relevance of the search records to be reviewed. It can be expressed in the form of relevance score, relevance level or binary judgment label.

[0134] Thinking chain data can refer to the data used to characterize the reasoning process of obtaining the first relevance degree in the large-scale relevance arbitration model. It can include analysis steps, key information identified, logical deduction process, etc.

[0135] Specifically, after analyzing the search records to be reviewed, the relevance arbitration model outputs a relevance review result, which includes primary relevance and thought chain data. The server can periodically perform relevance reviews and construct a negative sample by combining the primary relevance and thought chain data from the search records that do not meet the primary relevance condition with the primary relevance and thought chain data from their corresponding relevance review results. After all the search records collected in each period have been reviewed, one or more negative samples constructed in the current period can be used as optimization signals to optimize the preset relevance scoring model.

[0136] Optionally, the method of using one or more negative samples constructed in the current period as optimization signals to optimize the preset relevance scoring model may include: taking the search record to be reviewed as input, the first relevance as the supervision label, and the thought chain data as an additional supervision signal, and training the preset relevance scoring model through multi-task learning so that it not only learns to output the correct relevance, but also learns to generate or approximate the correct reasoning logic.

[0137] Specifically, during model training, the pre-defined relevance scoring model sets up two parallel output tasks for the same search record to be reviewed. One output task outputs the relevance score, and the loss is calculated between the output relevance score and the first relevance score. The other output task outputs the inference process, and the loss is calculated between the output inference process and the thought chain data. The weighted sum of the losses from the two output tasks is used together to update the model parameters. In this way, the pre-defined relevance scoring model learns both relevance judgment and inference logic generation during training, allowing the two to promote each other. Learning the inference logic helps the model more accurately grasp the basis for relevance judgment, while learning the relevance score provides the final verification target for the inference logic.

[0138] Optionally, the method of using one or more negative samples constructed in the current period as optimization signals to optimize the preset correlation scoring model may also include: converting the thought chain data into features or auxiliary loss, and guiding the model to focus on key inference steps during model training.

[0139] Specifically, the thought chain data is first analyzed to extract key reasoning steps or logical nodes. For example, for a relevance analysis task, the thought chain data may contain key steps such as "identifying query intent," "analyzing the core content of search results," and "comparing matching degrees," as well as key information in each step, such as "the search term 'held in location B' indicates the host location information" and "the search result 'advances' indicates the eligibility information." Subsequently, these key reasoning steps are transformed into auxiliary guidance signals during the training process, so that while optimizing the relevance prediction ability, the internal information processing path of the preset relevance scoring model is closer to the thinking mode of the relevance arbitration model, thereby improving the reasoning ability and generalization performance of the preset relevance scoring model.

[0140] Optionally, the method of transforming key reasoning steps into auxiliary guidance signals during the training process may include: transforming key steps into attention guidance targets of the intermediate layer of a pre-defined relevance scoring model, so that the pre-defined relevance scoring model can give higher attention weight to features related to these key steps when processing input data.

[0141] Optionally, transforming key reasoning steps into auxiliary guidance signals during training can also include converting key steps into auxiliary loss functions. Specifically, auxiliary outputs can be set at different levels of the pre-defined relevance scoring model, requiring these intermediate outputs to predict or approximate the reasoning results of the corresponding steps in the thought chain data. Multiple auxiliary losses and relevance prediction losses jointly participate in model training, guiding the pre-defined relevance scoring model to gradually form a reasoning logic consistent with the thought chain data throughout the forward propagation process.

[0142] Optionally, the relevance review cycle can be one day, that is, a relevance review and preset relevance scoring model optimization can be performed once a day, so as to more promptly detect and respond to sudden fluctuations in online search quality.

[0143] In this embodiment, when optimizing the preset relevance scoring model, not only are the relevance assessment results output by the relevance arbitration model utilized, but also thought chain data is introduced. Compared to the traditional optimization method that relies solely on the final label, firstly, thought chain data can provide richer supervisory signals for model optimization, enabling the preset relevance scoring model to learn not only the final relevance judgment result but also the logical basis for arriving at that judgment, thus gaining a deeper understanding of the inherent laws of relevance assessment. Secondly, by introducing the reasoning process, the preset relevance scoring model can learn the thinking methods of the relevance arbitration model, including how to identify key information, how to perform logical deduction, and how to make judgments by integrating multiple dimensions of factors. This deep knowledge transfer helps improve the reasoning ability and generalization performance of the preset relevance scoring model, enabling it to make more accurate judgments when faced with complex or unseen cases. Thirdly, the introduction of thought chain data makes the model optimization process more interpretable. The optimized model not only outputs relevance results, but its internal reasoning logic is also closer to the thinking methods of the relevance arbitration model, facilitating subsequent analysis, debugging, and improvement. Therefore, this embodiment uses both relevance and thought chain data as optimization signals to achieve a leap from result alignment to process alignment, effectively improving the optimization effect and reasoning ability of the preset relevance scoring model.

[0144] In one embodiment, the preset relevance scoring model is optimized based on the search records to be reviewed, the first relevance score, and the thought chain data, including:

[0145] The relevance scoring model to be optimized is used to evaluate the relevance of the search records to be reviewed, and the training relevance between the search result content features and the target search terms is obtained, as well as the mind chain training data. The mind chain training data is used to represent the reasoning process of the relevance scoring model to be optimized to obtain the training relevance. Based on the difference between the training relevance and the first relevance, and the difference between the mind chain training data and the mind chain data, the preset relevance scoring model is optimized.

[0146] The relevance scoring model to be optimized refers to the pre-defined relevance scoring model that needs optimization, which is used as the training object during the optimization process. It should be noted that although the pre-defined relevance scoring model has already evaluated the relevance of any search record to be reviewed during the online real-time response to search requests and output the relevance score, the relevance evaluation result of the relevance scoring model for the same input may change during the optimization training phase as the model parameters are iteratively updated. Therefore, in each optimization training iteration, the search record to be reviewed still needs to be used as input and re-input to the relevance scoring model to be optimized for relevance evaluation to obtain the training relevance and thought chain training data under the current parameter state.

[0147] Training relevance refers to the relevance output by the relevance scoring model to be optimized after evaluating the relevance of the search records to be reviewed.

[0148] The training data of the mind chain can refer to the data generated by the relevance scoring model to be optimized while outputting the training relevance, which is used to characterize its reasoning process. It can reflect the internal reasoning logic of the relevance scoring model in the current state when evaluating the relevance of the search records to be reviewed.

[0149] Specifically, the target search terms and search result content features from the search records to be reviewed are input into the relevance scoring model to be optimized. The relevance scoring model processes the input data and outputs training relevance and thought chain training data. Then, the obtained training relevance is compared with the first relevance output by the relevance arbitration model, and the difference between the two is calculated as the relevance prediction loss. Simultaneously, the obtained thought chain training data is compared with the thought chain data output by the relevance arbitration model, and the difference between the two is calculated as the inference process loss. Next, the relevance prediction loss and the inference process loss are weighted and summed according to preset weights to obtain the total loss. The model parameters of the relevance scoring model to be optimized are updated based on the total loss using the backpropagation algorithm.

[0150] The correlation prediction failure can be calculated using methods such as mean squared error and cross-entropy, depending on the specific form of the correlation.

[0151] The inference process loss can be calculated using methods such as sequence generation loss and KL divergence (also known as relative entropy), which are used to measure the similarity between the inference process generated by the model to be optimized and the inference process of the large model.

[0152] In this embodiment, by aligning the training data and the data from the thought chain, the reasoning ability of the large-scale relevance arbitration model is transferred to the model to be optimized. By allowing the relevance scoring model to be optimized to learn the reasoning logic of the large-scale relevance arbitration model, its judgment accuracy and generalization ability when facing complex cases can be effectively improved. Furthermore, through supervision of the reasoning process, the optimization process of the relevance scoring model becomes more interpretable. By observing the differences between the training data and the data from the thought chain, specific problems in the reasoning process of the relevance scoring model can be identified, such as errors in the key information identification stage or deviations in the logical deduction stage, providing a basis for subsequent targeted optimization.

[0153] As the pre-defined relevance scoring model is continuously iterated and optimized, search quality improves accordingly. However, relying solely on the model's spontaneous iterative optimization may not solve all existing problems. These issues often require manual intervention to update and adjust the optimization method itself, such as revising relevance conditions or optimizing adjustment strategies. Therefore, after constructing negative samples, in addition to iteratively optimizing the pre-defined relevance scoring model based on these negative samples to continuously improve search quality, the accumulated negative samples can also be used to evaluate the search quality at each stage of the optimization process. This provides quantitative feedback on the search function's performance, allowing relevant managers to promptly grasp the model's optimization effectiveness and intervene manually when necessary.

[0154] In one embodiment, the method further includes:

[0155] Obtain at least one target negative sample corresponding to a preset first historical time period; evaluate the search quality within the preset first historical time period based on each target negative sample, and obtain the search quality evaluation result.

[0156] The preset first historical time period can refer to a pre-defined historical time interval used for search quality assessment. It should be noted that the search quality assessment can be performed periodically according to a preset assessment cycle. This assessment cycle can be the same as or different from the search quality optimization cycle; this embodiment does not impose any restrictions.

[0157] The target negative sample can refer to the negative sample that has been optimized by the preset correlation scoring model constructed within the preset first historical period.

[0158] Search quality assessment results can refer to the quantitative or qualitative conclusions obtained after analyzing the search quality within a preset first historical period based on the target negative sample.

[0159] Optionally, the search quality assessment results may include the number or proportion of target negative samples. A higher number or proportion of target negative samples indicates more misclassification cases by the model during that period, and thus lower search quality.

[0160] Optionally, the search quality assessment results may include the type of misclassification. The type of misclassification for each target negative sample can be identified by clustering the thought chain data within the target negative samples.

[0161] Specifically, after constructing negative samples, the constructed negative samples and their timestamps can be stored together in the data storage system. Upon triggering the search quality assessment, based on a pre-defined first historical time period, all negative samples constructed within that period are retrieved from the data storage system using timestamp filtering, serving as target negative samples. Subsequently, each target negative sample is analyzed to extract statistical information or indicators reflecting search quality, forming a search quality assessment result for the search quality within the pre-defined first historical time period.

[0162] In this embodiment, negative samples represent typical cases where the preset relevance scoring model has failed, directly reflecting specific problems in search quality. Evaluation based on negative samples allows for more precise identification of weaknesses in search quality. Simultaneously, the construction and optimization of negative samples are carried out concurrently, possessing inherent real-time characteristics. By aggregating negative samples by historical time periods, continuous control over search quality trends can be achieved, enabling timely detection of fluctuations or deterioration in search quality. Furthermore, negative samples not only contain relevance judgment results but also inference process information such as thought chain data. This allows evaluation based on negative samples to not only quantify search quality levels but also to deeply analyze the types, causes, and distribution characteristics of misjudgments, providing a basis for subsequent targeted optimization decisions.

[0163] In one embodiment, the search quality within a preset first historical time period is evaluated based on each target negative sample to obtain the search quality evaluation result, including:

[0164] The total number of negative samples for each target negative sample is counted, as well as the total number of search records corresponding to the preset first historical period. Based on the total number of negative samples and the total number of records, the negative sample rate is determined, and the negative sample rate is used as the search quality assessment result.

[0165] The total number of negative samples can refer to the sum of all target negative samples generated and confirmed within the preset first historical period.

[0166] The total number of records can refer to the sum of all search records generated within the preset first historical time period.

[0167] The negative sample rate refers to the ratio of the total number of negative samples to the total number of records. It is used to quantify the relative frequency of misclassifications of relevance in a unit of search request. For example, if 500,000 search records are generated in a preset first historical period, and 1,250 negative samples are constructed, the negative sample rate = 1,250 ÷ 500,000 × 100% = 0.25%, meaning that approximately 25 verified misclassifications of relevance occur per 10,000 searches.

[0168] Specifically, for each target negative sample acquired within a preset first historical time period, a statistical operation is performed to obtain the total number of negative samples; simultaneously, a statistical operation is performed on all corresponding search records within the same preset first historical time period to obtain the total number of records. Then, the total number of negative samples is divided by the total number of records, and the ratio is determined as the negative sample rate. The calculated negative sample rate is then determined as one of the search quality evaluation results corresponding to the preset first historical time period.

[0169] In this embodiment, the negative sample rate directly reflects the probability of misjudging the verified relevance in a unit search request, thereby effectively reducing the impact of fluctuations in search request volume on search quality assessment conclusions and making search quality comparable across different time periods and traffic scenarios. Simultaneously, the calculation logic of the negative sample rate is transparent and the data source is clear, thus significantly reducing assessment implementation costs and subjective bias. Therefore, the operations and algorithm teams can quickly identify trends in search quality improvement or deterioration, accurately verify the actual effectiveness of model optimization measures, and provide intuitive and reliable data support for the continuous iteration of the search system, thereby improving quality detection efficiency and the scientific nature of decision-making.

[0170] In one embodiment, the relevance review result includes negative sample attribution information; based on each target negative sample, the search quality within a preset first historical period is evaluated to obtain the search quality evaluation result, including:

[0171] For each preset misjudgment type, based on the negative sample attribution information corresponding to each target negative sample, the number of classified negative samples belonging to each target negative sample belonging to the misjudgment type is counted, as well as the total number of search records corresponding to the preset first historical period; based on the number of classified negative samples and the total number of records, the classified negative sample rate of the misjudgment type is determined; the classified negative sample rate of each misjudgment type is determined as the search quality assessment result.

[0172] Among them, the misjudgment type can refer to a pre-defined set of categories used to classify relevance misjudgment patterns.

[0173] Optionally, misjudgment types may include answer relevance misjudgment, topic relevance misjudgment, core word relevance misjudgment, visual relevance misjudgment, and other relevance misjudgments.

[0174] Misjudgment of answer relevance can refer to situations where, for question-type queries—such as search terms in question form, or search terms that implicitly seek answers, such as queries about efficacy, strategies, guides, or functions—the search results fail to effectively meet the needs of answering the query question. For example, a search term like "What is thunderstorm asthma?" might only result in general information about thunderstorms and asthma without explaining the connection between the two.

[0175] Misjudgment of topic matching relevance can occur when the overall topic of the search results is inconsistent with the topic of the search term. For example, the search term is "prenatal and postnatal care," which points to content related to "people," but the search results point to "prenatal and postnatal care of sows," resulting in a significant topic-level deviation.

[0176] Misjudgment of core keyword relevance refers to a situation where search results, while hitting some words in the search term, fail to hit one or more core keywords, resulting in a broadened or narrowed scope of results that does not match the main intent of the search term. For example, if the search term is "Ramon Scepter Dry Red Wine," the search results may only involve "Ramon" or "dry red wine," without simultaneously hitting the core qualifier "scepter."

[0177] Visual relevance misjudgment refers to a situation where the video footage or image content in the search results is inconsistent with the visual elements specified by the search term. For example, the search term is "Teddy dog," but the video footage that appears in the search results is of a Golden Retriever, and the visual content does not match the query intent.

[0178] Negative sample attribution information can refer to the structured labels in the correlation review results that identify the misclassification type of the target negative sample.

[0179] The number of negative samples can refer to the total count of target negative samples that, according to the negative sample attribution information, belong to a specific misclassification type within a pre-defined first historical period.

[0180] The classification negative sample rate refers to the ratio of the number of classified negative samples to the total number of records for a single misjudgment type. It quantifies the relative frequency of relevance misjudgments under that misjudgment type. For example, if 500,000 search records are generated in a given historical time period, and 100 of them are classified as answer relevance misjudgments, 400 as topic relevance misjudgments, 600 as keyword relevance misjudgments, 100 as visual relevance misjudgments, and 50 as other relevance misjudgments, then the classification negative sample rate for the answer relevance misjudgment type can be calculated as: 100 ÷ 50,000 0 × 100% = 0.02%, the negative sample rate for the misclassification of topic relevance = 400 ÷ 500000 × 100% = 0.08%, the negative sample rate for the misclassification of core word relevance = 600 ÷ 500000 × 100% = 0.12%, the negative sample rate for the misclassification of visual relevance = 100 ÷ 500000 × 100% = 0.02%, and the negative sample rate for the misclassification of other relevance types = 50 ÷ 500000 × 100% = 0.01%.

[0181] Specifically, after obtaining the target negative samples, cluster analysis can be performed on the thought chain data of each target negative sample to identify the misjudgment type of each target negative sample and form negative sample attribution information. Subsequently, for each preset misjudgment type, based on the negative sample attribution information corresponding to each target negative sample, the number of target negative samples belonging to that preset misjudgment type within a preset first historical period is counted as the number of categorical negative samples corresponding to that preset misjudgment type; simultaneously, a quantity counting operation is performed on all search records corresponding to the same preset first historical period to obtain the total number of records. Then, the number of categorical negative samples is divided by the total number of records, and the ratio is determined as the categorical negative sample rate corresponding to that misjudgment type. After completing the above statistics and calculations for all preset misjudgment types, the categorical negative sample rates corresponding to each misjudgment type can be used together as the search quality evaluation result.

[0182] The classification negative sample rate reflects the proportion of cases in which the model misjudged a specific type of error among all search records. Since the denominator is the total number of records, the classification negative sample rates for different error types are comparable and can directly reflect the degree of impact of various types of misjudgments on the overall search quality.

[0183] In this embodiment, attribution analysis based on misjudgment type is introduced to achieve a quantitative assessment of the frequency of different misjudgment types. Furthermore, the frequencies of different misjudgment types are directly comparable, clearly demonstrating the impact weight of each type of misjudgment on overall search quality. By controlling the changing trend of the negative sample rate of each category over time, the main driving factors of search quality fluctuations can be accurately identified, providing a clear basis for optimizing resource allocation. Priority is given to optimizing misjudgment types with high negative sample rates, providing strong support for refined detection and targeted optimization of search quality.

[0184] In one embodiment, the search quality within a preset first historical time period is evaluated based on each target negative sample to obtain the search quality evaluation result, including:

[0185] For each target negative sample, query the target traffic entry point corresponding to the target negative sample; for each preset traffic entry point, based on the target traffic entry point corresponding to each target negative sample, count the number of entry negative samples of the target negative sample that matches the preset traffic entry point, and the number of entry records of search records that match the traffic entry point in the preset first historical period; determine the entry negative sample rate of the preset traffic entry point based on the number of entry negative samples and the number of entry records; determine the entry negative sample rate of each preset traffic entry point as the search quality evaluation result.

[0186] It's important to note that different traffic entry points exhibit fundamental differences in user operation patterns, content distribution, and search intent, resulting in unique characteristics of their search quality issues. Using a uniform standard for evaluation might mask specific problems at certain entry points. For instance, video search entry points have a significantly higher misjudgment rate regarding visual relevance than general search entry points, but this difference is diluted in the overall statistics. The lack of granular search quality assessment based on traffic entry points makes it difficult for managers to develop differentiated optimization strategies tailored to the characteristics of different entry points, and also hinders the accurate evaluation of the search quality optimization effects of each entry point.

[0187] In this context, "traffic entry point" refers to the functional entry point or search scenario where a user initiates a search, used to identify the source channel of the search request. "Target traffic entry point" refers to the traffic entry point corresponding to the target negative sample, i.e., the source channel or search scenario of that target negative sample. "Preset traffic entry point" refers to pre-defined traffic entry point categories whose search quality needs to be evaluated; these can be categorized according to actual business requirements, and may include, for example, a general search box, video search tabs, image and text search modules, and in-app search modules.

[0188] For any given traffic entry point, the number of negative samples corresponding to that entry point can be defined as the total count of target negative samples matching that entry point within a preset first historical time period. Matching a target negative sample with a preset traffic entry point means that the target traffic entry point corresponding to the target negative sample is the same as the preset traffic entry point. For example, assuming that the target traffic entry point corresponding to target negative sample S1 is E1, then when counting the number of negative samples corresponding to traffic entry point E1, target negative sample S1 can be included in the statistics.

[0189] For any given traffic entry point, the number of corresponding entry records can refer to the total count of search records that match the preset traffic entry point within a preset first historical time period.

[0190] The negative sample rate refers to the ratio of the number of negative samples to the total number of records for a single preset traffic entry point. For example, if 500,000 search records are generated within a preset first historical period, and 600 of them are from mixed search traffic entry points, 600 are from video search traffic entry points, and 250 are from community search traffic entry points, then the negative sample rate for mixed search traffic entry points can be calculated as follows: 600 ÷ 500,000 × 100% = 0.12%; the negative sample rate for video search traffic entry points is also 600 ÷ 500,000 × 100% = 0.12%; and the negative sample rate for community search traffic entry points is 250 ÷ 500,000 × 100% = 0.05%.

[0191] Specifically, after obtaining the target negative samples, the target traffic entry point corresponding to each target negative sample can be queried first. Then, for each preset traffic entry point, the number of target negative samples that match the target traffic entry point within a preset first historical time period is counted, and this number is taken as the entry negative sample count for that preset traffic entry point. Simultaneously, for search records that match the target traffic entry point within the same preset first historical time period, a count operation is performed to obtain the total number of entry records. Then, the entry negative sample count is divided by the total number of entry records, and the ratio is determined as the entry negative sample rate corresponding to that traffic entry point. After completing the above statistics and calculations for all preset traffic entry points, the entry negative sample rates corresponding to each traffic entry point can be collectively used as the search quality assessment result. Presenting the search quality assessment results by traffic entry point can reflect the search quality level under different search scenarios.

[0192] In this embodiment, a dimensional analysis based on traffic entry points is introduced to achieve a quantitative assessment of search quality across different search scenarios. The negative sample rate at each entry point reveals differences in search quality across different traffic entry points, allowing managers to clearly understand the actual performance of traffic entry points such as general search, video search, and community search. By controlling the trend of the negative sample rate over time for each traffic entry point, the specific search scenarios where search quality fluctuations occur can be accurately identified. This provides a clear basis for optimizing the differentiated allocation of resources, prioritizing optimization of search scenarios with higher negative sample rates at each entry point, and developing appropriate optimization strategies for different traffic entry points.

[0193] In the aforementioned embodiments, by calculating the negative sample rate, the classification negative sample rate, and the entry negative sample rate, a multi-dimensional quantitative evaluation of search quality was achieved from the overall dimension, the misjudgment type dimension, and the traffic entry dimension. Only by promptly detecting abnormal fluctuations in search quality based on these indicators and triggering alarm mechanisms when necessary can the stable operation of the search function be effectively guaranteed.

[0194] In one embodiment, the search quality assessment results include negative sample rate, classification negative sample rate, and entry negative sample rate; the negative sample rate represents the proportion of target negative samples in the search records within a preset first historical period; the classification negative sample rate represents the proportion of target negative samples corresponding to each preset misjudgment type in the search records within the preset first historical period; the entry negative sample rate represents the proportion of target negative samples corresponding to each preset traffic entry point in the search records within the preset first historical period; the method further includes:

[0195] A search quality alarm is triggered if at least one of the negative sample rate, the category negative sample rate, and the entry negative sample rate meets the search quality alarm conditions. The search quality alarm conditions include at least one of the following: the negative sample rate exceeds a preset negative sample rate threshold, the category negative sample rate exceeds a preset category negative sample rate threshold, and the entry negative sample rate exceeds a preset entry negative sample rate threshold.

[0196] Among them, search quality alarm conditions refer to pre-defined rules for judging whether search quality is abnormal and whether an alarm should be triggered. Search quality alarm conditions can be defined based on various quantitative indicators in the search quality assessment results and their corresponding preset thresholds. When any indicator exceeds its corresponding threshold, the search quality is considered abnormal, and an alarm mechanism should be triggered.

[0197] The search quality alert conditions specifically include at least one of the following: the negative sample rate exceeds the preset negative sample rate threshold, the category negative sample rate exceeds the preset category negative sample rate threshold, and the entry negative sample rate exceeds the preset entry negative sample rate threshold.

[0198] The negative sample rate reflects the overall search quality level. When it exceeds a preset negative sample rate threshold, it indicates abnormal fluctuations in overall search quality, potentially indicating systemic quality issues. The classification negative sample rate reflects the search quality level under specific misjudgment types. The same or different classification negative sample rate thresholds can be set for different misjudgment types. When the classification negative sample rate for a certain misjudgment type exceeds its corresponding threshold, it indicates an abnormality in that type of misjudgment problem, requiring close attention. The entry negative sample rate reflects the search quality level of a specific traffic entry point. The same or different entry negative sample rate thresholds can be set for different traffic entry points. When the entry negative sample rate for a certain traffic entry point exceeds its corresponding threshold, it indicates abnormal search quality for that traffic entry point, potentially indicating problems related to the characteristics of that traffic entry point.

[0199] Specifically, such as Figure 6 As shown, at least one target negative sample corresponding to a preset first historical period can be obtained first. Then, the search quality evaluation of the target negative sample within the preset first historical period can be performed according to the search quality evaluation method provided in the above embodiment to obtain the negative sample rate, classification negative sample rate and entry negative sample rate corresponding to the preset first historical period.

[0200] Subsequently, the negative sample rate is compared with a preset negative sample rate threshold. If the negative sample rate exceeds the preset negative sample rate threshold, the search quality alarm condition is met, and an alarm is triggered.

[0201] For the classification negative sample rate, the classification negative sample rate of each misclassification type can be compared with its corresponding classification negative sample rate threshold. If the classification negative sample rate of any misclassification type exceeds its corresponding classification negative sample rate threshold, the alarm condition is met, and an alarm is triggered.

[0202] For the negative sample rate at the ingress point, the negative sample rate at each traffic ingress point can be compared with its corresponding negative sample rate threshold. If the negative sample rate at any traffic ingress point exceeds its corresponding negative sample rate threshold, the alarm condition is met and an alarm is triggered.

[0203] A search quality alarm mechanism is triggered if any one of the three conditions mentioned above is met. The alarm can be implemented in ways including, but not limited to, at least one of the following: generating an alarm record in the quality system; sending an email, SMS, or instant messaging message to designated administrators; highlighting the abnormal indicator in the operations dashboard; or automatically generating an anomaly report. The alarm content can include detailed information such as the specific indicator that triggered the alarm, the abnormal value, the extent to which the threshold was exceeded, the corresponding time period, and the type of misjudgment or traffic entry point involved, so that administrators can quickly understand the anomaly and take appropriate measures.

[0204] In this embodiment, a multi-dimensional and differentiated automated alarm mechanism enables proactive monitoring and timely response to search quality anomalies. By setting independent thresholds for the overall situation, misjudgment types, and traffic entry points, the specific dimensions of anomalies can be accurately identified, providing clear direction for rapid intervention and troubleshooting by management personnel and improving the efficiency of handling search quality issues.

[0205] In one embodiment, obtaining at least one target search record includes:

[0206] Identify at least one traffic entry point corresponding to a preset search scenario; for each traffic entry point, obtain at least one target search record within a preset second historical time period from the data source corresponding to the traffic entry point.

[0207] It's important to note that search systems typically involve multiple traffic entry points. The volume of search requests varies significantly across these entry points. Global random sampling might dilute the sample size of entry points with smaller search request volumes, making it difficult to obtain a sufficiently representative sample. Therefore, implementing differentiated sampling across multiple entry points is crucial for improving the targeting and effectiveness of search quality optimization.

[0208] The preset search scenarios refer to search business categories categorized based on user search intent, content consumption needs, and the product environment in which the search operation occurs. Each search scenario corresponds to a type of search demand with similar characteristics, representing an abstract classification of that type of search at the business level. For example, search scenarios may include general search scenarios, video search scenarios, and image / text search scenarios. Each search scenario can correspond to one or more traffic entry points. A general search scenario might correspond to the "general search box" entry point, while a video search scenario might correspond to entry points such as the "video search tab" or the "short video search module."

[0209] The preset second historical time period can refer to a pre-defined historical time interval for collecting target search records. It can be flexibly determined according to optimization needs, and this embodiment does not impose any restrictions on it.

[0210] Specifically, in response to the sampling configuration operation, the user-configured preset search scenarios that need to be sampled and their corresponding traffic entry points are determined, and a sampling operation is performed for each entry point. For each selected traffic entry point, at least one target search record constructed within a preset second historical time period is obtained from the data source corresponding to that traffic entry point.

[0211] Search records from different traffic entry points can be stored in different data tables, or partitioned within the same data table using the entry point identifier field.

[0212] When acquiring target search records, data can be collected according to a preset sampling strategy. The sampling strategy may include, but is not limited to, random sampling, sampling based on relevance score intervals, and stratified sampling based on user groups. The specific method can be determined according to optimization requirements, and this embodiment does not impose any restrictions on it.

[0213] Optionally, a hybrid sampling strategy based on search term type can be adopted when acquiring target search records. Specifically, search records corresponding to random search terms, high-frequency search terms, and long-tail search terms are collected from the data sources corresponding to each traffic entry point to form the target search record set. Among them, random search terms are used to maintain the unbiasedness of the sample and cover the natural distribution of user search behavior; high-frequency search terms reflect the search experience of the vast majority of users and are used to detect the quality performance of core search scenarios; long-tail search terms are used to test the model's generalization ability under sparse data and identify potential problems in complex or rare query scenarios. By combining the above three types of search terms for sampling, it is possible to ensure the representativeness of the sample while taking into account the coverage of core experience and peripheral scenarios, providing a comprehensive and balanced data foundation for subsequent screening, review, and optimization.

[0214] Optionally, the sampling quantity for different traffic entry points can be differentiated based on at least one of the following: the volume of search requests, business importance, and optimization needs. For core traffic entry points with a large volume of search requests and a significant impact on the overall experience, a larger sampling quantity can be set; for vertical traffic entry points or traffic entry points with a small volume of search requests, an appropriate sampling quantity can be set according to actual needs.

[0215] In this embodiment, by sampling different traffic entry points separately, it can be ensured that each traffic entry point obtains a sufficient number of samples, avoiding the dilution of samples from traffic entry points with small search request volumes, and providing a balanced data foundation for subsequent screening, verification and optimization.

[0216] In one embodiment, retrieving at least one target search record within a preset second historical time period from the data source corresponding to the traffic ingress includes:

[0217] From the data source corresponding to the traffic entry, query at least one search record set within a preset second historical time period. The search record set includes multiple search records, and the search terms in each search record are the same. For each search record set, according to the display sorting of each search record in the search record set, obtain a preset number of search records in the order of the display sorting from front to back, as the target search records.

[0218] A search record set refers to a collection of multiple search records corresponding to the same search term, with each search record corresponding to a specific search result. After a user enters a search term, they typically receive multiple search results, and each search result can be combined with that search term to form a search record. Search records containing the same search term can form a search record set.

[0219] Display order refers to the order in which search results are displayed on a search results page. The display order of search results directly affects the distribution of user attention and click behavior. Generally, results that are ranked higher or more prominently displayed are more likely to be prioritized and clicked by users. Therefore, after obtaining multiple search results based on user-inputted search terms, the terminal can comprehensively rank these results according to a preset ranking algorithm. During the ranking process, in addition to considering the relevance between the search result content features and the search terms, factors such as timeliness, content quality, user preferences, and source authority can be combined for weighted calculations to determine the final ranking order of the search results, which are then displayed on the search results page accordingly. For example, ... Figure 7As shown, assuming the search results for "video of the press conference for the finals of Game A held in location B" (701), "related news and information about the finals of Game A held in location B" (702), and "related news and information about teams from location B entering the finals of Game A" (703) are sorted as 1, 2, and 3 respectively, then... Figure 7 As shown in (a), a linear list layout can be used, placing the search result "Video of the press conference for the finals of Game A held in location B" 701 at the top of the search results page, while the search results "Related news and information about the finals of Game A held in location B" 702 and "Related news and information about teams from location B entering the finals of Game A" 703 are placed below the search result "Video of the press conference for the finals of Game A held in location B" 701. This layout follows the user's top-to-bottom browsing habit, allowing the results ranked higher to enter the user's field of vision first, which helps to improve the exposure efficiency of highly relevant results; or, as Figure 7 As shown in (b), a differentiated layout can be adopted, placing the search result "Video of the press conference for the finals of Game A held in location B" 701 in the core area that occupies the largest area on the search results page, making it more visually prominent. Meanwhile, the search results "Related news and information about the finals of Game A held in location B" 702 and "Related news and information about the teams from location B entering the finals of Game A" 703 are placed in secondary areas that occupy smaller areas, and are still arranged from top to bottom according to the order. This layout method further strengthens the visual weight of the higher-ranked search results through the contrast of area size, which is suitable for display scenarios that emphasize the importance of the first result, and can achieve the highlighting of key content within the limited page space.

[0220] The preset sampling number can refer to the number of search records to be collected from each search record set in advance. For example, if the preset sampling number is 10, it means that each time a sample is taken, the top 10 search records displayed in the search record set corresponding to each search term will be collected as the target search records.

[0221] Optionally, the preset sampling quantity can be different for different traffic entry points. For scenarios that require focused analysis of the quality of core ranking positions, the preset sampling quantity for the corresponding traffic entry point can be set to a smaller value, such as the top 3 positions; for scenarios that require coverage of a wider range of display positions, the preset sampling quantity for the corresponding traffic entry point can be set to a larger value, such as the top 10 positions. The setting of the preset quantity can also be determined comprehensively based on the representativeness of the samples and the scale of the data collection.

[0222] Specifically, firstly, search records generated within a preset second historical time period are queried from the data source corresponding to the specified traffic entry point. These records are then grouped according to search terms to form multiple search record sets. Each search record set corresponds to the same search term, and each search record corresponds to a specific search result, recording its display ranking position on the search results page. Subsequently, for each search record set, the records are traversed in descending order of their display ranking. Starting from the first position in the display ranking, search records are selected sequentially until the number of selected search records reaches a preset sampling quantity. These selected search records can then serve as the target search records for that search term, used for subsequent search quality optimization.

[0223] In this embodiment, by selecting search records sequentially according to their display order, the sample ensures that it includes the top-ranked results for each search term. These results have the most significant impact on user experience, making the collected target search records more representative in the user's perception. For multiple search results under the same search term, by controlling the preset number collected for each search term, sample diversity can be maintained while avoiding oversampling of low-exposure results. This allows subsequent review and optimization resources to focus more on the core aspects actually perceived by the user. Thus, the sampling process is aligned with user perception, providing a high-quality sample foundation for precise optimization of search quality.

[0224] In one embodiment, such as Figure 8 As shown, a search quality optimization method is provided. This method can be executed by a computer device, which can be a terminal or a server. In this embodiment, the method includes the following steps:

[0225] Step 801: Determine at least one traffic entry point corresponding to a preset search scenario;

[0226] Step 802: For each traffic entry point, query at least one set of search records within a preset second historical time period from the data source corresponding to the traffic entry point. The set of search records includes multiple search records, and the search terms in each search record are the same.

[0227] Step 803: For each search record set, according to the display sorting of each search record in the search record set, obtain a preset number of search records in the order of the display sorting from front to back, as target search records. The target search record includes the target search term and the search result content features, and the relevance between the search result content features and the target search term satisfies the first relevance condition. The relevance is evaluated by a preset relevance scoring model.

[0228] Step 804: Using at least one large relevance analysis model, perform preliminary relevance checks on the target search terms and search result content features in the target search records respectively, and obtain the preliminary relevance check results output by each large relevance analysis model. The large relevance analysis model is obtained by fine-tuning the pre-trained large language model using the target search record training samples. The preliminary relevance check results include at least one of the second relevance degree and the confidence degree of the second relevance degree.

[0229] Step 805: For any target search record, extract the second relevance and the confidence of the second relevance from the initial kernel results of each relevance corresponding to the target search record.

[0230] Step 806: Determine the confidence level that is less than the preset confidence threshold as the target confidence level, determine the second relevance that is less than the preset relevance threshold as the target second relevance, and determine the maximum relevance difference value based on the maximum value of the second relevance and the minimum value of the second relevance.

[0231] Step 807: If at least one of the second relevance conditions is met in the preliminary verification results of each relevance, the target search record is determined as the search record to be verified. The second relevance conditions include: the number of target confidences exceeds a preset first quantity threshold, the number of target second relevances exceeds a preset second quantity threshold, and the maximum relevance difference value exceeds a preset difference threshold.

[0232] Step 808: For any search record to be reviewed, the relevance of the target search term and the content features of the search results in the search record to be reviewed is reviewed through the relevance arbitration big model to obtain the relevance review result. The relevance arbitration big model is obtained by reinforcement learning training on the pre-trained big language model.

[0233] Step 809: If the relevance verification result does not meet the first relevance condition, construct a negative sample based on the search record to be verified and the relevance verification result. The relevance verification result includes the first relevance and the thought chain data.

[0234] Step 810: Using the relevance scoring model to be optimized, the relevance of the search records to be reviewed is evaluated to obtain the training relevance between the search result content features and the target search term, as well as the mind chain training data. The mind chain training data is used to characterize the reasoning process of the relevance scoring model to be optimized to obtain the training relevance.

[0235] Step 811: Optimize the preset relevance scoring model based on the difference between the training relevance and the first relevance, as well as the difference between the training data and the data of the mind chain.

[0236] Step 812: Obtain at least one target negative sample corresponding to the preset first historical time period;

[0237] Step 813: Count the total number of negative samples for each target negative sample and the total number of search records corresponding to the preset first historical time period; determine the negative sample rate based on the total number of negative samples and the total number of records;

[0238] Step 814: For each preset misjudgment type, based on the negative sample attribution information corresponding to each target negative sample, count the number of classified negative samples belonging to each target negative sample of the misjudgment type, and the total number of search records corresponding to the preset first historical period; determine the classified negative sample rate of the misjudgment type based on the number of classified negative samples and the total number of records.

[0239] Step 815: For each target negative sample, query the target traffic entry corresponding to the target negative sample; for each preset traffic entry, based on the target traffic entry corresponding to each target negative sample, count the number of entry negative samples of the target negative sample that matches the preset traffic entry, and the number of entry records of the search records that match the traffic entry in the preset first historical period; determine the entry negative sample rate of the preset traffic entry based on the number of entry negative samples and the number of entry records.

[0240] Step 816: If at least one of the negative sample rate, the classification negative sample rate, and the entry negative sample rate meets the search quality alarm condition, a search quality alarm is triggered; wherein, the search quality alarm condition includes at least one of the following: the negative sample rate exceeds a preset negative sample rate threshold, the classification negative sample rate exceeds a preset classification negative sample rate threshold, and the entry negative sample rate exceeds a preset entry negative sample rate threshold.

[0241] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0242] Based on the same inventive concept, this application also provides a search quality optimization apparatus for implementing the search quality optimization method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more search quality optimization apparatus embodiments provided below can be found in the limitations of the search quality optimization method described above, and will not be repeated here.

[0243] In one embodiment, such as Figure 9 As shown, a search quality optimization device is provided, including: an acquisition module 902, a filtering module 904, a relevance verification module 906, and a negative sample construction module 908, wherein:

[0244] The acquisition module 902 is used to acquire at least one target search record, the target search record includes target search terms and search result content features, and the relevance between the search result content features and the target search terms satisfies the first relevance condition, the relevance is evaluated by a preset relevance scoring model;

[0245] The filtering module 904 is used to filter out at least one search record to be reviewed from each target search record that meets the second relevance condition;

[0246] The relevance verification module 906 is used to verify the relevance between the target search term and the content features of the search results in any search record to be verified, and to obtain the relevance verification result; the evaluation method of relevance is different from the evaluation method of relevance verification.

[0247] The negative sample construction module 908 is used to construct negative samples based on the search records to be reviewed and the relevance review results when the relevance review results do not meet the first relevance condition. The negative samples are used to optimize the preset relevance scoring model.

[0248] In one embodiment, the filtering module 904 is further configured to:

[0249] Using at least one large-scale relevance analysis model, preliminary relevance checks are performed on the target search terms and search result content features in the target search records, resulting in preliminary relevance check results output by each large-scale relevance analysis model. The large-scale relevance analysis model is obtained by fine-tuning a pre-trained large language model using target search record training samples. The model parameters of the large-scale relevance analysis model are larger than those of the preset relevance scoring model, and the evaluation methods for relevance differ between the preliminary relevance check and the relevance review. From each target search record, at least one search record to be reviewed is selected whose preliminary relevance check results satisfy the second relevance condition.

[0250] In one embodiment, the correlation verification module 906 is further configured to:

[0251] The relevance arbitration model is used to verify the relevance between the target search terms and the content features of the search results in the search records to be reviewed, and the relevance verification results are obtained. The relevance arbitration model is obtained by reinforcement learning training on a pre-trained large language model.

[0252] In one embodiment, the relevance verification result includes a first relevance degree and thought chain data, wherein the thought chain data is used to characterize the reasoning process of obtaining the first relevance degree from the relevance arbitration large model; the search quality optimization device further includes an optimization module, which is used to:

[0253] Based on the search records to be reviewed, the first relevance score, and the thought chain data, the preset relevance scoring model is optimized.

[0254] In one embodiment, the optimization module is further configured to:

[0255] The relevance scoring model to be optimized is used to evaluate the relevance of the search records to be reviewed, and the training relevance between the search result content features and the target search terms is obtained, as well as the mind chain training data. The mind chain training data is used to represent the reasoning process of the relevance scoring model to be optimized to obtain the training relevance. Based on the difference between the training relevance and the first relevance, and the difference between the mind chain training data and the mind chain data, the preset relevance scoring model is optimized.

[0256] In one embodiment, the initial correlation result includes at least one of the second correlation degree and the confidence level of the second correlation degree; the screening module 904 is further configured to:

[0257] For any target search record, extract the second relevance and the confidence level of the second relevance from the initial relevance verification results corresponding to the target search record; determine the confidence level that is less than a preset confidence threshold as the target confidence level, and determine the second relevance that is less than a preset relevance threshold as the target second relevance level; and determine the maximum relevance difference value based on the maximum value of the second relevance level and the minimum value of the second relevance level; if at least one second relevance condition is met in each initial relevance verification result, the target search record is determined as a search record to be verified; wherein, the second relevance condition includes: the number of target confidence levels exceeds a preset first quantity threshold, the number of target second relevance levels exceeds a preset second quantity threshold, and the maximum relevance difference value exceeds a preset difference threshold.

[0258] In one embodiment, the search quality optimization apparatus further includes a quality assessment module, which is used for:

[0259] Obtain at least one target negative sample corresponding to a preset first historical time period; evaluate the search quality within the preset first historical time period based on each target negative sample, and obtain the search quality evaluation result.

[0260] In one embodiment, the quality assessment module is also used for:

[0261] The total number of negative samples for each target negative sample is counted, as well as the total number of search records corresponding to the preset first historical period. Based on the total number of negative samples and the total number of records, the negative sample rate is determined, and the negative sample rate is used as the search quality assessment result.

[0262] In one embodiment, the correlation verification result includes negative sample attribution information; the quality assessment module is also used for:

[0263] For each preset misjudgment type, based on the negative sample attribution information corresponding to each target negative sample, the number of classified negative samples belonging to each target negative sample belonging to the misjudgment type is counted, as well as the total number of search records corresponding to the preset first historical period; based on the number of classified negative samples and the total number of records, the classified negative sample rate of the misjudgment type is determined; the classified negative sample rate of each misjudgment type is determined as the search quality assessment result.

[0264] In one embodiment, the quality assessment module is also used for:

[0265] For each target negative sample, query the target traffic entry point corresponding to the target negative sample; for each preset traffic entry point, based on the target traffic entry point corresponding to each target negative sample, count the number of entry negative samples of the target negative sample that matches the preset traffic entry point, and the number of entry records of search records that match the traffic entry point in the preset first historical period; determine the entry negative sample rate of the preset traffic entry point based on the number of entry negative samples and the number of entry records; determine the entry negative sample rate of each preset traffic entry point as the search quality evaluation result.

[0266] In one embodiment, the search quality assessment result includes negative sample rate, classification negative sample rate, and entry negative sample rate; the negative sample rate represents the proportion of target negative samples in the search records within a preset first historical period; the classification negative sample rate represents the proportion of target negative samples corresponding to each preset misjudgment type in the search records within the preset first historical period; the entry negative sample rate represents the proportion of target negative samples corresponding to each preset traffic entry point in the search records within the preset first historical period; the search quality optimization device further includes an alarm module, which is used for:

[0267] A search quality alarm is triggered if at least one of the negative sample rate, the category negative sample rate, and the entry negative sample rate meets the search quality alarm conditions. The search quality alarm conditions include at least one of the following: the negative sample rate exceeds a preset negative sample rate threshold, the category negative sample rate exceeds a preset category negative sample rate threshold, and the entry negative sample rate exceeds a preset entry negative sample rate threshold.

[0268] In one embodiment, the acquisition module 902 is further configured to:

[0269] Identify at least one traffic entry point corresponding to a preset search scenario; for each traffic entry point, obtain at least one target search record within a preset second historical time period from the data source corresponding to the traffic entry point.

[0270] In one embodiment, the acquisition module 902 is further configured to:

[0271] From the data source corresponding to the traffic entry, query at least one search record set within a preset second historical time period. The search record set includes multiple search records, and the search terms in each search record are the same. For each search record set, according to the display sorting of each search record in the search record set, obtain a preset number of search records in the order of the display sorting from front to back, as the target search records.

[0272] Each module in the aforementioned search quality optimization device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0273] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores the data involved in the aforementioned methods. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a search quality optimization method.

[0274] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a search quality optimization method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0275] Those skilled in the art will understand that Figure 10 and Figure 11The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0276] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method embodiments.

[0277] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method embodiments.

[0278] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described method embodiments.

[0279] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0280] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, etc., and are not limited to these.

[0281] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0282] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A search quality optimization method, characterized in that, The method includes: At least one target search record is obtained, the target search record includes target search terms and search result content features, and the relevance between the search result content features and the target search terms satisfies a first relevance condition, the relevance being evaluated by a preset relevance scoring model; Using at least one large-scale relevance analysis model, a preliminary relevance check is performed on the target search terms and search result content features in the target search records, respectively, to obtain the preliminary relevance check result output by each large-scale relevance analysis model; the preliminary relevance check result includes a second relevance degree and a confidence level of the second relevance degree; For any target search record, if at least one of the preliminary relevance verification results corresponding to the target search record satisfies a second relevance condition, the target search record is determined as a search record to be verified. The second relevance condition includes: the number of confidence scores less than a preset confidence threshold exceeds a preset first quantity threshold; the number of second relevance scores less than a preset relevance threshold exceeds a preset second quantity threshold; and the difference between the largest and smallest second relevance scores exceeds a preset difference threshold. For any search record to be reviewed, the relevance between the target search term and the search result content features in the search record to be reviewed is reviewed to obtain the relevance review result; the evaluation method of the relevance is different from the evaluation method of the relevance review. If the relevance review result does not meet the first relevance condition, a negative sample is constructed based on the search record to be reviewed and the relevance review result. The negative sample is used to optimize the preset relevance scoring model.

2. The method according to claim 1, characterized in that, The large-scale correlation analysis model is obtained by fine-tuning a pre-trained large language model using target search record training samples; the number of parameters in the large-scale correlation analysis model is larger than the number of parameters in the preset correlation scoring model, and the initial correlation kernel and the secondary correlation kernel have different evaluation methods for correlation.

3. The method according to claim 1, characterized in that, The relevance verification of the target search terms and search result content features in the search records to be verified, to obtain the relevance verification result, includes: The relevance arbitration model is used to verify the relevance between the target search terms and the content features of the search results in the search records to be verified, and the relevance verification result is obtained. The relevance arbitration model is obtained by reinforcement learning training on a pre-trained large language model.

4. The method according to claim 3, characterized in that, The relevance verification result includes a first relevance degree and thought chain data, wherein the thought chain data is used to characterize the reasoning process of obtaining the first relevance degree from the relevance arbitration big model; the method further includes: Based on the search records to be reviewed, the first relevance, and the thought chain data, the preset relevance scoring model is optimized.

5. The method according to claim 4, characterized in that, The step of optimizing the preset relevance scoring model based on the search records to be reviewed, the first relevance, and the thought chain data includes: The relevance of the search records to be reviewed is evaluated using the relevance scoring model to be optimized, and the training relevance between the content features of the search results and the target search term is obtained, as well as the mind chain training data. The mind chain training data is used to characterize the reasoning process of the relevance scoring model to be optimized to obtain the training relevance. Based on the difference between the training relevance and the first relevance, and the difference between the training data and the data of the thought chain, the preset relevance scoring model is optimized.

6. The method according to claim 1, characterized in that, If, in the case that each of the preliminary relevance verification results corresponding to the target search record satisfies at least one of the second relevance conditions, the target search record is determined as a search record to be verified, including: From the initial relevance results corresponding to the target search record, extract the second relevance and the confidence level of the second relevance respectively; The confidence level that is less than the preset confidence threshold is determined as the target confidence level, the second relevance that is less than the preset relevance threshold is determined as the target second relevance, and the maximum relevance difference value is determined based on the maximum value of the second relevance and the minimum value of the second relevance. If, in each of the preliminary relevance verification results, at least one of the second relevance conditions is met, the target search record is determined as a search record to be verified.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Obtain at least one target negative sample corresponding to a preset first historical time period; Based on each target negative sample, the search quality within a preset first historical period is evaluated to obtain the search quality evaluation result.

8. The method according to claim 7, characterized in that, The step of evaluating the search quality within a preset first historical time period based on each of the target negative samples to obtain the search quality evaluation result includes: The total number of negative samples of each target negative sample is counted, as well as the total number of search records corresponding to the preset first historical time period; Based on the total number of negative samples and the total number of records, the negative sample rate is determined, and the negative sample rate is used as the search quality assessment result.

9. The method according to claim 7, characterized in that, The relevance verification result includes negative sample attribution information; the step of evaluating the search quality within a preset first historical period based on each of the target negative samples to obtain the search quality evaluation result includes: For each preset misjudgment type, based on the negative sample attribution information corresponding to each target negative sample, the number of classified negative samples belonging to each of the preset misjudgment types and the total number of search records corresponding to the preset first historical period are counted. The classification negative sample rate of the misclassification type is determined based on the number of classified negative samples and the total number of records; The negative sample rate of each misclassification type is determined as the search quality assessment result.

10. The method according to claim 7, characterized in that, The step of evaluating the search quality within a preset first historical time period based on each of the target negative samples to obtain the search quality evaluation result includes: For each target negative sample, query the target traffic entry point corresponding to the target negative sample; For each preset traffic entry point, based on the target traffic entry point corresponding to each target negative sample, the number of entry negative samples matching the preset traffic entry point and the number of entry records matching the search records within a preset first historical time period are counted. The negative sample rate of the preset traffic entry point is determined based on the number of negative samples at the entry point and the number of entry records. The negative sample rate of each preset traffic entry point is determined as the search quality evaluation result.

11. The method according to claim 7, characterized in that, The search quality assessment results include negative sample rate, category negative sample rate, and entry negative sample rate; the negative sample rate represents the proportion of target negative samples in the search records within a preset first historical period. The classification negative sample rate represents the proportion of target negative samples corresponding to each preset misjudgment type in the search records within a preset first historical period. The negative sample rate at the entry point represents the proportion of the target negative sample corresponding to each preset traffic entry point in the search records within a preset first historical period. The method further includes: If at least one of the negative sample rate, the classification negative sample rate, and the entry negative sample rate meets the search quality alarm condition, a search quality alarm will be issued. The search quality alarm conditions include at least one of the following: the negative sample rate exceeds a preset negative sample rate threshold, the category negative sample rate exceeds a preset category negative sample rate threshold, and the entry negative sample rate exceeds a preset entry negative sample rate threshold.

12. The method according to any one of claims 1 to 6, characterized in that, The step of obtaining at least one target search record includes: Identify at least one traffic entry point corresponding to a preset search scenario; For each traffic entry point, at least one target search record within a preset second historical time period is obtained from the data source corresponding to the traffic entry point.

13. The method according to claim 12, characterized in that, The step of obtaining at least one target search record within a preset second historical time period from the data source corresponding to the traffic entry point includes: From the data source corresponding to the traffic entry, query at least one set of search records within a preset second historical time period, wherein the set of search records includes multiple search records and the search terms in each search record are the same; For each set of search records, according to the display order of each search record in the set, a preset number of search records are obtained sequentially from front to back in the display order, and these are used as target search records.

14. A search quality optimization device, characterized in that, The apparatus is used to implement the method of any one of claims 1 to 13, comprising: The acquisition module is used to acquire at least one target search record, the target search record including target search terms and search result content features, and the relevance between the search result content features and the target search terms satisfies a first relevance condition, the relevance being evaluated by a preset relevance scoring model; A filtering module is used to perform preliminary relevance checks on the target search terms and search result content features in the target search records using at least one large-scale relevance analysis model, and to obtain preliminary relevance check results output by each large-scale relevance analysis model. The preliminary relevance check results include a second relevance score and a confidence score of the second relevance score. For any target search record, if at least one of the preliminary relevance check results corresponding to the target search record satisfies a second relevance condition, the target search record is determined as a search record to be reviewed. The second relevance condition includes: the number of confidence scores less than a preset confidence threshold exceeds a preset first quantity threshold; the number of second relevance scores less than a preset relevance threshold exceeds a preset second quantity threshold; and the difference between the largest and smallest second relevance scores exceeds a preset difference threshold. The relevance verification module is used to perform a relevance verification on the target search terms and search result content features in any search record to be verified, and obtain a relevance verification result; the evaluation method of the relevance is different from the evaluation method of the relevance verification. The negative sample construction module is used to construct negative samples based on the search records to be reviewed and the correlation review results when the correlation review results do not meet the first correlation condition. The negative samples are used to optimize the preset correlation scoring model.

15. The apparatus according to claim 14, characterized in that, The large-scale correlation analysis model is obtained by fine-tuning a pre-trained large language model using target search record training samples; the number of parameters in the large-scale correlation analysis model is larger than the number of parameters in the preset correlation scoring model, and the initial correlation kernel and the secondary correlation kernel have different evaluation methods for correlation.

16. The apparatus according to claim 14, characterized in that, The correlation verification module is also used for: The relevance arbitration model is used to verify the relevance between the target search terms and the content features of the search results in the search records to be verified, and the relevance verification result is obtained. The relevance arbitration model is obtained by reinforcement learning training on a pre-trained large language model.

17. The apparatus according to claim 16, characterized in that, The relevance verification result includes a first relevance degree and thought chain data, wherein the thought chain data is used to characterize the reasoning process by which the relevance arbitration big model obtains the first relevance degree; the search quality optimization device further includes an optimization module, which is used for: Based on the search records to be reviewed, the first relevance, and the thought chain data, the preset relevance scoring model is optimized.

18. The apparatus according to claim 17, characterized in that, The optimization module is also used for: The relevance of the search records to be reviewed is evaluated using the relevance scoring model to be optimized, and the training relevance between the content features of the search results and the target search term is obtained, as well as the mind chain training data. The mind chain training data is used to characterize the reasoning process of the relevance scoring model to be optimized to obtain the training relevance. Based on the difference between the training relevance and the first relevance, and the difference between the training data and the data of the thought chain, the preset relevance scoring model is optimized.

19. The apparatus according to claim 14, characterized in that, The filtering module is also used for: From the initial relevance results corresponding to the target search record, extract the second relevance and the confidence level of the second relevance respectively; The confidence level that is less than the preset confidence threshold is determined as the target confidence level, the second relevance that is less than the preset relevance threshold is determined as the target second relevance, and the maximum relevance difference value is determined based on the maximum value of the second relevance and the minimum value of the second relevance. If, in each of the preliminary relevance verification results, at least one of the second relevance conditions is met, the target search record is determined as a search record to be verified.

20. The apparatus according to any one of claims 14 to 19, characterized in that, The search quality optimization device further includes a quality assessment module, which is used for: Obtain at least one target negative sample corresponding to a preset first historical time period; Based on each target negative sample, the search quality within a preset first historical period is evaluated to obtain the search quality evaluation result.

21. The apparatus according to claim 20, characterized in that, The quality assessment module is also used for: The total number of negative samples of each target negative sample is counted, as well as the total number of search records corresponding to the preset first historical time period; Based on the total number of negative samples and the total number of records, the negative sample rate is determined, and the negative sample rate is used as the search quality assessment result.

22. The apparatus according to claim 20, characterized in that, The correlation verification results include negative sample attribution information; the quality assessment module is also used for: For each preset misjudgment type, based on the negative sample attribution information corresponding to each target negative sample, the number of classified negative samples belonging to each of the preset misjudgment types and the total number of search records corresponding to the preset first historical period are counted. The classification negative sample rate of the misclassification type is determined based on the number of classified negative samples and the total number of records; The negative sample rate of each misclassification type is determined as the search quality assessment result.

23. The apparatus according to claim 20, characterized in that, The quality assessment module is also used for: For each target negative sample, query the target traffic entry point corresponding to the target negative sample; For each preset traffic entry point, based on the target traffic entry point corresponding to each target negative sample, the number of entry negative samples matching the preset traffic entry point and the number of entry records matching the search records within a preset first historical time period are counted. The negative sample rate of the preset traffic entry point is determined based on the number of negative samples at the entry point and the number of entry records. The negative sample rate of each preset traffic entry point is determined as the search quality evaluation result.

24. The apparatus according to claim 20, characterized in that, The search quality assessment results include negative sample rate, category negative sample rate, and entry negative sample rate; the negative sample rate represents the proportion of target negative samples in the search records within a preset first historical period. The classification negative sample rate represents the proportion of target negative samples corresponding to each preset misjudgment type in the search records within a preset first historical period. The ingress negative sample rate represents the proportion of target negative samples corresponding to each preset traffic ingress in the search records within a preset first historical time period; the search quality optimization device further includes an alarm module, which is used for: If at least one of the negative sample rate, the classification negative sample rate, and the entry negative sample rate meets the search quality alarm condition, a search quality alarm will be issued. The search quality alarm conditions include at least one of the following: the negative sample rate exceeds a preset negative sample rate threshold, the category negative sample rate exceeds a preset category negative sample rate threshold, and the entry negative sample rate exceeds a preset entry negative sample rate threshold.

25. The apparatus according to any one of claims 14 to 19, characterized in that, The acquisition module is also used for: Identify at least one traffic entry point corresponding to a preset search scenario; For each traffic entry point, at least one target search record within a preset second historical time period is obtained from the data source corresponding to the traffic entry point.

26. The apparatus according to claim 25, characterized in that, The acquisition module is also used for: From the data source corresponding to the traffic entry, query at least one set of search records within a preset second historical time period, wherein the set of search records includes multiple search records and the search terms in each search record are the same; For each set of search records, according to the display order of each search record in the set, a preset number of search records are obtained sequentially from front to back in the display order, and these are used as target search records.

27. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 13.

28. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.

29. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 13.

Images