System and method for searching link inspection and computing equipment

Abstract

The invention provides a system and a method for searching link routing inspection and computing equipment, and the system comprises a routing inspection module which is configured to simulate a user to perform correlation routing inspection on a search system so as to find a correlation judgment error; the problem attribution module is configured to perform search link attribution on correlation judgment errors and locate search link links with the correlation judgment errors; the dotting information module is configured to provide data support for the problem attribution module by constructing full burying points and shadow service links in a search system to disk full-link dotting information; and the knowledge base updating module is configured to update a plurality of knowledge bases used for searching links according to output results of the problem attribution module. According to the technical scheme provided by the invention, the accuracy of determining the correlation of the search link can be improved, the functions of evaluation, correlation problem attribution and the like are optimized, diagnosis and attribution are carried out through an intelligent means, and a solution is quickly positioned.

Description

Technical Field

[0001] This invention relates to the field of software development technology, and more specifically to a system, method, and computing device for searching link inspection. Background Technology

[0002] In the development of e-commerce, there are always two core issues that need to be balanced in user products: efficiency and experience. Efficiency metrics can usually be defined by clear indicators, but experience metrics are relatively vague and difficult to define. In daily optimization, although changes in experience-side metrics are considered while focusing on improving efficiency, the lack of unified and comprehensive observation metrics often leads to the neglect of the impact on search experience. A good user experience can not only reduce platform churn and increase user loyalty, but also bring new users to the platform.

[0003] Currently, e-commerce platforms obtain user feedback through questionnaires, follow-ups, and experience surveys, and conduct user research reports monthly to collect users' real needs. However, the current feedback loop has some problems, such as lag, long feedback cycles, significant differences in understanding, and difficulty in attributing problems. These problems make it difficult to quickly locate and discover the core issues in current searches and rapidly build solutions using existing user feedback data.

[0004] Therefore, a technical solution is needed that can improve the accuracy of determining the relevance of search links, optimize functions such as evaluation and attribution of relevance issues, and use intelligent means to diagnose and attribute problems to quickly locate solutions. Summary of the Invention

[0005] This invention aims to provide a system, method, and computing device for search link inspection, which can improve the accuracy of search link relevance determination, optimize evaluation and relevance problem attribution functions, and quickly locate solutions through intelligent means of diagnosis and attribution.

[0006] According to one aspect of the present invention, a system for searching link inspection is provided, the system comprising an inspection module, a problem attribution module, a point information module, and a knowledge base update module, wherein,

[0007] The inspection module is configured to simulate a user performing a relevance inspection on the search system, thereby discovering relevance judgment errors. The relevance judgment errors indicate that the user's query request is inconsistent with the query results returned by the search system.

[0008] The problem attribution module is configured to perform search link attribution on the relevance determination error and locate the search link where the relevance determination error occurred.

[0009] The tracking information module is configured to provide data support for the problem attribution module by constructing full-link tracking information for full-location tracking and shadow service link disk in the search system.

[0010] The knowledge base update module is configured to update multiple knowledge bases used in the search chain based on the output results of the problem attribution module.

[0011] According to some embodiments, the inspection module and the problem attribution module are uniformly constructed based on a correlation determination attribution model, which is trained based on a text processing model.

[0012] According to some embodiments, a debugging platform is used to perform manual investigation of the aforementioned correlation issues.

[0013] According to some embodiments, the referee module is configured to correct the results of the outsourced labeled samples based on the attribution model of the correlation determination, and to evaluate the actual online problem-solving ratio through A / B verification.

[0014] According to some embodiments, the plurality of knowledge bases includes one or more of the following: a query extended knowledge base, a product extended term knowledge base, a hypothetical extended knowledge base, and a title attribute conflict knowledge base.

[0015] According to some embodiments, the knowledge base update module is configured as follows:

[0016] In cases where low-level missed recalls are attributed to matching query entries but incorrect categories, update the query extended knowledge base to increase category recall knowledge.

[0017] According to some embodiments, the knowledge base update module is configured as follows:

[0018] If the low-level missed recall is attributed to the misidentification of category terms in the query, update the upper and lower level extended knowledge bases to increase category recall knowledge.

[0019] According to some embodiments, the knowledge base update module is configured as follows:

[0020] In cases where low-end product recalls are attributed to incorrect category term identification in the title, the product extended term knowledge base is updated to increase category recall knowledge.

[0021] According to some embodiments, the search chain includes:

[0022] The query understanding submodule is configured to analyze user query requests;

[0023] The resource understanding submodule is configured to analyze the searched resources;

[0024] The recall submodule is configured to match and retrieve the query request and the resource to obtain a candidate result set.

[0025] The relevance determination submodule is configured to calculate the relevance score between the candidate result set and the query request;

[0026] The sorting and strategy submodule is configured to generate a final sorting output based on the candidate result set according to the relevance score and the job strategy.

[0027] The knowledge base submodule is configured to provide knowledge support to each link of the search chain using the multiple knowledge bases.

[0028] According to some embodiments, the relevance determination submodule is based on a comprehensive relevance scoring model, which includes:

[0029] A wide network directly performs width-side fusion on the first input features, which include category score, product category score, modifier score, query term-product score, query term-title score, semantic score, brand score, and / or model score from the tracking information obtained through tracking.

[0030] A deep network, running in parallel with the wide network, generates embedding vectors and higher-order features from the second input features. The second input features include category matching information, product category matching information, modifier matching information, query term-product matching information, query term-title matching information, semantic matching information, brand matching information, and / or model matching information from the tracking information obtained through tracking.

[0031] The MLP network calculates and directly outputs the comprehensive relevance score, and obtains the relevance classification by threshold division.

[0032] According to another aspect of the present invention, a method for searching link inspection is provided, the method comprising:

[0033] The relevance determination attribution model is used to simulate users performing relevance checks on the search system, thereby discovering relevance determination errors. The relevance determination errors indicate that the user's query request is inconsistent with the product results returned by the search system. The relevance determination attribution model is based on a text processing model.

[0034] The correlation determination attribution model is used to search for the link attribution of the correlation determination error, and the attribution result is obtained.

[0035] The multiple knowledge bases used for the search chain are updated based on the output of the problem attribution module.

[0036] According to some embodiments, updating multiple knowledge bases used for the search chain based on the output of the problem attribution module includes:

[0037] In cases where low-tier missed recalls are attributed to matching query entries but incorrect categories, update the query extended knowledge base to increase category recall knowledge; and / or

[0038] In cases where low-level missed recalls are attributed to errors in category term identification within the query, update the upper and lower level extended knowledge bases to increase category recall knowledge; and / or

[0039] In cases where low-end product recalls are attributed to incorrect category term identification in the title, the product extended term knowledge base is updated to increase category recall knowledge.

[0040] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method as described in any of the preceding claims.

[0041] According to another aspect of the present invention, a computing device is provided, comprising:

[0042] processor;

[0043] A memory that stores a computer program, which, when executed by the processor, implements the method as described in any of the preceding methods.

[0044] According to embodiments of the present invention, a system for search link inspection is provided. The inspection module in the system simulates a user's relevance inspection of the search system to detect relevance judgment errors, i.e., errors where the user's query request and the query results returned by the search system are inconsistent. A problem attribution module performs search link attribution on the relevance judgment errors to locate the link in the search link where the relevance judgment error occurred. A data tracking module provides data support to the problem attribution module by constructing full-link tracking information for full-tracking and shadow service links in the search system. A knowledge base update module updates multiple knowledge bases used for the search link based on the output of the problem attribution module. By obtaining the causes of relevance problems and analyzing the proportion of problems solved through this system, and evaluating the data, knowledge, and accuracy produced at each link, the system can effectively solve relevance problems, improve the accuracy of relevance judgment in the search link, optimize evaluation and relevance problem attribution functions, and quickly locate solutions through intelligent diagnostic and attribution methods, completing the full search link inspection.

[0045] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit the invention. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0047] Figure 1 A schematic diagram of a search link inspection system according to an example embodiment is shown.

[0048] Figure 2 This diagram illustrates the operation of a correlation determination attribution model for search link inspection according to an example embodiment.

[0049] Figure 3 This diagram illustrates the process of determining the attribution model based on the relevance of an example embodiment.

[0050] Figure 4 This diagram illustrates the chain of attributing a problem to a relevant attribution model according to another example embodiment.

[0051] Figure 5 This illustrates a link diagram for search link attribution based on a relevance determination attribution model according to another example embodiment.

[0052] Figure 6 A block diagram of a computing device according to an exemplary embodiment is shown. Detailed Implementation

[0053] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that the invention will be thorough and complete, and the concept of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0054] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of embodiments of the invention. However, those skilled in the art will recognize that the technical solutions of the invention can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of the invention.

[0055] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0056] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0057] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first component discussed below may be referred to as the second component without departing from the teachings of the present invention. As used herein, the term "and / or" includes all combinations of any one and more of the associated listed items.

[0058] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0059] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily essential for implementing the present invention, and therefore cannot be used to limit the scope of protection of the present invention.

[0060] In e-commerce search, basic relevance refers to whether the products a user sees match the user's search query; the degree of match between the user's query and the returned products or documents. It's a core function of search engines, designed to ensure users can find information or products relevant to their needs. Search relevance experience includes the degree of match between user-expressed content such as category, style, efficacy, and material.

[0061] The current search feedback chain has several problems. For example, long feedback cycles lead to significant delays; monthly user research reports are lengthy, making it difficult to understand specific user needs and often resulting in delayed feedback, making problem localization difficult. The types of relevance issues are numerous and vary in importance; different users have different understandings of the issues, leading to unclear problem types and an inability to quantify the problem scale in outsourced and user research feedback. Category, style, and efficacy are more external factors and are not aligned with the actual links in the search chain (query understanding, product understanding, knowledge base, relevance determination, etc.). After a problem is discovered, it is difficult to quickly pinpoint the cause; user issues are not quickly fed back into the search chain to locate the specific module that is malfunctioning. The search chain has many modules, with interdependencies between upstream and downstream components, requiring a lengthy problem-solving cycle.

[0062] The aforementioned issues make it difficult for the platform to quickly locate and discover the core problems in the current search using existing user feedback data, and thus hinder the rapid development of solutions.

[0063] To address this, the present invention proposes a system and method for search link inspection, which can improve the accuracy of search link relevance determination, optimize daily evaluation and relevance problem attribution functions, and quickly locate solutions through intelligent means of diagnosis and attribution.

[0064] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention.

[0065] Figure 1 A schematic diagram of a search link inspection system according to an example embodiment is shown.

[0066] See Figure 1 The search link inspection system includes an inspection module 101, a problem attribution module 103, a point information module 105, and a knowledge base update module 107.

[0067] According to some embodiments, the inspection module 101 is configured to simulate a user performing a relevance inspection on the search system, thereby discovering a relevance determination error. The relevance determination error indicates that the user's query request is inconsistent with the query results returned by the search system.

[0068] According to some implementation examples, relevance in search refers to whether the products seen by the user are consistent with the query itself. It is a general concept of relevance, including the degree of matching of user-expressed content such as category, style, efficacy, and material, as well as the degree of product fraud.

[0069] According to some embodiments, the inspection module 101 is used to discover and locate correlation problems in the search link, and identify the causes, proportions and severity of the correlation problems.

[0070] According to some implementations, the advantage of text processing models is that they possess common sense and human-like judgment abilities, demonstrating a good ability to determine relevance issues. By building a relevance inspection capability for text processing models, problems can be quickly identified and located, revealing the core issues, their proportion, and their severity.

[0071] According to some embodiments, the inspection module 101 quickly discovers and locates problems by building a text processing model relevance inspection capability, further identifies and discovers the causes of relevance problems, and finds the current core problems, problem proportions, severity, etc. Compared with feedback links such as manual annotation or monthly evaluation, the text processing model inspection module has a short feedback cycle, small fluctuations, fast problem discovery, timely application, and can be updated at the daily level.

[0072] The current text processing model achieves a relevance assessment accuracy of 95.8%, comparable to that of manual annotation, effectively reducing labor costs. The model can generate 1 million text processing model inspection results daily, resulting in higher data processing volume and significantly improved inspection efficiency.

[0073] According to some embodiments, the problem attribution module 103 is configured to perform search link attribution on the relevance determination error, locating the search link where the relevance determination error occurred. The problem attribution module 103 can locate the specific link of the relevance problem, generate correct knowledge, and batch-produce the erroneous knowledge that occurred in each link.

[0074] According to some embodiments, the problem attribution module is further configured to: build a debugging platform for investigating the relevance issues; construct various knowledge bases; and support batch and / or single-time rapid import of knowledge bases. The knowledge bases provide data support for category prediction to expand recall, query term understanding, and identification of title category term errors.

[0075] According to some implementation examples, to address the attribution difficulty problem, various knowledge bases can be built using engineering platforms such as AWork, supporting rapid batch and single-time import of knowledge bases. Finally, combining manual annotation with the evaluation of data, knowledge, and accuracy at each stage of production can effectively save labor costs and improve the efficiency and accuracy of evaluation and verification.

[0076] According to some embodiments, the search link inspection system also includes a debugging platform for manual troubleshooting of the aforementioned relevance issues. Because the search system itself is very large, attributing user-perceived problems to specific search architectures is extremely difficult. To address this attribution difficulty, a debugging platform is built through engineering capabilities to enable manual problem troubleshooting.

[0077] According to some embodiments, the inspection module 101 and the problem attribution module 103 are uniformly constructed based on a relevance determination attribution model, which is trained based on a text processing model. For example, it is trained using general knowledge + SFT, DPO, etc.

[0078] According to some embodiments, the search link inspection system also includes a referee module, which is configured to correct the results of the outsourced labeled samples based on the relevance determination attribution model, and to evaluate the actual online problem resolution ratio through A / B verification.

[0079] According to some implementations, the judging module performs A / B verification for evaluation, including outsourced annotation training, semantic upgrade verification, and category prediction experiment accuracy verification. Outsourced annotation training: The target outsourced annotation accuracy is corrected based on the annotation results of the text processing model. Currently, the accuracy rate of capturing outsourced annotations is 94.8%. Re-annotating incorrectly labeled samples improves accuracy by 5%. Semantic upgrade verification: After the semantic model A / B optimization is launched, accuracy is verified using the text processing model. Recently, the capture of semantic model bad cases has shown an upward trend. After evaluation, the category word matching capability is optimized. Category prediction experiment accuracy verification: The current category prediction accuracy is evaluated, and the performance improvement after specific optimization is assessed. A / B verification shows a 3% improvement in broad word recognition accuracy.

[0080] Refer to Table 1, which shows data from a case study on the accuracy verification of a main component. After resolving the main component issues, the problem was evaluated using the referee module of the text processing model. A strict A / B comparison showed that the number of main component problems decreased by 78.5%, demonstrating that the processing by each module in the search link inspection system of this invention effectively reduced the number of related problems.

[0081] Table 1

[0082]

[0083] According to some embodiments, the tracking information module 105 is configured to provide data support for the problem attribution module by constructing full-link tracking information of full-location tracking and shadow service link disk landing in the search system.

[0084] According to some implementations, by building full-tracking capabilities and shadow SP full-link tracking, tracking information of the core search module is obtained. The construction of full-tracking capabilities aims to capture user behavior data through automated means, applicable to both front-end and back-end, and can significantly reduce the cost of manual tracking; shadow SP full-link tracking introduces shadow service providers to simulate real links and collect complete information for performance monitoring, link tracing, and anomaly detection.

[0085] According to some embodiments, the data entry module 105, by recording data entry points and combining it with the relevance determination results, enables the problem attribution module 103 to quickly locate the specific link in the relevance determination error and generate correct knowledge, providing data support for batch production of erroneous knowledge at each link. All the generated knowledge is then updated and consolidated into multiple knowledge bases for the search chain based on the output results of the problem attribution module 103. After accuracy verification and data cleaning by the referee module, it is applied online for knowledge base consumption, improving the accuracy of relevance determination in the search chain.

[0086] According to some embodiments, the knowledge base update module 107 is configured to update multiple knowledge bases used for the search link based on the output results of the problem attribution module.

[0087] According to some embodiments, after the problem attribution module 103 performs relevance problem attribution, it needs to delve into the search system to discover potential problems that need to be solved quickly. The knowledge base update module 107 provides knowledge data support for solving this problem. The multiple knowledge bases include one or more of the following: query extended knowledge base, product extended keyword knowledge base, hypernym / hypernym extended knowledge base, and title attribute conflict knowledge base.

[0088] According to some embodiments, a query extension knowledge base is used to store and manage a database of knowledge related to query extensions. In fields such as information retrieval, when a user's input query terms are short and vague, the query extension knowledge base can find related synonyms, near-synonyms, and related concepts, thereby expanding the original query and improving the accuracy and recall of the retrieval.

[0089] According to some embodiments, a product extended term knowledge base is a knowledge base that stores product-related extended terms, including product names, aliases, attribute words, brand words, category words, and other related terms and their relationships. For example, if a user searches for "ABC athletic shoes" on an e-commerce platform, the product extended term knowledge base can help the system understand that "ABC" is a brand term and "athletic shoes" is a category term. It may also provide synonyms or related terms for athletic shoes such as "running shoes" and "training shoes," thereby more accurately matching products and providing relevant recommendations. Typically, product-related terms are extracted from text data such as product data, product manuals, and user reviews on e-commerce platforms, and then classified, clustered, and analyzed using natural language processing technology to establish a network of relationships between the terms.

[0090] According to some embodiments, the hierarchical extended knowledge base is a knowledge base that primarily stores the hierarchical relationships between concepts. It is used to clarify the hierarchical structure between different concepts, i.e., which concepts are superordinate concepts (more abstract, broader concepts) and which are subordinate concepts (more specific, more detailed concepts).

[0091] According to some embodiments, the title attribute conflict knowledge base is a knowledge base used to record and manage information about conflicts between titles and attributes. In some data management and information processing scenarios, especially those involving product data and document data, there may be inconsistencies or conflicts between the content described in the title and the values ​​of related attribute fields. This knowledge base is used to store this conflict information, as well as related processing rules and solutions.

[0092] According to some embodiments, the knowledge base update module 107 is further configured to: update the query extended knowledge base to increase category recall knowledge when the low-end miss recall is due to the query entry matching but the category is incorrect; update the upper and lower extended knowledge bases to increase category recall knowledge when the low-end miss recall is due to the category word identification error in the query; and update the product extended word knowledge base to increase category recall knowledge when the low-end miss recall is due to the category word identification error in the title.

[0093] According to some embodiments, the search link of the search link inspection system of the present invention includes: a query understanding submodule, a resource understanding submodule, a recall submodule, a relevance determination submodule, a ranking and strategy submodule, and a knowledge base submodule.

[0094] According to some embodiments, the query understanding submodule is configured to analyze user query requests, parse the syntax and semantics of user queries, including identifying entities, attributes and relationships; handling ambiguities and expanding query intent; and interacting with a knowledge base to obtain background knowledge, etc.

[0095] According to some embodiments, the resource understanding submodule is configured to analyze the searched resource, extract the structured attributes of the searched resource, including constructing a semantic relationship network, generating vector representations, and interacting with the knowledge base to update the object model.

[0096] According to some embodiments, the recall submodule is configured to match and retrieve data between the query request and the resource to obtain a candidate result set. The recall submodule quickly filters out the candidate set related to the query from a massive amount of resources, including obtaining entity association information through interaction with a knowledge base.

[0097] According to some embodiments, the relevance determination submodule is configured to calculate a relevance score between the candidate result set and the query request, including but not limited to semantic relevance. The relevance determination submodule is based on a comprehensive relevance scoring model, which includes: a wide network that directly performs width-side fusion on a first input feature, the first input feature including category score, product category score, modifier score, query term-product score, query term-title score, semantic score, brand score, and / or model score from tracking information obtained through tracking; a deep network that runs parallel to the wide network, generating embedding vectors from a second input feature and generating higher-order features, the second input feature including category matching information, product category matching information, modifier matching information, query term-product matching information, query term-title matching information, semantic matching information, brand matching information, and / or model matching information from tracking information obtained through tracking; and an MLP network that calculates and directly outputs the comprehensive relevance score, and obtains relevance tiers through threshold division.

[0098] In some embodiments, the relevance determination submodule labels the training samples using a text processing model. In other embodiments, the relevance determination submodule can construct samples based on existing semantic model training data.

[0099] According to some embodiments, the ranking and strategy submodule is configured to generate a final ranking output based on the candidate result set according to the relevance score and the task strategy. The task strategy includes timeliness, diversity, business weight, etc.

[0100] According to some embodiments, the knowledge base submodule is configured to provide knowledge support to each link of the search chain using the multiple knowledge bases, providing auxiliary information such as synonym expansion, entity linking, knowledge graph, and domain rules to improve semantic consistency.

[0101] As previously stated, this invention provides a method for search link inspection. The method utilizes a relevance determination attribution model to simulate a user's relevance inspection of a search system, thereby identifying relevance determination errors. These errors indicate that the user's query request is inconsistent with the product results returned by the search system. The relevance determination attribution model is based on a text processing model. The method then uses the relevance determination attribution model to attribute the relevance determination errors to the search link, obtaining attribution results. Finally, it updates multiple knowledge bases used for the search link based on the output results of the problem attribution module.

[0102] According to some embodiments, when a low-end miss is attributed to a matching query entry but an incorrect category, the query extended knowledge base is updated to increase category recall knowledge; when a low-end miss is attributed to an incorrect identification of category terms in the query, the upper and lower extended knowledge bases are updated to increase category recall knowledge; and / or when a low-end miss is attributed to an incorrect identification of category terms in the title, the product extended term knowledge base is updated to increase category recall knowledge.

[0103] The underlying dependencies of this invention's search link inspection system include engineering capability development, such as the tracking of relevance information and the development of a debug platform. The underlying dependencies also include integrating with an annotation platform, providing annotation services to outsourced teams, and rapidly feeding back consumed data, ensuring the performance of each link is guaranteed by combining the search link inspection system with manually annotated data verification.

[0104] Figure 2 This diagram illustrates the operation of a correlation determination attribution model for search link inspection according to an example embodiment.

[0105] See Figure 2 According to some embodiments, the generated inspection samples are applied to guide the entire process iteration, thereby generating the ability to attribute bad cases. According to some embodiments, unified labeling can visualize and locate various issues currently labeled with relevance, such as identifying 30+ different types of user-related experience issues.

[0106] According to the example implementation, prompt words are designed by defining knowledge, defining thought chains, performing relevance judgment, specifying output format, clarifying requirements, and providing examples. In this way, the understanding ability of the text processing model can be increased through COT (thought chain), the accuracy of the text processing model can be improved through few-shot (few samples), and the relevance judgment criteria of the text processing model can be internalized by training the model, thereby being used for relevance judgment and attribution.

[0107] After designing the Prompt (providing the text processing model with contextual information and parameter information for the input), products can be randomly selected daily for data construction, cleaning, and labeling. The text processing model then performs search inspections and relevance assessments to obtain relevance determinations, problem explanations, and problem attributions, and performs conflict knowledge mining. Following this, manual sampling evaluation is conducted, and the text processing model is further fine-tuned using SFT (Supervised Fine-Tuning). The evaluated and deployed relevance assessment and attribution model can produce results daily and can be manually validated offline for further model fine-tuning.

[0108] According to some embodiments, 5K supervised fine-tuning (SFT) samples can be constructed by fusing multiple models, and general common sense samples can be constructed for fusion training. By utilizing the output results of multiple existing models, richer and more diverse training data can be generated, thereby improving the generalization ability of the SFT model.

[0109] According to the example implementation, the search link inspection system achieves an accuracy rate of 95.8% compared to manual methods, comparable to the level of outsourced manual labeling. Furthermore, the system can generate 1 million inspection results daily, significantly improving system efficiency.

[0110] Figure 3 This diagram illustrates the process of attributing a problem to a relevant attribution model based on an example embodiment.

[0111] Figure 4 This diagram illustrates the chain of attributing a problem to a relevant attribution model according to another example embodiment.

[0112] See Figure 3 If the current situation is determined to be a high-level misclassification (high relevance score, but actually irrelevant), the first step in attribution is to determine whether it is a semantic relevance error. If it is not a semantic relevance error, then it is determined whether it is a textual relevance error. Specifically, it is determined whether the category is mismatched and whether there is a category prediction error; further, it is determined whether the term is incorrect, with fine-grained attribution including two situations: knowledge base errors and knowledge base missingness, which can be addressed separately; finally, it is determined whether there is an attribute conflict.

[0113] See Figure 4 If the current situation is determined to be a low-level missed recall (low relevance score, but actually relevant), the first step in attribution is to determine whether it belongs to category prediction missed recall. Specifically, this involves determining whether there are misjudged query category terms or missed title category terms. Further, it involves determining whether there are SKU knowledge filtering issues. Finally, it involves determining whether there are missed recalls due to synonyms and hierarchical word mining.

[0114] Figure 5 This illustrates a link diagram for search link attribution based on a relevance determination attribution model according to another example embodiment.

[0115] See Figure 5 , Figure 5Create a link diagram for problem attribution in the problem attribution module. Determine if the problem-information pair (qi pair) needs semantic upgrade; if so, optimize the semantic relevance model. If semantic upgrade is not needed, perform category matching; if the categories do not match, optimize the category prediction model; if the categories match, determine synonyms or hyponyms / hypernyms. If the qi pair has errors in synonyms or hyponyms / hypernyms, it is confirmed as a positive knowledge base problem. Relevant knowledge base data is then confirmed, and text processing models or manual annotation are performed.

[0116] If the `qi` pair is determined to be neither a synonym nor a hyponym / hypernym, it is identified as an internal conflict within the reverse product. Conflict knowledge is mined based on the query term to determine the conflict type: title-title conflict, title-attribute conflict, attribute-attribute conflict, and image-text conflict. For title-title, title-attribute, and attribute-attribute conflicts, text processing models or manual labeling are used. For image-text conflicts, candidate words are obtained from the text side, and a multimodal image-text conflict determination model modifies the `qi` pair.

[0117] According to some embodiments, category prediction-based recall solves the problem of some low-tier missed recall results being missed due to incorrect category prediction tiers, and the low tier being too low due to incorrect category classifications in the low-tier missed recall evaluation data query terms that are terminology matching.

[0118] According to some implementation examples, during the inspection process, the relevance issues in the search chain were found to be as follows: 10% of the missed recall issues were identified as category word recognition problems; 25% of the queries had category word recognition errors; and more than 300,000 queries had misidentified category words. In the title, 75% of the category words were missed due to term weighting. The focus was on mining the category word knowledge that was missed at the product granularity level and incrementally identifying a huge number of incremental product category word terms.

[0119] According to some embodiments, current e-commerce platforms use text-based models to obtain relevance, resulting in low relevance experience levels. Under the current technological system, it is difficult to form a clear optimization direction. Looking towards a future where artificial intelligence (AI) is the core productive force, current rule strategies are no longer applicable. This invention converts all matching operators into instance features that can be input into the model in real time, establishing an overall architecture adapted to the engineering iteration of relevance algorithms for e-commerce platforms. It upgrades the existing relevance optimization framework, iterating from a text-based + model-assisted model to a long-term optimization direction of model-based + text-assisted, taking responsibility for the final error rate.

[0120] Specifically, the previous final relevance was a rule-based link. The upgrade is to make all the rules the model input, build a comprehensive relevance model, and directly output scores for application to online consumption links.

[0121] The search link inspection system of the present invention can effectively improve the accuracy of relevance determination of the text processing model by combining manual annotation and text processing model training technology. Furthermore, the current text processing model determination scheme can output the cause of error to optimize inspection and monitoring and improve the accuracy of outsourcing standards while maintaining a high determination accuracy.

[0122] According to the search link inspection system of the present invention, an accurate relevance model is trained by using the general knowledge of the text processing model and methods such as SFT (Supervised Fine-Tuning). Compared with the previous relevance models, it has stronger understanding ability and the accuracy can reach the level of human. Compared with the previous solutions, the overall accuracy of the text processing model has been qualitatively improved, and it can perform attribution and identification of error causes.

[0123] The search link inspection system of this invention is universal and can accurately perform relevance attribution of text processing models. Combined with the engineering team's construction of a debug platform and full-link data tracking, it can achieve rapid problem location and problem investigation link construction. Furthermore, based on the text processing model, it can perform error identification knowledge base attribution and omission identification knowledge expansion to improve terminology weight and the accuracy of the knowledge base.

[0124] Figure 6 A block diagram of a computing device according to an exemplary embodiment is shown.

[0125] like Figure 6 As shown, the computing device 30 includes a processor 12 and a memory 14. The computing device 30 may also include a bus 22, a network interface 16, and an I / O interface 18. The processor 12, memory 14, network interface 16, and I / O interface 18 can communicate with each other via the bus 22.

[0126] Processor 12 may include one or more general-purpose CPUs (Central Processing Units), microprocessors, or application-specific integrated circuits, for executing relevant program instructions. According to some embodiments, computing device 30 may also include a high-performance display adapter (GPU) 20 for accelerating processor 12.

[0127] Memory 14 may include a machine-readable medium in the form of volatile memory, such as random access memory (RAM), read-only memory (ROM), and / or cache memory. Memory 14 is used to store one or more programs containing instructions, as well as data. Processor 12 may read the instructions stored in memory 14 to perform the methods described above according to embodiments of the present invention.

[0128] The computing device 30 can also communicate with one or more networks via the network interface 16. The network interface 16 can be a wireless network interface.

[0129] Bus 22 can include address bus, data bus, control bus, etc. Bus 22 provides a path for exchanging information between components.

[0130] It should be noted that, in specific implementations, the computing device 30 may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the device described above may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0131] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), network storage devices, cloud storage devices, or any type of medium or device suitable for storing instructions and / or data.

[0132] This invention also provides a computer program product comprising a computer program operable to cause a computer to perform some or all of the steps of any of the methods described in the above method embodiments.

[0133] Those skilled in the art will clearly understand that the technical solutions of the present invention can be implemented by means of software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware capable of independently performing or cooperating with other components to perform a specific function, wherein the hardware may be, for example, a field-programmable gate array (FPGA), an integrated circuit, etc.

[0134] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0135] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0136] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0137] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0138] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0139] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention.

[0140] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0141] Exemplary embodiments of the present invention have been specifically shown and described above. It should be understood that the present invention is not limited to the detailed structures, arrangements, or implementations described herein; rather, the present invention is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended provisions.

Claims

1. A system for searching link inspection, the system comprising an inspection module, a problem attribution module, a point information module, and a knowledge base update module, wherein, The inspection module is configured to simulate a user performing a relevance inspection on the search system, thereby discovering relevance judgment errors. The relevance judgment errors indicate that the user's query request is inconsistent with the query results returned by the search system. The problem attribution module is configured to perform search link attribution on the relevance determination error and locate the search link where the relevance determination error occurred. The tracking information module is configured to provide data support for the problem attribution module by constructing full-link tracking information for full-location tracking and shadow service link disk in the search system. The knowledge base update module is configured to update multiple knowledge bases used in the search chain based on the output results of the problem attribution module.

2. The system according to claim 1, characterized in that, The inspection module and the problem attribution module are constructed in a unified manner based on the correlation determination attribution model, which is trained based on the text processing model.

3. The system according to claim 1, characterized in that, Also includes: A debugging platform is used to manually investigate the aforementioned related issues.

4. The system according to claim 2, characterized in that, Also includes: The referee module is configured to correct the results of the outsourced labeled samples based on the attribution model of the correlation determination, and to evaluate the actual online problem-solving ratio through A / B verification.

5. The system according to claim 2, characterized in that, The multiple knowledge bases include one or more of the following: query extended knowledge base, product extended keyword knowledge base, hypernym / hypernym extended knowledge base, and title attribute conflict knowledge base.

6. The system according to claim 5, characterized in that, The knowledge base update module is configured as follows: In cases where low-level missed recalls are attributed to matching query entries but incorrect categories, update the query extended knowledge base to increase category recall knowledge.

7. The system according to claim 5, characterized in that, The knowledge base update module is configured as follows: If the low-level missed recall is attributed to the misidentification of category terms in the query, update the upper and lower level extended knowledge bases to increase category recall knowledge.

8. The system according to claim 5, characterized in that, The knowledge base update module is configured as follows: In cases where low-end product recalls are attributed to incorrect category term identification in the title, the product extended term knowledge base is updated to increase category recall knowledge.

9. The system according to claim 1, characterized in that, The search chain includes: The query understanding submodule is configured to analyze user query requests; The resource understanding submodule is configured to analyze the searched resources; The recall submodule is configured to match and retrieve the query request and the resource to obtain a candidate result set. The relevance determination submodule is configured to calculate the relevance score between the candidate result set and the query request; The sorting and strategy submodule is configured to generate a final sorting output based on the candidate result set according to the relevance score and the job strategy. The knowledge base submodule is configured to provide knowledge support to each link of the search chain using the multiple knowledge bases.

10. The system according to claim 9, characterized in that, The correlation determination submodule is based on a comprehensive correlation scoring model, which includes: A wide network directly performs width-side fusion on the first input features, which include category score, product category score, modifier score, query term-product score, query term-title score, semantic score, brand score, and / or model score from the tracking information obtained through tracking. A deep network, running in parallel with the wide network, generates embedding vectors and higher-order features from the second input features. The second input features include category matching information, product category matching information, modifier matching information, query term-product matching information, query term-title matching information, semantic matching information, brand matching information, and / or model matching information from the tracking information obtained through tracking. The MLP network calculates and directly outputs the comprehensive relevance score, and obtains the relevance classification by threshold division.

11. A method for searching link inspection, characterized in that, The method includes: The relevance determination attribution model is used to simulate users performing relevance checks on the search system, thereby discovering relevance determination errors. The relevance determination errors indicate that the user's query request is inconsistent with the product results returned by the search system. The relevance determination attribution model is based on a text processing model. The correlation determination attribution model is used to search for the link attribution of the correlation determination error, and the attribution result is obtained. The multiple knowledge bases used for the search chain are updated based on the output of the problem attribution module.

12. The method according to claim 11, characterized in that, The multiple knowledge bases used in the search chain are updated based on the output of the problem attribution module, including: In cases where low-level missed recall is attributed to matching query entries but incorrect categories, update the query extended knowledge base to increase category recall knowledge; and / or In cases where low-level missed recalls are attributed to incorrect category term identification in the query, update the upper and lower level extended knowledge bases to increase category recall knowledge; and / or In cases where low-end product recalls are attributed to incorrect category term identification in the title, the product extended term knowledge base is updated to increase category recall knowledge.

13. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 11-12.

14. A computing device, characterized in that, include: processor; A memory storing a computer program that, when executed by the processor, implements the method as described in any one of claims 11-12.

Images