Training method of large model, gap evaluation method and device based on large model

Abstract

The present disclosure provides a large model training method, a gap evaluation method and device based on a large model, relates to the technical field of computers, especially to the technical fields of artificial intelligence, large models, data mining and the like, and can be applied to scenarios such as content search, generative search, intelligent editing of documents, intelligent assistants, virtual humans and the like. The specific implementation scheme comprises: obtaining a target search term; based on the target search term, performing content search in a specified scene of a target platform to obtain N content search results; for each content search result in the N content search results, using a target large model to obtain a quality evaluation result corresponding to the content search result; and based on the N quality evaluation results corresponding one-to-one to the N content search results, obtaining a content gap evaluation result of the target search term in the specified scene of the target platform. The present disclosure can improve the initiative of content gap evaluation, and at the same time, improve the accuracy of the content gap evaluation result.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, particularly to the fields of artificial intelligence, large models, and data mining, and can be applied to scenarios such as content search, generative search, intelligent document editing, intelligent assistants, and virtual humans. Specifically, it relates to a training method for large models, a gap assessment method based on large models, and an apparatus. Background Technology

[0002] With the rapid development of internet technology, content platforms have become the main entry point for users to obtain information. These platforms can aggregate massive amounts of multimedia resources and, relying on search engine technology, determine content search results from multimedia resources based on user-input search terms, and then recommend them to users. Summary of the Invention

[0003] This disclosure provides a training method for a large model, a gap assessment method based on the large model, and an apparatus.

[0004] According to a first aspect of this disclosure, a method for training a large model is provided, comprising: Obtain multiple search term samples; wherein at least some of the search term samples originate from a specified scenario on the target platform; For each of the multiple search term samples, a training data set corresponding to the search term sample is constructed. The training data set includes the model evaluation results and the ground truth values ​​of the search results samples corresponding to the search term samples. The search results samples are obtained by performing content searches in a specified scenario on the target platform based on the search term samples. The model evaluation results are obtained using the initial large model. Based on the training data set, the initial large model is trained to obtain the target large model.

[0005] According to a second aspect of this disclosure, a gap assessment method based on a large model is provided, comprising: Obtain the target search term; Based on the target search terms, a content search is performed in a specified scenario on the target platform to obtain N content search results; where N≥1 and N is an integer; For each of the N content search results, the target large model is used to obtain the quality assessment result corresponding to the content search result; Based on N quality assessment results that correspond one-to-one with N content search results, the content gap assessment results for the target search term in a specified scenario on the target platform are obtained.

[0006] According to a third aspect of this disclosure, a training apparatus for a large model is provided, comprising: The sample acquisition unit is used to acquire multiple search term samples; wherein at least some of the search term samples are from a specified scenario of the target platform; The data set construction unit is used to construct a training data set corresponding to each search term sample among multiple search term samples. The training data set includes the model evaluation results and the ground truth values ​​of the search results samples corresponding to the search term samples. The search results samples are obtained by performing content searches based on the search term samples in a specified scenario on the target platform. The model evaluation results are obtained using the initial large model. The model training unit is used to train the initial large model based on the training data set to obtain the target large model.

[0007] According to a fourth aspect of this disclosure, a gap assessment apparatus based on a large model is provided, comprising: The search term acquisition unit is used to acquire target search terms; The content search unit is used to perform content searches based on target search terms within a specified scenario on the target platform, and obtain N content search results; where N≥1 and N is an integer; The content evaluation unit is used to obtain a quality evaluation result corresponding to each of the N content search results using the target large model. The content gap assessment unit is used to obtain the content gap assessment result of the target search term in a specified scenario on the target platform, based on N quality assessment results that correspond one-to-one with N content search results.

[0008] According to a fifth aspect of this disclosure, an electronic device is provided, comprising: At least one processor; Memory that is communicatively connected to at least one processor; The memory stores instructions that can be executed by at least one processor, which are executed by at least one processor to enable the at least one processor to perform the method provided in the first aspect of this disclosure.

[0009] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions; wherein the computer instructions are used to cause a computer to perform the method provided in the first aspect of this disclosure.

[0010] According to a seventh aspect of this disclosure, a computer program product is provided, including a computer program; wherein, when executed by a processor, the computer program is capable of implementing the method provided in the first aspect of this disclosure.

[0011] Using this disclosure can improve the initiative in content gap assessment and at the same time improve the accuracy of content gap assessment results.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic diagram of a rich media dual-column scenario for a content platform. Figure 2 A flowchart illustrating a gap assessment method based on a large model provided in this disclosure embodiment; Figure 3 A complete flowchart illustrating a gap assessment method based on a large model, provided for embodiments of this disclosure; Figure 4 An illustrative diagram illustrating an example of a gap assessment method based on a large model provided in this disclosure; Figure 5 A flowchart illustrating a training method for a large model provided in this embodiment of the disclosure; Figure 6 A complete flowchart illustrating a training method for a large model provided in this disclosure embodiment; Figure 7 An illustrative diagram illustrating a training method for a large model provided in an embodiment of this disclosure; Figure 8 A schematic diagram illustrating an application scenario of a gap assessment method and / or a training method for a large model based on an embodiment of this disclosure; Figure 9 A schematic structural block diagram of a gap assessment device based on a large model provided in this disclosure embodiment; Figure 10 A schematic structural block diagram of a training device for a large model provided in an embodiment of this disclosure; Figure 11 This is a schematic structural block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0014] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0015] As mentioned earlier, content platforms can aggregate massive amounts of multimedia resources and, relying on search engine technology, determine content search results from these resources based on user-inputted search terms, then recommend them to the user. In some scenarios on content platforms, there may be multiple content search results. Figure 1 Taking the rich media dual-column scenario shown as an example, it can determine one or more video and / or text / image content search results from multimedia resources based on the user's input search term "bedtime stories for toddlers".

[0016] Currently, in order to determine whether these content search results meet user needs, it is usually necessary to conduct gap assessments, such as quantity gap assessments and / or quality gap assessments.

[0017] However, the inventors' research revealed the following shortcomings in traditional content gap assessment methods: The response mechanism is passive: it usually only conducts a content gap assessment on the content search results after the user's search results are unsuccessful (that is, the number of content search results is 0) or after receiving clear negative feedback from the user, and it is unable to proactively and timely obtain the content gap assessment results. The assessment is limited in scope, focusing primarily on quantity gap assessment while neglecting content quality. This results in generally low-quality and highly homogenized content search results, making it impossible to ensure the accuracy of content gap assessment results.

[0018] To address at least some of the above-mentioned problems, this disclosure provides a gap assessment method based on a large model, which can be applied to both service devices and terminal devices. The service device can be a server, workbench, mainframe computer, or other similar computing device; the terminal device can be a workbench, mainframe computer, conventional computer (e.g., desktop computer, laptop computer, tablet computer, etc.) or other similar computing device. The following will be combined with... Figure 2 The flowchart shown illustrates a gap assessment method based on a large model provided in this disclosure. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described in the flowchart may be performed in a different order.

[0019] Step S201: Obtain the target search term.

[0020] The target search term can originate from a specific scenario on the target platform; that is, the target search term can be a search term (Query) entered by the user in a specific scenario on the target platform. Here, the target platform can be a content platform, such as a social media platform, a self-media platform, or a search platform, while the specified scenario can be a rich media dual-column scenario.

[0021] Target search terms can also come from all platforms. "All platforms" can include all content platforms, and all content platforms include the target platform.

[0022] Step S202: Based on the target search term, perform a content search in the specified scenario of the target platform to obtain N content search results.

[0023] Where N≥1 and N is an integer.

[0024] In this embodiment of the disclosure, the content search results may include at least one of the following: title, specific content, publication time, publishing account, and number of followers of the account.

[0025] In this embodiment of the disclosure, after obtaining the target search term, content search can be performed based on the target search term within a specified scenario of the target platform to obtain N Uniform Resource Locators (URLs), and N content search results can be obtained based on the N URLs. Each of the N content search results corresponds one-to-one with one of the N URLs.

[0026] Step S203: For each of the N content search results, use the target large model to obtain the quality assessment result corresponding to the content search result.

[0027] The target large model can be a large language model (LLM) or a trained LLM. Here, the LLM can be a pre-trained neural network model (e.g., an autoregressive generative model with a Transformer architecture) that possesses general language knowledge, world knowledge, and domain-specific expertise (e.g., expertise in the field of computer technology).

[0028] In this embodiment of the disclosure, the quality assessment result can be a score to characterize the overall quality of the content search results. This score can be obtained by evaluating the content search results from multiple quality assessment dimensions. These multiple quality assessment dimensions can be fixed; for example, they can include at least one of "content relevance," "content accuracy," "demand coverage," and "timeliness." Alternatively, the multiple quality assessment dimensions can be dynamically set based on the target search terms; this embodiment of the disclosure does not impose any limitations on this.

[0029] Step S204: Based on the N quality assessment results that correspond one-to-one with the N content search results, obtain the content gap assessment result of the target search term in the specified scenario of the target platform.

[0030] Among them, the content gap assessment results can be used to characterize whether there is a quantity gap and / or quality gap for the target search term in a specified scenario on the target platform.

[0031] In this embodiment of the disclosure, if the content gap assessment result indicates that there is a quantity gap for the target search term in a specified scenario on the target platform, it indicates that the quantity of the content search results for the target search term in the specified scenario on the target platform needs to be optimized; if the content gap assessment result indicates that there is a quality gap for the target search term in a specified scenario on the target platform, it indicates that the quality of the content search results for the target search term in the specified scenario on the target platform needs to be optimized.

[0032] The gap assessment method based on a large model provided in this disclosure can obtain target search terms from a specified scenario on a target platform. Based on the target search terms, content searches are performed in the specified scenario on the target platform to obtain N content search results. Then, for each of the N content search results, a quality assessment result corresponding to the content search result is obtained using the target large model. Based on the N quality assessment results that correspond one-to-one with the N content search results, a content gap assessment result for the target search terms in the specified scenario on the target platform is obtained. In other words, this disclosure is equivalent to deploying a normalized, proactive content gap assessment mechanism based on a target large model on the target platform. Instead of passively waiting for users to search without results or receiving explicit negative user feedback before initiating content gap assessment, each user-input search term is regarded as a potential content gap assessment opportunity, realizing the proactivity of content gap assessment. At the same time, by introducing a target large model, a deep quality assessment of content search results can be performed, thereby improving the accuracy of content gap assessment results. In this way, the iteration cycle from content release to feedback can be significantly shortened, enabling the target platform to grasp the content supply status in real time and guide the optimization of content quantity and / or content quality in a timely manner, realizing the transformation from "post-event remediation" to "pre-event prevention and in-event control", effectively improving the user experience of the target platform.

[0033] Furthermore, in this embodiment of the present disclosure, when performing step S201, namely, "obtaining target search terms", one can obtain the fourth number of third historical search terms that have been searched the most times in a specified scenario on the target platform within a specified historical period, and use each of the fourth number of third historical search terms as the target search term; one can also obtain the fifth number of fourth historical search terms that have been searched the most times across the entire platform within a specified historical period, and use each of the fifth number of fourth historical search terms as the target search term; one can also obtain the fourth number of third historical search terms that have been searched the most times in a specified scenario on the target platform within a specified historical period, and obtain the fifth number of fourth historical search terms that have been searched the most times across the entire platform within a specified historical period, and use each of the fourth number of third historical search terms and the fifth number of fourth historical search terms as the target search term.

[0034] The specified historical period, the fourth quantity, and the fifth quantity can be set based on the application requirements of the gap assessment method of the large model, and this disclosure does not limit this.

[0035] In some optional implementations, step S203, "using the target large model to obtain the quality assessment result corresponding to the content search results," may include: Get assessment prompts; Using the target large model, and following the evaluation prompts, the content search results are evaluated to obtain quality evaluation results.

[0036] Among them, the evaluation prompt, as a prompt message, can be used to guide the target large model on how to evaluate content search results and obtain quality evaluation results.

[0037] In one example, "Get evaluation tips" could include: Obtain multiple quality assessment dimensions; Obtain multiple quality assessment standards that correspond one-to-one with multiple quality assessment dimensions; Based on multiple quality assessment dimensions and multiple quality assessment standards, assessment prompts are generated.

[0038] As mentioned above, in this embodiment of the disclosure, multiple quality assessment dimensions can be fixed. For example, multiple quality assessment dimensions can include at least one of "content relevance", "content accuracy", "demand coverage" and "timeliness". Multiple quality assessment dimensions can also be dynamically set based on target search terms, and this embodiment of the disclosure does not limit this.

[0039] In a specific example, when multiple quality assessment dimensions are dynamically set based on target search terms, the setting method can be: Obtain the parsed results of the target search term; Based on the analysis results, multiple quality assessment dimensions are obtained.

[0040] In a more specific example, an intent parsing model can be used to parse the target search term and obtain the parsing results. The intent parsing model can be an LLM (Limited Language Model); the parsing results can include multiple intent tags to describe the target search term from different descriptive dimensions. These multiple intent tags can include at least one of the following: intent type, search term domain, timeliness requirement, sentiment tendency, demand depth, demand form, demand granularity, product entity, product attribute, and demand dimension.

[0041] For example, if the target search term is "the reason for today's Bitcoin crash", the parsed results can include the following intent tags: Intent type: Information query / Cause-and-effect interpretation; Search term: Finance / Cryptocurrency; Timeliness requirement: High (today); Sentiment: Negative; Demand depth: Professional analysis.

[0042] For example, if the target search term is "braised pork recipe", the parsing results can include the following intent tags: Intent type: Method tutorial / step guide; Search term: food / cooking; Time sensitivity requirement: Low (classic recipes); Required format: video and / or text / image format; Requirements granularity: detailed steps.

[0043] For example, if the target search term is "buy a pair of men's running shoes", the parsing results can include the following intent tags: Intent type: Shopping decision / product comparison; Search term category: e-commerce / sports equipment; Product: Running shoes; Product Attributes: Men's; Demand dimensions: cost-effectiveness, comfort, brand.

[0044] In this embodiment of the disclosure, after obtaining the parsing results of the target search term, multiple quality assessment dimensions can be obtained based on the parsing results. In a more specific example, at least one basic assessment dimension can be obtained (for example, at least one basic assessment dimension may include "content relevance"), and based on the parsing results, at least one additional assessment dimension is matched from a pre-established "intent tag-quality assessment dimension" mapping library. Then, based on at least one basic assessment dimension and at least one additional assessment dimension, multiple quality assessment dimensions are obtained. For example, at least one basic assessment dimension and at least one additional assessment dimension can be used together as multiple quality assessment dimensions.

[0045] For example, if the parsing result includes "Timeliness requirement: High", based on the parsing result, at least one additional evaluation dimension matching from the pre-established "intent tag - quality assessment dimension" mapping library includes "timeliness". Therefore, the basic evaluation dimension "content relevance" and the additional evaluation dimension "timeliness" can be used together as multiple quality assessment dimensions. If the parsing result includes "intent type: method tutorial / step guide", based on the parsing result, at least one additional evaluation dimension matching from the pre-established "intent tag - quality assessment dimension" mapping library includes "step completeness / operability". Therefore, the basic evaluation dimension of "content relevance" and the additional evaluation dimension of "step completeness / operability" can be used together as multiple quality evaluation dimensions. When the parsing results include "intent type: shopping decision / product comparison" and "search term domain: e-commerce", based on the parsing results, at least one additional evaluation dimension that is matched from the pre-established "intent tag-quality evaluation dimension" mapping library includes "parameter coverage / positive and negative word-of-mouth". Therefore, the basic evaluation dimension of "content relevance" and the additional evaluation dimension of "parameter coverage / positive and negative word-of-mouth" can be used together as multiple quality evaluation dimensions.

[0046] In this embodiment of the disclosure, after obtaining multiple quality assessment dimensions, multiple quality assessment standards corresponding one-to-one with the multiple quality assessment dimensions can be obtained. In a specific example, multiple quality assessment standards can be matched from a pre-established "quality assessment dimension-quality assessment standard" mapping library based on the multiple quality assessment dimensions. The multiple quality assessment standards correspond one-to-one with the multiple quality assessment dimensions.

[0047] For example, the corresponding quality assessment criteria for "content relevance" could be: 3 points: The content search results are strongly related to the target search term. For example, more than 60% of the results described in the content are consistent with the main or core entity in the target search term.

[0048] 2 points: The content search results are partially related to the target search term. For example, more than 20% of the results described in the content are consistent with the main or core entity in the target search term.

[0049] 1 point: The content search results are weakly related to the target search term. For example, the results described in the content do not answer the question raised by the target search term, but they are of reference value.

[0050] 0 points: The content search results are completely irrelevant to the target search term. For example, the results described in the content do not match the main or core entity of the target search term.

[0051] For example, the corresponding quality assessment criteria for "timeliness" could be: 3 points: The search results include content related to the time point directly or indirectly referred to by the target search term; 0 points: The content is outdated, resulting in invalid search results that do not meet user needs.

[0052] In this embodiment of the disclosure, after obtaining multiple quality assessment dimensions and multiple quality assessment standards corresponding one-to-one with each quality assessment dimension, an assessment prompt can be generated based on the multiple quality assessment dimensions and multiple quality assessment standards. In one specific example, an assessment prompt including multiple quality assessment dimensions and multiple quality assessment standards can be generated; in another specific example, assessment examples can be obtained, and assessment prompts can be generated based on multiple quality assessment dimensions, multiple quality assessment standards, and assessment examples. For example, an assessment prompt including multiple quality assessment dimensions, multiple quality assessment standards, and assessment examples can be generated. There can be M1 assessment examples, and these M1 examples can be positive, used to better guide the target large model on how to evaluate content search results based on multiple quality assessment dimensions and multiple quality assessment standards to obtain quality assessment results. Here, 1 ≤ M1 ≤ 3, and M1 is an integer.

[0053] In this embodiment of the disclosure, after obtaining the evaluation prompts, the target large model can be used to evaluate the content search results according to the evaluation prompts to obtain a quality evaluation result. In one example, the target large model can be used to evaluate the content search results according to the evaluation prompts to obtain multiple single-dimensional evaluation results corresponding one-to-one with multiple quality evaluation dimensions. These multiple single-dimensional evaluation results are then weighted and fused to obtain the final quality evaluation result. The multiple weight parameters used for weighting and fusing the multiple single-dimensional evaluation results can be set based on the parsing results of the target search terms or based on the application requirements of the gap assessment method of the large model; this embodiment of the disclosure does not impose any limitations on this. Here, the multiple weight parameters correspond one-to-one with the multiple single-dimensional evaluation results.

[0054] Through the above methods, in this embodiment of the disclosure, evaluation prompts can be obtained, and the target large model can be used to evaluate the content search results according to the evaluation prompts to obtain quality evaluation results. In other words, in this embodiment of the disclosure, by introducing evaluation prompts as guidance information, the complex quality evaluation task can be transformed into an instruction form that the target large model can easily understand. This fully utilizes the powerful language understanding and knowledge reasoning capabilities of the target large model to achieve automated evaluation of content search results. This approach eliminates the need to retrain the target large model for each evaluation scenario, significantly reducing the cost of quality evaluation, while ensuring the flexibility and generalization ability of the quality evaluation process.

[0055] Furthermore, in this embodiment of the disclosure, when obtaining evaluation prompts, multiple quality evaluation dimensions and multiple quality evaluation standards corresponding one-to-one with each quality evaluation dimension can be obtained. Evaluation prompts are then generated based on these multiple quality evaluation dimensions and standards. Moreover, when obtaining multiple quality evaluation dimensions, the parsing results of the target search term can be obtained, and multiple quality evaluation dimensions can be obtained based on these results. In other words, in this embodiment of the disclosure, by performing intent parsing on the target search term and dynamically adapting the quality evaluation dimensions based on the parsing results, a precise alignment between the evaluation perspective and user needs can be achieved. For example, for search terms with high timeliness requirements, the "timeliness" quality evaluation dimension is automatically activated; for search intent types such as method tutorials / step guides, the "step completeness / operability" quality evaluation dimension is automatically activated. This avoids quality evaluation bias caused by using a fixed set of dimensions, improving not only the comprehensiveness of the quality evaluation results but also their accuracy.

[0056] Furthermore, in this embodiment of the disclosure, when generating evaluation prompts based on multiple quality assessment dimensions and multiple quality assessment standards, evaluation examples can be obtained, and evaluation prompts can be generated based on multiple quality assessment dimensions, multiple quality assessment standards, and evaluation examples. In other words, in this embodiment of the disclosure, by introducing evaluation examples into the evaluation prompts, an intuitive "few-shot learning" reference can be provided to the target large model, enabling it to more accurately understand the specific application of quality assessment standards in different content search results. This helps to standardize the output format of the target large model, reduce ambiguity in the target model's understanding of abstract quality assessment standards, and further improve the consistency and accuracy of quality assessment results.

[0057] In some optional implementations, step S204, namely, "obtaining the content gap assessment result of the target search term in a specified scenario on the target platform based on the N quality assessment results corresponding one-to-one with the N content search results," can be: obtaining the content gap assessment result of the target search term in a specified scenario on the target platform based on the N content search results and the N quality assessment results corresponding one-to-one with the N content search results. Specifically, it can include: Get the total number of search results for N content items; Based on N quality assessment results, determine the percentage of content search results that meet the quality assessment requirements among the N content search results; Based on the total number and the proportion of the number, the content gap assessment results are obtained.

[0058] The quality assessment requirement can be that the quality assessment result is greater than or equal to a preset score. Here, the preset score can be set based on the parsing results of the target search term, or it can be set based on the application requirements of the gap assessment method of the large model. For example, the preset score can be set to 2, and this embodiment of the disclosure does not limit it.

[0059] In one example, "obtaining the content gap assessment result based on the total number and the percentage of the total number" can include one of the following: When the total quantity is less than or equal to the quantity threshold, the first gap assessment result is obtained; When the total number is greater than the number threshold and the number percentage is less than or equal to the percentage threshold, the second gap assessment result is obtained; The third gap assessment result is obtained when the total quantity is less than or equal to the quantity threshold and the quantity percentage is less than or equal to the percentage threshold.

[0060] The quantity threshold can be determined based on the popularity level of the target search term.

[0061] In a specific example, the higher the popularity level of the target search term, the higher the numerical value can be set as the quantity threshold; conversely, the lower the popularity level of the target search term, the lower the numerical value can be set as the quantity threshold. The popularity level can be determined based on the number of times the target search term is searched on the target platform or across all platforms. For example, popularity can be positively correlated with the number of times the target search term is searched on the target platform or across all platforms; that is, the more times the target search term is searched on the target platform or across all platforms, the higher its popularity; and the fewer times the target search term is searched on the target platform or across all platforms, the lower its popularity.

[0062] In this embodiment of the disclosure, the percentage threshold can be set based on the application requirements of the gap assessment method for large models. For example, the preset score can be set to 30%, and this embodiment of the disclosure does not limit this.

[0063] Furthermore, it should be noted that in this embodiment of the disclosure, the first gap assessment result can be used to characterize a quantitative gap for the target search term in a specified scenario on the target platform; the second gap assessment result can be used to characterize a qualitative gap for the target search term in a specified scenario on the target platform; and the third gap assessment result can be used to characterize both a quantitative and qualitative gap for the target search term in a specified scenario on the target platform. It should also be noted that in this embodiment of the disclosure, when a content search is performed based on the target search term in a specified scenario on the target platform, and the number of content search results obtained is 0, the content gap assessment result for the target search term in the specified scenario on the target platform can be directly determined as the first gap assessment result to characterize a quantitative gap for the target search term in the specified scenario on the target platform.

[0064] Through the above methods, in this embodiment of the disclosure, the total number of N content search results can be obtained, and based on the N quality assessment results, the proportion of content search results that meet the quality assessment requirements can be determined. Then, based on the total number and the proportion, the content gap assessment result is obtained. In other words, in this embodiment of the disclosure, a comprehensive quantification of the content supply status can be achieved by comprehensively considering the total number of content search results and the proportion of high-quality content search results. This dual assessment mechanism of quantity and quality indicators effectively avoids the problem of "sufficient quantity but poor quality" caused by relying solely on quantity indicators. It can more accurately identify the real content gaps of the N content search results in terms of both "availability" and "quality," providing precise data support for subsequent content optimization (quantity optimization and / or quality optimization).

[0065] Furthermore, in this embodiment of the disclosure, when obtaining the content gap assessment result based on the total quantity and the quantity percentage, a first gap assessment result can be obtained if the total quantity is less than or equal to a quantity threshold; a second gap assessment result can be obtained if the total quantity is greater than the quantity threshold and the quantity percentage is less than or equal to a percentage threshold; and a third gap assessment result can be obtained if the total quantity is less than or equal to the quantity threshold and the quantity percentage is less than or equal to a percentage threshold. The first gap assessment result characterizes a quantity gap for the target search term in a specified scenario on the target platform; the second gap assessment result characterizes a quality gap for the target search term in a specified scenario on the target platform; and the third gap assessment result characterizes both a quantity gap and a quality gap for the target search term in a specified scenario on the target platform. In other words, in this embodiment of the disclosure, a hierarchical gap discrimination logic can be constructed by setting quantity thresholds and percentage thresholds, thereby clearly distinguishing between quantity gaps, quality gaps, and composite gaps where both coexist. This refined classification mechanism allows the target platform to adopt differentiated content optimization strategies for different types of content gaps—either quantity optimization, quality optimization, or a combination of both—significantly improving the targeting and efficiency of content supply optimization.

[0066] Furthermore, the gap assessment method based on a large model provided in this disclosure may also include: Based on the content gap assessment results, generate pass-through information; Based on the content gap assessment results, target devices were identified; Transmit the information to the target device.

[0067] Among them, "generating pass-through information based on content gap assessment results" can be either using the content gap assessment results as pass-through information, or using the content gap assessment results as one of the elements for generating pass-through information and generating pass-through information accordingly.

[0068] In one example, "using the content gap assessment results as one of the elements in generating pass-through information, and generating pass-through information accordingly" can include: Obtain impact assessment data related to the target search term; The impact of the gap is determined based on the impact assessment data; Based on the content gap assessment results and the impact of the gap, pass-through information is generated.

[0069] The impact assessment data may include page views (PV) and / or consumer information of historical search results obtained during a first historical period, based on the target search term and conducted across the entire platform. Here, the first historical period can be set based on the application requirements of the gap assessment method for large-scale models, and this embodiment does not impose any limitations on it; the consumer information may include at least one of user age distribution, user geographic coverage, and user occupation type.

[0070] Based on this, in this embodiment of the disclosure, when determining the gap impact based on the impact assessment data, if the impact assessment data includes PV (Page Views) volume, an impact characterization value positively correlated with PV volume can be obtained as the gap impact. Alternatively, if the impact assessment data includes PV volume and consumer user information, a first impact factor can be obtained based on PV volume, and a second impact factor can be obtained based on consumer user information. The gap impact can then be obtained based on the first and second impact factors. Specifically, PV volume can be normalized to obtain a first impact factor located in the numerical range [0, 1]. Simultaneously, user profile analysis can be performed on consumer user information to determine the breadth of user coverage, which serves as the second impact factor. For example, the wider the user age distribution, the greater the user geographic coverage, and the richer the user occupation types, the higher the value of the second impact factor will be; conversely, the lower the value of the second impact factor will be. After obtaining the first and second impact factors, a weighted fusion of the first and second impact factors can be performed to obtain the gap impact. The first weight (corresponding to the first influence factor) and the second weight (corresponding to the second influence factor) used for weighted fusion of the first influence factor and the second influence factor can be set based on the application requirements of the gap assessment method of the large model, and this embodiment of the disclosure does not impose any restrictions on this. Here, the first weight corresponds to the first influence factor; the second weight corresponds to the second influence factor.

[0071] In this embodiment of the disclosure, after obtaining the impact assessment data related to the target search term and determining the gap impact based on the impact assessment data, transparent information can be generated based on the content gap assessment results and the gap impact. For example, the content gap assessment results and the gap impact can be used together as transparent information.

[0072] In another example, "using the content gap assessment results as one of the elements in generating pass-through information, and generating pass-through information accordingly" can include: If the proportion of a given number of search results is less than or equal to the proportion threshold, then the search results that do not meet the quality assessment requirements among the N search results are identified as low-quality search results. Determine multiple single-dimensional evaluation results for low-quality search results; Supplementary instructions are generated based on multiple single-dimensional evaluation results; Based on the content gap assessment results and supplementary instructions, pass-through information is generated.

[0073] Among them, multiple single-dimensional evaluation results correspond one-to-one with multiple quality evaluation dimensions.

[0074] In this embodiment of the disclosure, "generating pass-through information based on content gap assessment results and supplementary instructions" can mean using both the content gap assessment results and supplementary instructions as pass-through information.

[0075] In this embodiment of the disclosure, the above two examples can also be combined to generate pass-through information. Specifically, impact assessment data related to the target search term can be obtained, and the gap impact can be determined based on the impact assessment data. Simultaneously, if the determined quantity percentage is less than or equal to a percentage threshold, content search results that do not meet the quality assessment requirements among N content search results are identified as low-quality search results. Multiple single-dimensional assessment results for the low-quality search results are then determined, and supplementary instructions are generated based on these multiple single-dimensional assessment results. Finally, pass-through information is generated based on the content gap assessment results, gap impact, and supplementary instructions. For example, the content gap assessment results, gap impact, and supplementary instructions can be used together as pass-through information.

[0076] In this embodiment of the disclosure, after generating pass-through information based on the content gap assessment results, the target device can be identified based on the content gap assessment results, and the pass-through information can be sent to the target device. The target device can be the terminal device of the target responsible party (e.g., a workbench, mainframe computer, conventional computer, or other similar computing device). After obtaining the pass-through information based on the target device, the target responsible party can determine which content optimization strategy to adopt to optimize the N content search results corresponding to the target search term.

[0077] For example, if the transparent information includes content gap assessment results, and these results are actually the first gap assessment results, used to characterize a quantity gap for the target search term in a specified scenario on the target platform, then the target device can be the first terminal device of the content purchaser, who is the party responsible for the target search term. After obtaining the transparent information based on the first terminal device, the party responsible for the target search term can determine a "quantity optimization" content optimization strategy based on this information, optimizing the N content search results corresponding to the target search term, that is, guiding the generation of more content search results for the target search term.

[0078] For example, if the transparent information includes content gap assessment results, and these results are actually secondary gap assessment results, used to characterize a quality gap in the target search term within a specified scenario on the target platform, then the target device can be the second terminal device of the content producer acting as the target responsible party. After obtaining the transparent information through the second terminal device, the target responsible party can determine a "quality optimization" content optimization strategy based on this information, optimizing the N content search results corresponding to the target search term. For instance, after clarifying the optimization direction based on supplementary instructions, it can guide the generation of high-quality content search results for the target search term.

[0079] For example, if the transparent information includes content gap assessment results, and these results are actually third-party gap assessment results, used to characterize the quantity and quality gaps of the target search term in a specified scenario on the target platform, then the target devices can include a first terminal device (the content purchaser, acting as the first target responsible party) and a second terminal device (the content producer, acting as the second target responsible party). After learning the transparent information based on the first terminal device, the first target responsible party can determine to adopt a "quantity optimization" content optimization strategy, optimizing the N content search results corresponding to the target search term, i.e., guiding the generation of more content search results for the target search term. Similarly, after learning the transparent information based on the second terminal device, the second target responsible party can determine to adopt a "quality optimization" content optimization strategy, optimizing the N content search results corresponding to the target search term. For example, after clarifying the optimization direction based on supplementary instructions, it can guide the generation of high-quality content search results for the target search term.

[0080] Through the above methods, in this embodiment of the disclosure, transparent information can be generated based on the content gap assessment results, and the target device can be determined based on the content gap assessment results. Then, the transparent information is sent to the target device, so that the target responsible party can determine what content optimization strategy to adopt based on the transparent information to optimize the N content search results corresponding to the target search term. In other words, in this embodiment, the content gap assessment results can be converted into transparent information and sent to the target device, realizing a closed-loop link from content gap identification to optimization instruction issuance. This allows content purchasers and / or content producers to promptly know the content supply defects under specific search terms and take targeted content optimization strategies accordingly. This effectively solves the problem of the disconnect between content gap assessment results and optimization actions in traditional solutions, and improves the adaptability and response efficiency of the target platform's content supply system.

[0081] Furthermore, in this embodiment of the disclosure, when generating pass-through information based on the content gap assessment results, impact assessment data related to the target search term can be obtained, and the gap impact can be determined based on the impact assessment data. Then, pass-through information is generated based on the content gap assessment results and the gap impact. In other words, in this embodiment of the disclosure, by introducing impact assessment data related to the target search term, the impact of the target platform's content gap on the target platform's user experience and commercial value can be quantified. Moreover, the pass-through information can distinguish the priority of content gaps for different search terms, guiding the responsible party to prioritize the handling of content gaps corresponding to high-popularity and high-value search terms, thereby achieving the most rational allocation of limited optimization resources.

[0082] Furthermore, in this embodiment of the disclosure, when generating pass-through information based on the content gap assessment results, if the proportion of content search results is less than or equal to a threshold, content search results that do not meet the quality assessment requirements among N content search results can be identified as low-quality search results. Multiple single-dimensional assessment results for these low-quality search results are then determined, and supplementary instructions are generated based on these results. Pass-through information is then generated based on the content gap assessment results and the supplementary instructions. In other words, this embodiment of the disclosure can further determine low-quality search results and their specific defects in each quality assessment dimension based on the identified quality gap, and generate supplementary instructions accordingly. This ensures that the pass-through information not only informs of the existence of a quality gap but also clarifies "which quality assessment dimensions the quality gap specifically manifests in," providing the responsible party with a clear and actionable optimization direction, significantly improving the accuracy and efficiency of content quality optimization.

[0083] The following, combined with Figure 3 The complete process of a gap assessment method based on a large model provided in the embodiments of this disclosure is described.

[0084] Step S301: Obtain the target search term.

[0085] Step S302: Based on the target search term, perform a content search in the specified scenario of the target platform to obtain N content search results.

[0086] Where N≥1 and N is an integer.

[0087] Step S303: For each content search result among the N content search results, obtain multiple quality assessment dimensions; obtain multiple quality assessment standards that correspond one-to-one with the multiple quality assessment dimensions; generate assessment prompts based on the multiple quality assessment dimensions, multiple quality assessment standards, and assessment examples; and use the target large model to assess the content search results according to the assessment prompts to obtain the quality assessment results.

[0088] Step S304: Obtain the total number of N content search results.

[0089] Step S305: Based on the N quality assessment results, determine the percentage of content search results that meet the quality assessment requirements among the N content search results.

[0090] Furthermore, in this embodiment of the disclosure, step S306 can be executed when the total quantity is less than or equal to the quantity threshold; step S307 can be executed when the total quantity is greater than the quantity threshold and the quantity percentage is less than or equal to the percentage threshold; and step S308 can be executed when the total quantity is less than or equal to the quantity threshold and the quantity percentage is less than or equal to the percentage threshold.

[0091] Step S306 yields the first gap assessment result.

[0092] The first gap assessment result is used to characterize the quantity gap of the target search term in a specified scenario on the target platform.

[0093] Step S307 yields the second gap assessment result.

[0094] The second gap assessment result is used to characterize the quality gap of the target search term in the specified scenario of the target platform.

[0095] Step S308 yields the third gap assessment result.

[0096] The third gap assessment result is used to characterize whether there is a quantity gap or a quality gap for the target search term in a specified scenario on the target platform.

[0097] Step S309: Based on the content gap assessment results, generate pass-through information; based on the content gap assessment results, determine the target device; and send the pass-through information to the target device.

[0098] In this embodiment of the disclosure, the specific functions and examples of the above steps can be found in the relevant descriptions of the corresponding steps in the aforementioned embodiment of the gap assessment method based on a large model, and will not be repeated here.

[0099] The following will combine Figure 4 The present invention will further illustrate a gap assessment method based on a large model through specific examples.

[0100] (1) Obtaining target search terms and content search results Retrieve the fifth number of fourth historical search terms that were searched most frequently across the entire platform within a specified historical period (e.g., including "authentic way to make cabbage vermicelli"), and use each of the fifth number of fourth historical search terms as the target search term.

[0101] Suppose that when the fourth historical search term "authentic way to make cabbage and vermicelli" is used as the target search term, a content search is performed based on this term within a specific scenario on the target platform, resulting in 30 URLs. Based on these 30 URLs, 30 content search results are obtained. Further suppose that these 30 URLs include URLs (https: / / ...), and based on these URLs (https: / / ...), the resulting content search results include: Title: If you love Chinese cabbage, you must save this recipe! Learn how to make this Chinese cabbage and glass noodle dish – it tastes amazing! Specific steps include: cutting Chinese cabbage into petals, soaking in cold water, and steaming with glass noodles; Release date: October 20, 2023; Posted by: XXYYZZ; Number of followers: 1006.

[0102] (2) Quality assessment of content search results For each of the 30 content search results, the target large model is used to obtain the quality assessment result corresponding to the content search result.

[0103] The quality assessment result is the score value located in the numerical range [0, 3].

[0104] (3) Obtain the content gap assessment results of the target search term in the specified scenario of the target platform. Suppose that for the target search term "authentic way to make cabbage vermicelli", the quantity threshold is 50, the quantity percentage is 30%, and the quality assessment requirement is that the quality assessment result is greater than or equal to a preset score of 2. Further suppose that, based on the target search term "authentic way to make cabbage vermicelli", in a specified scenario on the target platform, a content search is conducted, and among the 30 content search results obtained, the percentage of content search results that meet the quality assessment requirement is 16.7%. Therefore, it can be determined that the total number of content search results (30) is less than or equal to the quantity threshold of 50, and the quantity percentage (16.7%) is less than or equal to the percentage threshold of 30%. Thus, a third gap assessment result can be obtained to characterize the existence of both a quantity gap and a quality gap for the target search term "authentic way to make cabbage vermicelli" in the specified scenario on the target platform.

[0105] (4) Transmit downstream Based on the content gap assessment results (i.e., the third gap assessment results), pass-through information is generated, and based on the content gap assessment results, the target device is determined, and then the pass-through information is sent to the target device.

[0106] Among them, the third gap assessment results are used to characterize the quantity gap and quality gap of the target search term "authentic way to make cabbage vermicelli" in the specified scenario of the target platform.

[0107] The target devices may include a first terminal device, acting as the content purchaser and the second terminal device, acting as the content producer and the second target party. Based on this, we have: The first target responsible party: After obtaining the transparent information based on the first terminal device, it can determine the content optimization strategy of "quantity optimization" based on the transparent information, and optimize the content search results corresponding to the target search term "authentic way to make cabbage and vermicelli", that is, guide the generation of more content search results for the target search term "authentic way to make cabbage and vermicelli". The second target responsible party: After obtaining the transparent information based on the second terminal device, it can determine the content optimization strategy of "quality optimization" based on the transparent information, and optimize the content search results corresponding to the target search term "authentic way to make cabbage vermicelli". For example, after clarifying the optimization direction based on the supplementary instructions, it can guide the generation of high-quality content search results for the target search term "authentic way to make cabbage vermicelli".

[0108] In this embodiment of the disclosure, after performing the gap assessment method based on the large model described above, the content search results that meet the quality assessment requirements among the 30 content search results corresponding to the target search term "authentic way to make cabbage and vermicelli" can be regarded as high-quality search results, so as to help improve the recommendation accuracy of content search results when users search for "authentic way to make cabbage and vermicelli" in a specified scenario on the target platform.

[0109] As previously described, in this embodiment of the disclosure, the target large model can be a trained LLM. In this case, this embodiment also provides a method for training a large model, which can be applied to both service devices and terminal devices. The service device can be a server, workbench, mainframe computer, or other similar computing device; the terminal device can be a workbench, mainframe computer, conventional computer, or other similar computing device. The following will be combined with… Figure 5 The flowchart shown illustrates a method for training a large model according to an embodiment of this disclosure. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described in the flowchart may be performed in a different order.

[0110] Step S501: Obtain multiple search term samples.

[0111] Among these, at least some of the search term samples originate from a specific scenario on the target platform. That is, at least some of the search term samples can be search terms entered by users within a specific scenario on the target platform. Here, the target platform can be a content platform, such as a social media platform, a self-media platform, or a search platform, while the specified scenario can be a rich media dual-column scenario.

[0112] Step S502: For each search term sample among multiple search term samples, construct a training data set corresponding to the search term sample.

[0113] The training data set may include model evaluation results and ground truth values ​​of the evaluation results for the search results samples corresponding to the search term samples.

[0114] In this embodiment of the disclosure, the search result sample can be obtained by performing a content search within a specified scenario on the target platform based on the search term sample, including at least one of the following: title sample, specific content sample, publication time sample, publishing account sample, and account follower count sample. In this embodiment of the disclosure, after obtaining multiple search term samples, for each search term sample, a content search can be performed within the specified scenario on the target platform based on the search term sample to obtain a URL sample, and then a search result sample can be obtained based on the URL sample.

[0115] In this embodiment of the disclosure, the model evaluation result can be obtained using an initial large model. The model evaluation result can be a score value used to characterize the overall quality of the search result sample, which can be obtained by evaluating the search result sample from multiple quality evaluation dimension samples. Here, the multiple quality evaluation dimension samples can be fixed; for example, the multiple quality evaluation dimension samples can include at least one of "content relevance," "content accuracy," "demand coverage," and "timeliness." Alternatively, the multiple quality evaluation dimension samples can be dynamically set based on search term samples; this embodiment of the disclosure does not limit this.

[0116] In this embodiment of the disclosure, the true value of the evaluation result can be the evaluation labeling result obtained through manual evaluation. The evaluation labeling result can be a score value used to characterize the overall quality of the search result sample, which can be obtained by evaluating the search result sample from multiple quality evaluation dimensions.

[0117] Step S503: Based on the training data set, train the initial large model to obtain the target large model.

[0118] It should be noted that in this embodiment, step S503 can be executed cyclically. Specifically, when executing step S502, multiple training data sets corresponding one-to-one with multiple search term samples are obtained. Subsequently, when executing step S503, the initial large model (initial LLM) can be trained using the first training data set among the multiple training data sets to obtain the target large model. Then, the target large model is used as a new initial large model, and the first training data set among the multiple training data sets is used to train it, and so on, to obtain the target large model for the first training round.

[0119] The target large model from the first training round can be used to execute the gap assessment method. This target large model can also serve as a new initial large model, and through iterative execution of steps S501, S502, and S503, it can be trained again to obtain a new target large model for executing the gap assessment method. Here, the multiple search term samples used in iterative execution of steps S501, S502, and S503 are different from those used in the first training round.

[0120] The large-scale model training method provided in this disclosure allows for the acquisition of multiple search term samples (at least some of which originate from a specified scenario on the target platform). For each of these samples, a training data set is constructed. This training data set includes model evaluation results and ground truth values ​​for search results corresponding to the search term samples. The search results are obtained by performing content searches based on the search term samples within a specified scenario on the target platform. The model evaluation results are obtained using an initial large-scale model. Finally, the initial large-scale model is trained using the training data set to obtain a target large-scale model. In other words, this disclosure allows for the construction of a training data set containing model evaluation results and ground truth values ​​based on search term samples from a specified scenario on the target platform. This data set is then used to specifically train the initial large-scale model, resulting in a target large-scale model. As a trained initial large-scale model, the target large-scale model not only retains its original general language knowledge, world knowledge, and domain-specific expertise but also gains a deep adaptation capability to the specific content gap assessment criteria of the specified scenario through specialized learning of search results samples. In this way, when applying the target big model to the gap assessment method, we can more accurately understand the user needs of specific scenarios and output content gap assessment results that are more in line with the business expectations of the target platform, thereby providing a more reliable basis for subsequent content optimization decisions.

[0121] In some optional implementations, step S501, namely, "obtaining multiple search term samples", may include: Retrieve multiple primary search terms from a specified scenario on the target platform; Based on multiple primary search terms, multiple search term samples were obtained; Alternatively, multiple secondary search terms from across the platform can be obtained to generate multiple search term samples based on multiple primary and secondary search terms.

[0122] In one example, "retrieving multiple primary search terms from a specified scenario on the target platform" could include: Retrieve multiple first historical search terms from a specified scenario on the target platform; From multiple first historical search terms, identify the first number of first historical search terms that were searched the most, and the second number of first historical search terms that were searched the least. Based on a first number of first historical search terms and a second number of second historical search terms, multiple first search terms are obtained.

[0123] The multiple first historical search terms can be all search terms that have been used to search for content within a specified scenario on the target platform, based on the target search term, during the second historical time period. Here, the second historical time period can be set based on the application requirements of the large model's training method, and this embodiment does not impose any limitations on it.

[0124] In this embodiment of the disclosure, the first quantity and the second quantity can be set based on the application requirements of the training method for large models. For example, the first quantity and the second quantity can be set to 10. This embodiment of the disclosure does not limit this.

[0125] In this embodiment of the disclosure, after obtaining multiple first historical search terms from a specified scenario of the target platform, and determining the first number of first historical search terms with the most searched times and the second number of first historical search terms with the fewest searched times from the multiple first historical search terms, multiple first search terms can be obtained based on the first number of first historical search terms and the second number of second historical search terms. For example, the first number of first historical search terms and the second number of second historical search terms can be used together as multiple first search terms.

[0126] In this embodiment of the disclosure, after obtaining multiple first search terms from a specified scenario of the target platform, multiple search term samples can be obtained based on the multiple first search terms. For example, the multiple first search terms can be used as multiple search term samples. Alternatively, multiple second search terms from all platforms can be obtained, and multiple search term samples can be obtained based on the multiple first search terms and the multiple second search terms. For example, the multiple first search terms and the multiple second search terms can be used together as multiple search term samples. In one example, "obtaining multiple second search terms from all platforms" may include: Obtain multiple secondary historical search terms from across the entire platform; From a plurality of second historical search terms, a third number of second historical search terms are randomly selected; Based on the third number of second historical search terms, multiple second search terms are obtained.

[0127] The multiple second historical search terms can be all search terms that have been used to search for content across the entire platform based on the target search term during the third historical time period. Here, the third historical time period can be set based on the application requirements of the training method of the large model, and this embodiment does not limit it.

[0128] In this embodiment of the disclosure, after obtaining multiple second historical search terms from the entire platform, a third number of second historical search terms can be randomly selected from the multiple second historical search terms, and multiple second search terms can be obtained based on the third number of second historical search terms. For example, the third number of second historical search terms can be used as multiple second search terms. The third number can be set based on the application requirements of the training method of the large model. For example, the third number can be set to 10, and this embodiment of the disclosure does not limit this.

[0129] Through the above methods, in this embodiment of the disclosure, multiple first search terms from a specified scenario of the target platform can be obtained, and multiple search term samples can be obtained based on the multiple first search terms; or, multiple second search terms from the entire platform can be obtained, and multiple search term samples can be obtained based on the multiple first search terms and the multiple second search terms. In other words, in this embodiment of the disclosure, by obtaining first search terms from a specified scenario of the target platform and combining them with second search terms from the entire platform to form a search term sample set, it ensures that the training data specifically covers the specified scenario of the target platform, while also introducing the broad semantic diversity of the entire platform. This multi-source fusion sample construction strategy enables the trained target model to accurately adapt to the requirements of the specified scenario while still maintaining good generalization ability to general search intents.

[0130] Furthermore, in this embodiment of the disclosure, when obtaining multiple first search terms from a specified scenario of the target platform, multiple first historical search terms from the specified scenario of the target platform can be obtained. From these multiple first historical search terms, a first number of first historical search terms with the highest search frequency and a second number of first historical search terms with the lowest search frequency are determined. Then, based on the first number of first historical search terms and the second number of second historical search terms, multiple first search terms are obtained. In other words, in this embodiment of the disclosure, by selecting the historical search terms with the highest and lowest search frequency from the first historical search terms of a specified scenario of the target platform as first search terms, a training sample set including top-level hot keywords and bottom-level long-tail keywords is constructed. This sampling strategy covering both extremes of search term frequency ensures that the target large model can learn the evaluation patterns of high-frequency common queries while also mastering the processing capabilities of low-frequency scarce queries, thus avoiding overfitting of the target large model to popular keywords and neglecting long-tail needs.

[0131] Furthermore, in this embodiment of the disclosure, when obtaining multiple second search terms from the entire platform, multiple second historical search terms from the entire platform can be obtained, and a third number of second historical search terms can be randomly selected from these multiple second historical search terms. Based on these third number of second historical search terms, multiple second search terms are then obtained. In other words, in this embodiment of the disclosure, by randomly selecting a third number of search terms from the second historical search terms from the entire platform as second search terms, a set of unbiased and diverse general search term samples is introduced. This random sampling method can effectively supplement the search term types that may be missing on the target platform, enrich the semantic coverage of the training data, and thus enhance the robustness and transferability of the target large model when dealing with cross-domain and cross-scenario search terms.

[0132] Furthermore, it should be noted that in this embodiment of the disclosure, before executing step S502, that is, "constructing a training data set corresponding to each search term sample among multiple search term samples", the following can be done: Based on search term samples, content search is performed in a specified scenario on the target platform to obtain multiple candidate search results with a recommendation order. From multiple candidate search results, select the third-ranked candidate search results that are recommended in the order of priority; Based on the third number of candidate search results, a sample of search results is obtained.

[0133] The third quantity can be set based on the application requirements of the training method for large models. For example, the third quantity can be set to 10, but this embodiment does not limit it.

[0134] In this embodiment of the disclosure, based on the search term sample, a content search is performed in a specified scenario of the target platform to obtain multiple candidate search results with a recommendation order. After selecting the third number of candidate search results with the highest recommendation order from the multiple candidate search results, a search result sample can be obtained based on the third number of candidate search results. For example, each candidate search result in the third candidate search results can be used as the result sample.

[0135] Through the above methods, in this embodiment of the disclosure, content search can be performed based on search term samples within a specified scenario of the target platform to obtain multiple candidate search results with a recommendation order. From these multiple candidate search results, the third-ranked candidate search results are selected, and then a search result sample is obtained based on this third-ranked candidate search result. In other words, in this embodiment of the disclosure, content search can be performed based on search term samples within a specified scenario of the target platform, and the candidate search results with the highest recommendation order can be selected as the search result sample. This ensures that the training data originates from the actual search ranking logic of the target platform and has high relevance and representativeness.

[0136] In some optional implementations, to obtain the model evaluation results of the search result samples corresponding to the search term samples, step S502, "constructing the training data set corresponding to the search term samples," may include: Obtain a sample of assessment prompts; Using the initial large model, the search result samples are evaluated according to the evaluation prompt samples to obtain the model evaluation results.

[0137] Among them, the evaluation prompt samples serve as prompt information, which can be used to guide the initial large model on how to evaluate the search result samples and obtain the model evaluation results.

[0138] In one example, "Get evaluation prompt sample" may include: Obtain samples from multiple quality assessment dimensions; Obtain multiple quality assessment standard samples that correspond one-to-one with samples from multiple quality assessment dimensions; Based on samples from multiple quality assessment dimensions and multiple quality assessment standards, an assessment prompt sample is generated.

[0139] As mentioned above, in this embodiment of the disclosure, the multiple quality assessment dimension samples can be fixed. For example, the multiple quality assessment dimension samples can include at least one of "content relevance", "content accuracy", "demand coverage" and "timeliness". The multiple quality assessment dimension samples can also be dynamically set based on search term samples, and this embodiment of the disclosure does not limit this.

[0140] In a specific example, when multiple quality assessment dimension samples are dynamically set based on search term samples, the setting method can be: Obtain the parsed results of the search term sample; Based on the parsed sample results, samples for multiple quality assessment dimensions are obtained.

[0141] In a more specific example, an intent parsing model can be used to parse the intent of search term samples, resulting in parsed result samples. The intent parsing model can be an LLM (Limited Linear Model); the parsed result samples can include multiple intent tag samples to describe the search term samples from different descriptive dimensions. These multiple intent tag samples can include at least one of the following: intent type, search term domain, timeliness requirement, sentiment tendency, demand depth, demand form, demand granularity, product entity, product attribute, and demand dimension.

[0142] In this embodiment of the disclosure, after obtaining the parsing result sample of the search term sample, multiple quality assessment dimension samples can be obtained based on the parsing result sample. In a more specific example, at least one basic assessment dimension sample can be obtained (for example, at least one basic assessment dimension sample may include "content relevance"), and based on the parsing result sample, at least one additional assessment dimension sample is matched from a pre-established "intent tag sample - quality assessment dimension sample" mapping library. Then, based on at least one basic assessment dimension sample and at least one additional assessment dimension sample, multiple quality assessment dimension samples are obtained. For example, at least one basic assessment dimension sample and at least one additional assessment dimension sample can be used together as multiple quality assessment dimension samples.

[0143] In this embodiment of the disclosure, after obtaining multiple quality assessment dimension samples, multiple quality assessment standard samples corresponding one-to-one with the multiple quality assessment dimension samples can be obtained. In a specific example, multiple quality assessment standard samples can be matched from a pre-established "quality assessment dimension sample - quality assessment standard sample" mapping library based on the multiple quality assessment dimension samples. The multiple quality assessment standard samples correspond one-to-one with the multiple quality assessment dimension samples.

[0144] In this embodiment of the disclosure, after obtaining multiple quality assessment dimension samples and multiple quality assessment standard samples corresponding one-to-one with the multiple quality assessment dimension samples, an assessment prompt sample can be generated based on the multiple quality assessment dimension samples and the multiple quality assessment standard samples. In one specific example, an assessment prompt sample including multiple quality assessment dimension samples and multiple quality assessment standard samples can be generated; in another specific example, assessment example samples can be obtained, and an assessment prompt sample can be generated based on the multiple quality assessment dimension samples, multiple quality assessment standard samples, and assessment example samples. For example, an assessment prompt sample including multiple quality assessment dimension samples, multiple quality assessment standard samples, and assessment examples can be generated. There can be M² assessment example samples, which can be positive, to better guide the initial large model on how to evaluate the search result samples based on the multiple quality assessment dimension samples and multiple quality assessment standard samples to obtain the model evaluation result. Here, 1 ≤ M² ≤ 3, and M² is an integer.

[0145] In this embodiment of the disclosure, after obtaining the evaluation prompt samples, the initial large model can be used to evaluate the search result samples according to the evaluation prompt samples to obtain the model evaluation result. In one example, the initial large model can be used to evaluate the model search results according to the evaluation prompt samples to obtain multiple single-dimensional evaluation result samples that correspond one-to-one with the multiple quality evaluation dimension samples. These multiple single-dimensional evaluation result samples are then weighted and fused to obtain the model evaluation result. The multiple weight parameter samples used for weighting and fusing the multiple single-dimensional evaluation result samples can be set based on the parsed result samples of the search term samples or based on the application requirements of the large model's training method; this embodiment of the disclosure does not impose any limitations on this. Here, the multiple weight parameter samples correspond one-to-one with the multiple single-dimensional evaluation result samples.

[0146] Through the above methods, in this embodiment of the disclosure, evaluation prompt samples can be obtained, and the initial large model can be used to evaluate the search result samples according to the evaluation prompt samples to obtain the model evaluation results. In this embodiment of the disclosure, by introducing evaluation prompt samples as guidance information, the complex quality evaluation task can be transformed into an instruction form that is easy for the initial large model to understand. By making full use of the powerful language knowledge, world knowledge, and professional knowledge of various fields of the initial large model, the automated evaluation of search result samples can be achieved, thereby improving the accuracy of the model evaluation results.

[0147] Furthermore, in this embodiment of the disclosure, when obtaining evaluation prompt samples, multiple quality evaluation dimension samples can be obtained, along with multiple quality evaluation standard samples corresponding one-to-one with the multiple quality evaluation dimension samples. Evaluation prompt samples are then generated based on the multiple quality evaluation dimension samples and the multiple quality evaluation standard samples. Moreover, when obtaining multiple quality evaluation dimension samples, parsing result samples of search term samples can be obtained, and multiple quality evaluation dimension samples can be obtained based on the parsing result samples. In other words, in this embodiment of the disclosure, by performing intent parsing on search term samples and dynamically adapting quality evaluation dimension samples based on the parsing result samples, precise alignment between the evaluation perspective and user needs can be achieved. For example, for search terms with high timeliness requirements, the "timeliness" quality evaluation dimension sample is automatically activated; for search intent types such as method tutorials / step guides, the "step completeness / operability" quality evaluation dimension sample is automatically activated. This avoids quality evaluation bias caused by using a fixed dimension set, not only improving the comprehensiveness of the model evaluation results but also further improving the accuracy of the model evaluation results.

[0148] Furthermore, in this embodiment of the disclosure, when generating evaluation prompt samples based on multiple quality assessment dimension samples and multiple quality assessment standard samples, evaluation example samples can be obtained, and evaluation prompt samples can be generated based on multiple quality assessment dimension samples, multiple quality assessment standard samples, and evaluation example samples. In other words, in this embodiment of the disclosure, by introducing evaluation example samples into the evaluation prompt samples, an intuitive "few-shot learning" reference can be provided to the initial large model, enabling it to more accurately understand the specific application of quality assessment standard samples on different search result samples. This helps to standardize the output format of the initial large model, reduce ambiguity in the initial model's understanding of abstract quality assessment standard samples, and further improve the consistency and accuracy of the model's evaluation results.

[0149] In some optional implementations, to obtain the true value of the evaluation result of the search result sample corresponding to the search term sample, step S502, "constructing the training data set corresponding to the search term sample", may include: Obtain the evaluation and annotation results of the search result samples; Obtain user behavior feedback results based on the search results samples; The true value of the evaluation result is obtained based on the evaluation annotation results and user behavior feedback results.

[0150] The evaluation and labeling results can be obtained through manual evaluation. Here, the labeling and evaluation results can be numerical scores used to characterize the overall quality of the search result samples, which can be obtained by evaluating the search result samples from multiple quality evaluation dimensions.

[0151] In one example, "obtaining user behavior feedback results for a sample of search results" may include: Obtain samples from multiple quality assessment dimensions; Obtain multiple single-dimensional behavioral feedback results that correspond one-to-one with samples from multiple quality assessment dimensions; User behavior feedback results are obtained based on multiple single-dimensional behavioral feedback results.

[0152] In a specific example, after obtaining multiple quality assessment dimension samples, multiple behavioral feedback dimensions can be matched from a pre-established "quality assessment dimension sample - behavioral feedback dimension" mapping library based on these samples. Each behavioral feedback dimension corresponds one-to-one with a quality assessment dimension sample.

[0153] In this embodiment, after acquiring multiple quality assessment dimension samples and matching multiple behavioral feedback dimensions from a pre-established "quality assessment dimension sample - behavioral feedback dimension" mapping library based on these samples, multiple behavioral feedback indicators corresponding one-to-one with each behavioral feedback dimension can be obtained. Based on these indicators, multiple single-dimensional behavioral feedback results corresponding one-to-one with each quality assessment dimension sample can be obtained. Specifically, for each behavioral feedback dimension, at least one behavioral feedback indicator can be collected. For example, for the quality assessment dimension sample of "content relevance," the corresponding behavioral feedback dimension could be "user click behavior," and the corresponding behavioral feedback indicators could include the click-through rate of search result samples, the average dwell time after clicking, and the interaction rate after clicking. Therefore, the click-through rate, average dwell time after clicking, and interaction rate can be normalized. Then, the normalization results are weighted and summed to obtain the single-dimensional behavioral feedback result corresponding to the "content relevance" quality assessment dimension sample.

[0154] Similarly, after obtaining multiple single-dimensional behavioral feedback results corresponding one-to-one with samples from multiple quality assessment dimensions, multiple behavioral feedback weights corresponding one-to-one with these single-dimensional behavioral feedback results can be obtained. Based on the multiple single-dimensional behavioral feedback results and their weights, a weighted sum is calculated to obtain the user behavioral feedback result. The weights for these multiple behavioral feedback results can be set based on the application requirements of the training method for the large model; this embodiment does not impose any limitations on this.

[0155] Through the above methods, evaluation annotation results of search result samples can be obtained, along with user behavior feedback results for these samples. Based on these evaluation annotation results and user behavior feedback results, the true value of the evaluation result can then be derived. In other words, this embodiment of the disclosure combines manually obtained evaluation annotation results with user behavior feedback results to jointly construct the true value of the evaluation result, achieving a complementary advantage between subjective judgment and objective behavior. The evaluation annotation results ensure the professionalism and standardization of the quality assessment, while user behavior feedback accurately reflects market acceptance and user experience. The fusion of these two effectively avoids biases caused by a single data source, providing higher-quality supervision signals that are closer to actual business needs for the initial large-scale model training, thereby improving training effectiveness.

[0156] Furthermore, when obtaining user behavior feedback results for search result samples, multiple quality assessment dimension samples can be obtained, along with multiple single-dimensional behavior feedback results corresponding one-to-one with these quality assessment dimension samples. Then, based on these multiple single-dimensional behavior feedback results, the user behavior feedback result is obtained. In other words, in this embodiment, by establishing a mapping relationship between quality assessment dimension samples and behavior feedback dimensions, and collecting multi-indicator behavior data under each behavior feedback dimension, a quantitative expression of user behavior under fine-grained quality assessment dimension samples is achieved. This multi-dimensional deconstruction approach ensures that the user behavior feedback result is no longer a general aggregated indicator, but rather accurately reflects the true performance of content across different quality assessment dimension samples, providing more interpretive data support for subsequent integration with assessment annotation results.

[0157] In some optional implementations, step S503, namely, "training the initial large model based on the training data set to obtain the target large model," may include: Based on the model evaluation results and the true values ​​of the evaluation results, the overall evaluation loss is obtained; Based on the overall evaluation loss, the initial large model is trained to obtain the target large model.

[0158] In this embodiment of the disclosure, when obtaining the overall evaluation loss based on the model evaluation result and the true value of the evaluation result, the loss of the model evaluation result relative to the true value of the evaluation result can be obtained as the overall evaluation loss; alternatively: Based on the model evaluation results and the true values ​​of the evaluation results, the first evaluation loss is obtained; For each quality assessment dimension sample among multiple quality assessment dimension samples, the second assessment loss is obtained based on the first single-dimensional assessment result corresponding to the quality assessment dimension sample and the second single-dimensional assessment result corresponding to the quality assessment dimension sample. Based on the first assessment loss and multiple second assessment losses corresponding one-to-one with samples of multiple quality assessment dimensions, the overall assessment loss is obtained.

[0159] In one example, "obtaining the first evaluation loss based on the model evaluation result and the true value of the evaluation result" can be the loss of the model evaluation result relative to the true value of the evaluation result, which is used as the first evaluation loss; "obtaining the second evaluation loss based on the first single-dimensional evaluation result corresponding to the quality evaluation dimension sample and the second single-dimensional evaluation result corresponding to the quality evaluation dimension sample" can be the loss of the first single-dimensional evaluation result corresponding to the quality evaluation dimension sample relative to the second single-dimensional evaluation result corresponding to the quality evaluation dimension sample, which is used as the second evaluation loss.

[0160] After obtaining the first evaluation loss based on the model evaluation results and the true values ​​of the evaluation results, and obtaining the second evaluation loss for each quality evaluation dimension sample among multiple quality evaluation dimension samples based on the first single-dimensional evaluation result and the second single-dimensional evaluation result corresponding to the quality evaluation dimension sample, the overall evaluation loss can be obtained based on the first evaluation loss and multiple second evaluation losses. For example, the first evaluation loss and multiple second evaluation losses can be weighted and fused to obtain the overall evaluation loss. The multiple fusion weights used for weighting and fusing the first evaluation loss and multiple second evaluation losses (including the first weight corresponding to the first evaluation loss and multiple second weights corresponding one-to-one with the multiple second evaluation losses) can be set based on the parsing result samples of the search term samples, or based on the application requirements of the training method of the large model. This disclosure embodiment does not limit this.

[0161] Through the above methods, in this embodiment of the disclosure, the overall evaluation loss can be obtained based on the model evaluation results and the true values ​​of the evaluation results. Based on this overall evaluation loss, the initial large model is trained to obtain the target large model. In other words, in this embodiment of the disclosure, the overall evaluation loss can be constructed by using the loss between the model evaluation results and the true values ​​of the evaluation results as the overall evaluation loss. Training the initial large model based on this overall evaluation loss achieves targeted optimization of the model output towards the real-world annotations. This end-to-end supervised learning mechanism enables the target large model to continuously narrow the gap with human evaluation standards, gradually improving the accuracy of quality evaluation of search result samples. Moreover, the calculation method of the overall evaluation loss is simple, which can improve model training efficiency.

[0162] Furthermore, when obtaining the overall evaluation loss based on the model evaluation results and the true values ​​of the evaluation results, a first evaluation loss can be obtained based on the model evaluation results and the true values ​​of the evaluation results. For each quality evaluation dimension sample among multiple quality evaluation dimension samples, a second evaluation loss is obtained based on the first single-dimensional evaluation result corresponding to the quality evaluation dimension sample and the second single-dimensional evaluation result corresponding to the quality evaluation dimension sample. Then, based on the first evaluation loss and multiple second evaluation losses corresponding one-to-one with multiple quality evaluation dimension samples, the overall evaluation loss is obtained. In other words, in this embodiment of the disclosure, when calculating the overall evaluation loss, both the first evaluation loss for the total score dimension and the second evaluation loss for each quality evaluation dimension sample are introduced simultaneously, realizing joint constraints on the model output at both the macro-level and micro-level. This multi-task learning strategy not only requires the model to provide an accurate comprehensive score but also requires the alignment of its single-dimensional evaluation results with the corresponding true values ​​for each quality evaluation dimension sample, thereby enabling the target large model to learn a more fine-grained and structured quality evaluation capability.

[0163] The following, combined with Figure 6 The complete process of a training method for a large model provided in the embodiments of this disclosure will be described.

[0164] Step S601: Obtain multiple first search terms from a specified scenario on the target platform.

[0165] In one example, step S601, namely, "obtaining multiple first search terms from a specified scenario on the target platform", may include: Retrieve multiple first historical search terms from a specified scenario on the target platform; From multiple first historical search terms, identify the first number of first historical search terms that were searched the most, and the second number of first historical search terms that were searched the least. Based on a first number of first historical search terms and a second number of second historical search terms, multiple first search terms are obtained.

[0166] Step S602: Obtain multiple secondary search terms from across the entire platform.

[0167] In one example, step S602, namely, "obtaining multiple second search terms from across the platform", may include: Obtain multiple secondary historical search terms from across the entire platform; From a plurality of second historical search terms, a third number of second historical search terms are randomly selected; Based on the third number of second historical search terms, multiple second search terms are obtained.

[0168] Step S603: Based on multiple first search terms and multiple second search terms, multiple search term samples are obtained.

[0169] Furthermore, in this embodiment of the disclosure, steps S604, S605, S606, S607 and S608 can be performed for each of the multiple search term samples.

[0170] Step S604: Based on the search term sample, perform content search in the specified scenario of the target platform to obtain multiple candidate search results with a recommendation order; select the third number of candidate search results with the highest recommendation order from the multiple candidate search results; obtain a search result sample based on the third number of candidate search results.

[0171] Step S605: Obtain multiple quality assessment dimension samples; obtain multiple quality assessment standard samples that correspond one-to-one with the multiple quality assessment dimension samples; generate assessment prompt samples based on the multiple quality assessment dimension samples, multiple quality assessment standard samples, and assessment example samples; use the initial large model to evaluate the search result samples according to the assessment prompt samples to obtain the model evaluation results.

[0172] Step S606: Obtain the true value of the evaluation result of the search result sample corresponding to the search term sample.

[0173] Step S607: Based on the model evaluation results and the true values ​​of the evaluation results of the search result samples corresponding to the search term samples, construct the training data set corresponding to the search term samples.

[0174] Step S608: Based on the training data set, train the initial large model to obtain the target large model.

[0175] Please combine Figure 7In summary, in this embodiment, firstly, multiple first search terms from a specified scenario on the target platform and multiple second search terms from the entire platform are obtained through search term sampling, so as to obtain multiple search term samples based on the multiple first search terms and multiple second search terms; then, a downsampling process is performed, that is, for each search term sample in the multiple search term samples, content search is performed in the specified scenario on the target platform based on the search term sample to obtain multiple candidate search results with a recommendation order, and from the multiple candidate search results, the third number of candidate search results with the highest recommendation order are selected, and then a search result sample is obtained based on the third number of candidate search results; finally, the initial large model is used to evaluate the search result sample according to the evaluation prompt sample to obtain the model evaluation result, and the true value of the evaluation result of the search result sample corresponding to the search term sample is obtained, and then a training data set corresponding to the search term sample is constructed based on the model evaluation result and the true value of the evaluation result of the search result sample corresponding to the search term sample, so as to train the initial large model based on the training data set to obtain the target large model.

[0176] Furthermore, it should be noted that the specific functions and examples of the above steps in the embodiments of this disclosure can be found in the relevant descriptions of the corresponding steps in the aforementioned large model training method embodiments, and will not be repeated here.

[0177] Further, please refer to Figure 8 This is a schematic diagram illustrating an application scenario of a gap assessment method and / or a training method for a large model based on an embodiment of this disclosure.

[0178] The gap assessment method and / or large model training method provided in this disclosure are applied to electronic devices. The electronic device can be a service device or a terminal device. Here, the service device can be a server, workbench, mainframe computer, or other similar computing device; the terminal device can be a workbench, mainframe computer, conventional computer, or other similar computing device.

[0179] When applying large-model-based gap assessment methods to electronic devices, the electronic devices are used for: Obtain the target search term; Based on the target search terms, a content search is performed in a specified scenario on the target platform to obtain N content search results; where N≥1 and N is an integer; For each of the N content search results, the target large model is used to obtain the quality assessment result corresponding to the content search result; Based on N quality assessment results that correspond one-to-one with N content search results, the content gap assessment results for the target search term in a specified scenario on the target platform are obtained.

[0180] When applying large model training methods to electronic devices, the electronic devices are used for: Obtain multiple search term samples; wherein at least some of the search term samples originate from a specified scenario on the target platform; For each of the multiple search term samples, a training data set corresponding to the search term sample is constructed. The training data set includes the model evaluation results and the ground truth values ​​of the search results samples corresponding to the search term samples. The search results samples are obtained by performing content searches in a specified scenario on the target platform based on the search term samples. The model evaluation results are obtained using the initial large model. Based on the training data set, the initial large model is trained to obtain the target large model.

[0181] It should be noted that, in the embodiments disclosed herein, Figure 8 The application scenario diagrams shown are for illustrative purposes only and are not restrictive. Those skilled in the art can use them as a basis for their own interpretation. Figure 8 The examples may be modified in various obvious ways and / or substitutions, and the resulting technical solutions still fall within the scope of the disclosure of the embodiments of this disclosure.

[0182] To better implement the gap assessment method based on large models, this disclosure also provides a gap assessment device based on large models, which can be integrated into an electronic device. The electronic device can be a service device or a terminal device. Here, the service device can be a server, workbench, mainframe computer, or other similar computing device; the terminal device can be a workbench, mainframe computer, conventional computer, or other similar computing device. The following will be combined with… Figure 9 The schematic block diagram shown illustrates a gap assessment device 900 based on a large model provided in the disclosed embodiment.

[0183] A gap assessment device 900 based on a large model includes: Search term acquisition unit 901 is used to acquire target search terms; Content search unit 902 is used to perform content search based on target search terms and in a specified scenario on the target platform to obtain N content search results; Content evaluation unit 903 is used to obtain a quality evaluation result corresponding to each content search result in N content search results using the target large model; The content gap assessment unit 904 is used to obtain the content gap assessment result of the target search term in a specified scenario on the target platform based on N quality assessment results that correspond one-to-one with N content search results.

[0184] In some alternative implementations, the content evaluation unit 903 is used for: Get assessment prompts; Using the target large model, and following the evaluation prompts, the content search results are evaluated to obtain quality evaluation results.

[0185] In some alternative implementations, the content evaluation unit 903 is used for: Obtain multiple quality assessment dimensions; Obtain multiple quality assessment standards that correspond one-to-one with multiple quality assessment dimensions; Based on multiple quality assessment dimensions and multiple quality assessment standards, assessment prompts are generated.

[0186] In some alternative implementations, the content evaluation unit 903 is used for: Obtain the parsed results of the target search term; Based on the analysis results, multiple quality assessment dimensions are obtained.

[0187] In some alternative implementations, the content evaluation unit 903 is used for: Get an evaluation example; Based on multiple quality assessment dimensions, multiple quality assessment standards, and assessment examples, assessment prompts are generated.

[0188] In some alternative implementations, the content gap assessment unit 904 is used for: Get the total number of search results for N content items; Based on N quality assessment results, determine the percentage of content search results that meet the quality assessment requirements among the N content search results; Based on the total number and the proportion of the number, the content gap assessment results are obtained.

[0189] In some alternative implementations, the content gap assessment unit 904 is used for one of the following: If the total number is less than or equal to the number threshold, the first gap assessment result is obtained; wherein, the first gap assessment result is used to characterize the existence of a quantity gap for the target search term in a specified scenario on the target platform; When the total number is greater than the number threshold and the number percentage is less than or equal to the percentage threshold, the second gap assessment result is obtained; wherein, the second gap assessment result is used to characterize that the target search term has a quality gap in the specified scenario of the target platform; When the total number is less than or equal to the quantity threshold and the quantity percentage is less than or equal to the percentage threshold, the third gap assessment result is obtained; the third gap assessment result is used to characterize the existence of quantity gap and quality gap of the target search term in the specified scenario of the target platform.

[0190] In some alternative implementations, the large-model-based gap assessment device 900 further includes an indication transmission unit for: Based on the content gap assessment results, generate pass-through information; Based on the content gap assessment results, target devices were identified; Transmit the information to the target device.

[0191] In some alternative implementations, the instruction transparent transmission unit is used for: Obtain impact assessment data related to the target search term; wherein, the impact assessment data includes the page views of historical search results and / or the consumer information of the historical search results obtained by conducting content searches on the entire platform based on the target search term within a first historical period. The impact of the gap is determined based on the impact assessment data; Based on the content gap assessment results and the impact of the gap, pass-through information is generated.

[0192] In some alternative implementations, the instruction transparent transmission unit is used for: If the proportion of a given number of search results is less than or equal to the proportion threshold, then the search results that do not meet the quality assessment requirements among the N search results are identified as low-quality search results. Determine multiple single-dimensional evaluation results for low-quality search results; where each single-dimensional evaluation result corresponds one-to-one with a multiple quality evaluation dimension. Supplementary instructions are generated based on multiple single-dimensional evaluation results; Based on the content gap assessment results and supplementary instructions, pass-through information is generated.

[0193] In this embodiment of the disclosure, the specific functions and examples of each unit in the gap assessment device 900 based on the large model can be found in the relevant descriptions of the corresponding steps in the aforementioned gap assessment method embodiment based on the large model, and will not be repeated here.

[0194] To better implement large-scale model training methods, this disclosure also provides a large-scale model training apparatus, which can be integrated into an electronic device. The electronic device can be a service device or a terminal device. Here, the service device can be a server, workbench, mainframe computer, or other similar computing device; the terminal device can be a workbench, mainframe computer, conventional computer, or other similar computing device. The following will be combined with… Figure 10 The schematic block diagram shown illustrates a training device 1000 for a large model provided in a disclosed embodiment.

[0195] The training device for the large model, 1000, includes: The sample acquisition unit 1001 is used to acquire multiple search term samples; wherein at least some of the multiple search term samples are from a specified scenario of the target platform; The data set construction unit 1002 is used to construct a training data set corresponding to each search term sample among multiple search term samples. The training data set includes the model evaluation results and the ground truth values ​​of the search results samples corresponding to the search term samples. The search results samples are obtained by performing content searches based on the search term samples in a specified scenario on the target platform. The model evaluation results are obtained using the initial large model. Model training unit 1003 is used to train the initial large model based on the training data set to obtain the target large model.

[0196] In some optional implementations, the sample acquisition unit 1001 is used for: Retrieve multiple primary search terms from a specified scenario on the target platform; Based on multiple primary search terms, multiple search term samples were obtained; Alternatively, multiple secondary search terms from across the platform can be obtained to generate multiple search term samples based on multiple primary and secondary search terms.

[0197] In some optional implementations, the sample acquisition unit 1001 is used for: Retrieve multiple first historical search terms from a specified scenario on the target platform; From multiple first historical search terms, identify the first number of first historical search terms that were searched the most, and the second number of first historical search terms that were searched the least. Based on a first number of first historical search terms and a second number of second historical search terms, multiple first search terms are obtained.

[0198] In some optional implementations, the sample acquisition unit 1001 is used for: Obtain multiple secondary historical search terms from across the entire platform; From a plurality of second historical search terms, a third number of second historical search terms are randomly selected; Based on the third number of second historical search terms, multiple second search terms are obtained.

[0199] In some alternative implementations, the data group construction unit 1002 is used for: Obtain a sample of assessment prompts; Using the initial large model, the search result samples are evaluated according to the evaluation prompt samples to obtain the model evaluation results.

[0200] In some alternative implementations, the data group construction unit 1002 is used for: Obtain samples from multiple quality assessment dimensions; Obtain multiple quality assessment standard samples that correspond one-to-one with samples from multiple quality assessment dimensions; Based on samples from multiple quality assessment dimensions and multiple quality assessment standards, an assessment prompt sample is generated.

[0201] In some alternative implementations, the data group construction unit 1002 is used for: Obtain the parsed results of the search term sample; Based on the parsed sample results, samples for multiple quality assessment dimensions are obtained.

[0202] In some alternative implementations, the data group construction unit 1002 is used for: Obtain an evaluation sample; Based on samples from multiple quality assessment dimensions, multiple quality assessment standards, and assessment examples, an assessment prompt sample is generated.

[0203] In some alternative implementations, the model training unit 1003 is used for: Based on the model evaluation results and the true values ​​of the evaluation results, the overall evaluation loss is obtained; Based on the overall evaluation loss, the initial large model is trained to obtain the target large model.

[0204] In some alternative implementations, the model training unit 1003 is used for: Based on the model evaluation results and the true values ​​of the evaluation results, the first evaluation loss is obtained; For each quality assessment dimension sample among multiple quality assessment dimension samples, the second assessment loss is obtained based on the first single-dimensional assessment result corresponding to the quality assessment dimension sample and the second single-dimensional assessment result corresponding to the quality assessment dimension sample. Based on the first assessment loss and multiple second assessment losses corresponding one-to-one with samples of multiple quality assessment dimensions, the overall assessment loss is obtained.

[0205] In some alternative implementations, the data group construction unit 1002 is used for: Obtain the evaluation and annotation results of the search result samples; Obtain user behavior feedback results based on the search results samples; The true value of the evaluation result is obtained based on the evaluation annotation results and user behavior feedback results.

[0206] In some alternative implementations, the data group construction unit 1002 is used for: Obtain samples from multiple quality assessment dimensions; Obtain multiple single-dimensional behavioral feedback results that correspond one-to-one with samples from multiple quality assessment dimensions; User behavior feedback results are obtained based on multiple single-dimensional behavioral feedback results.

[0207] In some alternative implementations, the training apparatus 1000 for the large model also includes a search result acquisition unit for: Before constructing the training data set corresponding to the search term samples, based on the search term samples, content search is performed in a specified scenario on the target platform to obtain multiple candidate search results with a recommendation order; From multiple candidate search results, select the third-ranked candidate search results that are recommended in the order of priority; Based on the third number of candidate search results, a sample of search results is obtained.

[0208] In this embodiment of the disclosure, the specific functions and examples of each unit in the large model training device 1000 can be found in the relevant descriptions of the corresponding steps in the aforementioned large model training method embodiments, and will not be repeated here.

[0209] The collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0210] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0211] Figure 11 A schematic structural block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. Electronic device 1100 is intended to represent various forms of digital computers, such as in-vehicle computing devices, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 1100 may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0212] like Figure 11As shown, the electronic device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. The RAM 1103 may also store various programs and data required for the operation of the electronic device 1100. The computing unit 1101, ROM 1102, and RAM 1103 are interconnected via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0213] Multiple components in electronic device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, etc.; output unit 1107, such as various types of renderers, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows electronic device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0214] The computing unit 1101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose AI computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1101 performs the various methods and processes described above, such as gap estimation methods and / or large model training methods. For example, in some embodiments, gap estimation methods and / or large model training methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by computing unit 1101, one or more steps of the large model-based gap assessment method and / or large model training method described above can be performed. Alternatively, in other embodiments, computing unit 1101 can be configured for the large model-based gap assessment method and / or large model training method by any other suitable means (e.g., by means of firmware).

[0215] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0216] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data optimization device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0217] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM) or flash memory, optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0218] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a rendering device (e.g., a cathode ray tube (CRT) renderer or a liquid crystal display (LCD)) for rendering information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices are also used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0219] The systems and technologies described herein can be implemented in computing systems that include back-end components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0220] A computer system can include client and server components. Clients and servers are generally located far apart and typically interact via a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, a server in a distributed system, or a server incorporating blockchain technology.

[0221] This disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute a gap assessment method based on a large model and / or a training method for a large model.

[0222] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements a gap assessment method based on a large model and / or a training method for a large model.

[0223] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure is achieved, and this is not limited herein. Furthermore, in this disclosure, relational terms such as "first," "second," and "third" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Additionally, "multiple" in this disclosure can be understood as at least two.

[0224] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A training method for a large model, comprising: Obtain multiple search term samples; wherein at least some of the multiple search term samples originate from a specified scenario of the target platform; For each of the plurality of search term samples, a training data set corresponding to the search term sample is constructed; wherein, the training data set includes the model evaluation results and ground truth values ​​of the search result samples corresponding to the search term samples; the search result samples are obtained by performing content searches based on the search term samples in a specified scenario on the target platform; the model evaluation results are obtained using an initial large model; Based on the training data set, the initial large model is trained to obtain the target large model.

2. The method according to claim 1, wherein, The process of obtaining multiple search term samples includes: Obtain multiple first search terms from a specified scenario originating from the target platform; Based on the multiple first search terms, the multiple search term samples are obtained; Alternatively, multiple second search terms from across the entire platform can be obtained to generate multiple search term samples based on the multiple first search terms and the multiple second search terms.

3. The method according to claim 2, wherein, The acquisition of multiple first search terms from a specified scenario originating from the target platform includes: Obtain multiple first historical search terms from a specified scenario originating from the target platform; From the plurality of first historical search terms, determine the first number of first historical search terms that were searched the most times, and the second number of first historical search terms that were searched the least times; Based on the first number of first historical search terms and the second number of second historical search terms, the plurality of first search terms are obtained.

4. The method according to claim 2, wherein, The acquisition of multiple second search terms from across the platform includes: Obtain multiple second historical search terms from the entire platform; From the plurality of second historical search terms, a third number of second historical search terms are randomly selected; Based on the third number of second historical search terms, the plurality of second search terms are obtained.

5. The method according to any one of claims 1 to 4, wherein, The construction of the training data set corresponding to the search term sample includes: Obtain a sample of assessment prompts; Using the initial large model, the search result sample is evaluated according to the evaluation prompt sample to obtain the model evaluation result.

6. The method according to claim 5, wherein, The process of obtaining evaluation prompt samples includes: Obtain samples from multiple quality assessment dimensions; Obtain multiple quality assessment standard samples that correspond one-to-one with the samples of the multiple quality assessment dimensions; The evaluation prompt sample is generated based on the multiple quality assessment dimension samples and the multiple quality assessment standard samples.

7. The method according to claim 6, wherein, The process of obtaining samples across multiple quality assessment dimensions includes: Obtain the parsing result sample of the search term sample; Based on the parsing results sample, obtain samples of the multiple quality assessment dimensions.

8. The method according to claim 6, wherein, The process of generating the assessment prompt sample based on the multiple quality assessment dimension samples and the multiple quality assessment standard samples includes: Obtain an evaluation sample; The evaluation prompt sample is generated based on the multiple quality assessment dimension samples, the multiple quality assessment standard samples, and the evaluation example samples.

9. The method according to claim 6, wherein, The step of training the initial large model based on the training data set to obtain the target large model includes: Based on the model evaluation results and the true values ​​of the evaluation results, the overall evaluation loss is obtained; Based on the overall evaluation loss, the initial large model is trained to obtain the target large model.

10. The method according to claim 9, wherein, The overall evaluation loss is obtained based on the model evaluation results and the true values ​​of the evaluation results, including: Based on the model evaluation results and the true values ​​of the evaluation results, the first evaluation loss is obtained; For each of the multiple quality assessment dimension samples, a second assessment loss is obtained based on the first single-dimensional assessment result corresponding to the quality assessment dimension sample and the second single-dimensional assessment result corresponding to the quality assessment dimension sample. Based on the first evaluation loss and the multiple second evaluation losses corresponding one-to-one with the samples of the multiple quality evaluation dimensions, the overall evaluation loss is obtained.

11. The method according to any one of claims 1 to 4, wherein, The construction of the training data set corresponding to the search term sample includes: Obtain the evaluation and labeling results of the search results sample; Obtain user behavior feedback results for the search results sample; Based on the evaluation labeling results and the user behavior feedback results, the true value of the evaluation result is obtained.

12. The method according to claim 11, wherein, The step of obtaining user behavior feedback results for the search result sample includes: Obtain samples from multiple quality assessment dimensions; Obtain multiple single-dimensional behavioral feedback results that correspond one-to-one with samples from multiple quality assessment dimensions; The user behavior feedback results are obtained based on the multiple single-dimensional behavior feedback results.

13. The method according to claim 1, further comprising: Before constructing the training data set corresponding to the search term sample, based on the search term sample, a content search is performed in a specified scenario of the target platform to obtain multiple candidate search results with a recommendation order; From the multiple candidate search results, select the third number of candidate search results that are ranked first in the recommendation order; Based on the third number of candidate search results, the search result sample is obtained.

14. A gap assessment method based on a large model, comprising: Obtain the target search term; Based on the target search terms, a content search is performed in a specified scenario on the target platform to obtain N content search results; where N≥1 and N is an integer; For each of the N content search results, the target large model is used to obtain the quality assessment result corresponding to the content search result; Based on the N quality assessment results that correspond one-to-one with the N content search results, the content gap assessment result of the target search term in the specified scenario of the target platform is obtained.

15. The method according to claim 14, wherein, The process of using a target large model to obtain quality assessment results corresponding to the content search results includes: Get assessment prompts; Using the target large model and following the evaluation prompts, the content search results are evaluated to obtain the quality evaluation results.

16. The method according to claim 15, wherein, The process of obtaining evaluation prompts includes: Obtain multiple quality assessment dimensions; Obtain multiple quality assessment standards that correspond one-to-one with the multiple quality assessment dimensions; The assessment prompts are generated based on the multiple quality assessment dimensions and the multiple quality assessment standards.

17. The method according to claim 16, wherein, The acquisition of multiple quality assessment dimensions includes: Obtain the parsing results of the target search term; Based on the analysis results, the multiple quality assessment dimensions are obtained.

18. The method according to claim 16, wherein, The process of generating the assessment prompts based on the multiple quality assessment dimensions and the multiple quality assessment standards includes: Get an evaluation example; The assessment prompts are generated based on the multiple quality assessment dimensions, the multiple quality assessment standards, and the assessment examples.

19. The method of claim 14, wherein, The process of obtaining the content gap assessment result for the target search term in a specified scenario on the target platform, based on the N quality assessment results that correspond one-to-one with the N content search results, includes: Obtain the total number of search results for the N content items; Based on the N quality assessment results, determine the percentage of content search results that meet the quality assessment requirements among the N content search results; Based on the total number and the percentage of the number, the content gap assessment result is obtained.

20. The method according to claim 19, wherein, The content gap assessment result obtained based on the total number and the proportion of the number includes one of the following: If the total number is less than or equal to the number threshold, a first gap assessment result is obtained; wherein, the first gap assessment result is used to characterize that there is a number gap for the target search term in the specified scenario of the target platform; When the total number is greater than the number threshold and the number proportion is less than or equal to the proportion threshold, a second gap assessment result is obtained; wherein, the second gap assessment result is used to characterize that the target search term has a quality gap in the specified scenario of the target platform; When the total number is less than or equal to the quantity threshold and the quantity percentage is less than or equal to the percentage threshold, a third gap assessment result is obtained; wherein, the third gap assessment result is used to characterize that the target search term has a quantity gap and a quality gap in the specified scenario of the target platform.

21. The method of claim 20, further comprising: Based on the content gap assessment results, pass-through information is generated; Based on the content gap assessment results, the target devices were identified; The transparent information is sent to the target device.

22. The method according to claim 21, wherein, The generation of pass-through information based on the content gap assessment results includes: Obtain impact assessment data related to the target search term; wherein, the impact assessment data includes the page views of historical search results and / or the consumer information of the historical search results obtained by conducting content searches on the entire platform based on the target search term within a first historical period; Based on the aforementioned impact assessment data, the impact of the gap is determined; Based on the content gap assessment results and the gap impact, the transparent transmission information is generated.

23. The method according to claim 21, wherein, The generation of pass-through information based on the content gap assessment results includes: If the proportion of the quantity is less than or equal to the proportion threshold, the content search results that do not meet the quality assessment requirements among the N content search results are identified as low-quality search results. Determine multiple single-dimensional evaluation results for the low-quality search results; wherein, the multiple single-dimensional evaluation results correspond one-to-one with multiple quality evaluation dimensions; Supplementary instructions are generated based on the multiple single-dimensional evaluation results; Based on the content gap assessment results and the supplementary instructions, the transparent information is generated.

24. A training device for a large model, comprising: A sample acquisition unit is used to acquire multiple search term samples; wherein at least some of the multiple search term samples are derived from a specified scenario of the target platform; A data set construction unit is used to construct a training data set corresponding to each of the plurality of search term samples; wherein, the training data set includes model evaluation results and ground truth values ​​of search result samples corresponding to the search term samples; the search result samples are obtained by performing content searches based on the search term samples in a specified scenario of the target platform; the model evaluation results are obtained using an initial large model; The model training unit is used to train the initial large model based on the training data set to obtain the target large model.

25. A gap assessment device based on a large model, comprising: The search term acquisition unit is used to acquire target search terms; The content search unit is used to perform content search based on the target search term and in a specified scenario on the target platform to obtain N content search results; where N≥1 and N is an integer; The content evaluation unit is used to obtain a quality evaluation result corresponding to each of the N content search results using the target large model. The content gap assessment unit is used to obtain the content gap assessment result of the target search term in a specified scenario on the target platform based on N quality assessment results that correspond one-to-one with the N content search results.

26. An electronic device, comprising: At least one processor; A memory that is communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method according to any one of claims 1 to 23.

27. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 23.

28. A computer program product comprising a computer program; wherein, When the computer program is executed by a processor, it can implement the method of any one of claims 1 to 23.

Images