A method for measuring the degree of mention of things in a large language model

Abstract

This invention relates to the field of large language model question-answering analysis technology, and discloses a method for measuring the degree of mention of things in a large language model. The method includes: identifying the specific thing to be measured and multiple sampling platforms; defining keywords and selecting a single sampling platform for multiple samplings; calculating the weighted value of each sampling; calculating the standardized score of the sampling platform; calculating the platform weight based on the number of users on the platform; calculating the flicker index based on the standardized score and platform weight; sampling on multiple sampling platforms; and finally obtaining a multi-platform flicker index for measuring the degree of mention of a specific thing and obtaining the analysis results. This invention can effectively adapt to the dynamic characteristics of large language models and the needs of multiple industries, supports multi-platform comparison, and significantly improves the efficiency, comprehensiveness, and accuracy of the evaluation of the degree of mention of things in large language models, providing strong support for enterprise marketing effectiveness evaluation and content competitiveness analysis.

Description

Technical Field

[0001] This invention relates to the field of large language model question-answering analysis technology, and in particular to a method for measuring the degree of mention of things in a large language model. Background Technology

[0002] With the widespread application of Large Language Models (LLM) in question-answering scenarios, accurately measuring the degree of mention of specific keyword-related items in feedback results (such as mention frequency, content priority, information completeness, etc.) is of great significance for enterprise marketing effectiveness evaluation and content competitiveness analysis. Existing technologies urgently need improvement and breakthroughs in this field.

[0003] The existing quantitative dimensions are relatively singular, and currently rely mainly on the single indicator of "keyword coverage" to measure the degree of mention. This fails to comprehensively consider the combined impact of factors such as sampling order (timeliness), platform differences (user base), and semantic association (keyword combination) on the degree of mention. This singular quantitative method is difficult to fully and accurately reflect the actual mention situation of a specific thing in the feedback results.

[0004] The user base and semantic analysis capabilities of different LLM platforms vary significantly, and existing technologies lack a unified and standardized evaluation method. This makes it difficult to conduct effective horizontal comparisons when comparing cross-platform data, and it is difficult to accurately assess the differences in the degree of mention of specific things on different platforms. Consequently, it affects enterprises' comprehensive analysis and judgment of the marketing effectiveness and content competitiveness of different platforms.

[0005] Large language models have the characteristic of "real-time learning and updating," but most existing technologies have not designed timeliness weights for sampling order to take into account this characteristic, and cannot reasonably distinguish the value of old information from new information. This makes it impossible to fully consider the timeliness of information when measuring the degree of mention, resulting in inaccurate and untimely evaluation results, which are difficult to adapt to the rapidly changing market environment and user needs. Summary of the Invention

[0006] The purpose of this invention is to propose a method for measuring the degree of mention of things in a large language model in order to solve the problems in the prior art.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a method for measuring the degree of mention of things in a large language model, comprising the following steps:

[0008] Step S1: Based on the needs of the user or enterprise, determine the specific thing to be measured and multiple sampling platforms;

[0009] Step S2: Define keywords based on specific things, sample on a single sampling platform, and record the number of samplings, the ranking order of each sampling, and the number of keywords.

[0010] Step S3: Calculate the weighted value of a single sample based on the ranking order and the number of keywords in the single sample.

[0011] Step S4: Define the full score of the sample and calculate the standardized score of the sampling platform based on the single sampling weight value and the full score of the sample.

[0012] Step S5: Define a constant for millions of users, and calculate the platform weight of the sampling platform by combining the constant for millions of users with the number of platform users of the sampling platform.

[0013] Step S6: Calculate the flicker index of the sampling platform based on the standardized score of the sampling platform and the corresponding platform weight, and calculate the multi-platform flicker index based on the flicker index of multiple sampling platforms.

[0014] Step S7: Analyze the flicker index across multiple platforms and present the analysis results to users or businesses in a visual manner.

[0015] The beneficial effects of the technical solution provided by this invention include at least the following:

[0016] This invention systematically characterizes the mention features of specific things in Large Language Model (LLM) responses from three dimensions: frequency (number of samplings), priority (order weight), and completeness (score standardization). By comprehensively considering these three dimensions, it can more comprehensively and accurately reflect the degree of mention of things in the feedback results.

[0017] This invention targets the "real-time learning and updating" characteristic of large language models. By using a power decay mechanism of sampling order weights, it naturally adapts to the dynamic changes of LLM. This method can prioritize capturing the value of the latest information and ensure that the timeliness of information is fully considered when measuring the degree of mention, thereby providing more accurate and forward-looking evaluation results.

[0018] This invention is based on a standardized design of "millions of users constant", which effectively eliminates the differences in user base across different platforms and supports horizontal comparisons between multiple LLM platforms. This standardized design makes cross-platform data comparisons more fair and effective, providing strong support for enterprises to analyze marketing effectiveness and content competitiveness on different platforms.

[0019] The parameters (a, k, M) mentioned in this invention can be flexibly configured according to specific application scenarios, adapting to keyword association scenarios in multiple industries such as scenic spots, retail, and manufacturing. At the same time, through formulaic calculation, automated analysis from sampling data to index output is realized. Compared with traditional manual sampling methods, it significantly improves implementation efficiency and saves time and resources for enterprises and researchers. Attached Figure Description

[0020] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a method provided in an embodiment of the present invention. Detailed Implementation

[0022] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a method for measuring the degree of mention of things in a large language model according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0025] The following description, in conjunction with the accompanying drawings, details a specific scheme for measuring the degree of mention of things in a large language model provided by this invention.

[0026] Please see Figure 1 The diagram illustrates a flowchart of a method for measuring the degree of mention of things in a large language model according to an embodiment of the present invention. The method includes the following steps:

[0027] Step S1: Based on the needs of the user or enterprise, determine the specific thing to be measured and multiple sampling platforms;

[0028] Step S1 also includes the following sub-steps:

[0029] S1-1, Identify the industry in which the user or company operates, and determine the user or company's needs through market research, and determine the specific things to be measured based on the user or company's needs;

[0030] S1-2, select multiple specific large language model platforms as sampling platforms based on the needs of users or enterprises and the specific things to be measured;

[0031] S1-3 records the user's or company's needs, the specific thing to be measured, the name of the sampling platform, and the total number of sampling platforms selected. .

[0032] It should be noted that market research methods include:

[0033] Questionnaire survey: Design standardized questionnaires and distribute them to users or company employees through email, social media, online survey tools (such as SurveyMonkey), etc. Ask users about their concerns, frequency of use and evaluation of specific things, and use statistical software (such as SPSS) to analyze the questionnaire results and extract the needs of users or companies.

[0034] Interviews: Select representative interviewees, such as corporate decision-makers, industry experts, or ordinary users, and gain a deeper understanding of their needs and expectations through direct dialogue with them.

[0035] Industry report and literature research: Collect relevant data from industry reports (such as IDC), academic journals, industry blogs, etc., summarize the collected data, extract key information and trends, understand industry dynamics, and analyze the needs of users or enterprises.

[0036] The criteria for platform selection include:

[0037] User scale: Platforms with a larger user base are usually more representative, and the results sampled from these platforms are usually more persuasive and credible, such as OpenAI's ChatGPT and Baidu's Wenxin Yiyan, which are mainstream large language model platforms.

[0038] Functional characteristics: You can select a platform that matches the functional characteristics of the thing to be measured. For example, if the specific thing to be measured involves multilingual content, you can choose a platform that supports multiple languages.

[0039] Data accessibility: Platforms that provide API interfaces or have public datasets can be selected, making it easy to sample and obtain question and answer data.

[0040] Step S2: Define keywords based on specific things, sample on a single sampling platform, and record the number of samplings, the ranking order of each sampling, and the number of keywords.

[0041] Step S2 also includes the following sub-steps:

[0042] S2-1: Analyze the attributes of a specific thing, define keywords based on the attributes of the specific thing, and formulate a sampling strategy. The sampling strategy includes the query statement, sampling time interval, and sampling number. ;

[0043] S2-2, Select a single sampling platform, based on the sampling time interval and the number of samples. Enter the query statement sequentially on the selected sampling platform and obtain the feedback results;

[0044] S2-3 defines each process from inputting a query statement to obtaining a response result as a sampling, and records the ranking order of each sampling. and the number of keywords contained in the feedback results obtained from this sampling. .

[0045] It should be noted that the specific thing to be measured includes a brand, product, service, or location. The relevant attributes of the thing to be measured include brand name, product model, service type, or location name. If the company is a tourism company, the thing to be measured may be a popular tourist attraction, and its relevant attributes are the name, type, rating, and location of the popular tourist attraction.

[0046] Ranking order It can reflect the timeliness of information. The larger the value, the later the information acquisition order (sampling order) is.

[0047] Step S3: Calculate the weighted value of a single sample based on the ranking order and the number of keywords in the single sample.

[0048] Step S3 also includes the following sub-steps:

[0049] S3-1, Define the ranking order weight coefficients based on the needs of users and enterprises. Keyword combination weight coefficient ;

[0050] S3-2, establish the timeliness decay model using the following formula, and define the ranking order weight. :

[0051]

[0052] In the formula, For ranking order weight, For ranking order, This is the ranking order weighting coefficient;

[0053] S3-3, Keyword combination weight is defined using the following formula. :

[0054]

[0055] In the formula, For keyword combination weight, For the number of keywords, This refers to the keyword combination weight coefficient;

[0056] S3-4, calculate the single-sample weighting value using the following formula. :

[0057]

[0058] In the formula, This is a weighted value for a single sample. For ranking order, Ranking order weight, This is the weight of the keyword combination.

[0059] It should be noted that the ranking order weighting coefficient It is a constant coefficient that can be flexibly configured according to specific application scenarios (such as keyword association scenarios in industries such as scenic spots, retail, and manufacturing). Its function is to control the decay rate of the order weight. The larger the weight coefficient 'a', the faster the order weight decays.

[0060] The timeliness logic used to implement "the newer the sample, the more dynamically the weight is adjusted" prioritizes capturing the value of the latest information, and it is naturally adapted to the "real-time learning and updating" characteristics of LLM.

[0061] Keyword combination weight coefficient It is a constant coefficient that can be flexibly configured according to specific application scenarios (such as keyword association scenarios in industries such as scenic spots, retail, and manufacturing). It is used to control the decay rate of keyword combination weight. The larger the keyword combination weight coefficient k is, the faster the keyword combination weight decays.

[0062] Combination analysis is used to adapt to scenarios involving multiple keyword associations. The more keywords obtained in a single sampling, the higher the keyword combination weight. The lower the value, the more invalid associations are eliminated due to keyword stuffing.

[0063] Single sampling weighted value This is a quantification of the contribution of a single sample that integrates "ranking order weight" and "keyword combination weight".

[0064] Step S4: Define the full score of the sample and calculate the standardized score of the sampling platform based on the single sampling weight value and the full score of the sample.

[0065] Step S4 also includes the following sub-steps:

[0066] S4-1, the full score of the sample is defined by the following formula. :

[0067] , =1

[0068] In the formula, The full score for the sample. This is a weighted value for a single sample. For ranking order, Ranking order weight, Weighting of keyword combinations;

[0069] S4-2, the single-sample weighting value is calculated using the following formula. Convert to a percentage score :

[0070]

[0071] In the formula, The score is out of 100. This is a weighted value for a single sample. The full score for the sample;

[0072] S4-3, calculate the total score of the sampling platform using the following formula. :

[0073]

[0074] In the formula, The total score of the sampling platform is n, where n is the number of samplings. The score is out of 100.

[0075] S4-4, the standardized score of the sampling platform is calculated using the following formula. :

[0076]

[0077] in, denoted as the total score of the sampling platform, and n as the number of sampling attempts.

[0078] It should be noted that the sample full score The benchmark for representing a full score in a single sampling is... The single-sample weighted value when =1 is used to calculate the single-sample score (i.e., the single-sample weighted value) in subsequent steps. Convert to a percentage score .

[0079] Step S5: Define a constant for millions of users, and calculate the platform weight of the sampling platform by combining the constant for millions of users with the number of platform users of the sampling platform.

[0080] Step S5 also includes the following sub-steps:

[0081] S5-1 defines constants for millions of users based on user and enterprise needs. ;

[0082] S5-2, Obtain the number of platform users of the sampling platform from a third-party data agency. ;

[0083] S5-3, calculate the platform weight of the sampling platform using the following formula. :

[0084]

[0085] in, For the number of platform users, For millions of users, this is a constant.

[0086] It should be noted that, for The number of users on the sampling platform can be flexibly configured in units of millions, depending on the specific application scenario (such as keyword association scenarios in industries like scenic spots, retail, and manufacturing). Platform weight The contribution, generally speaking, is the number of users. The larger the value, the greater the platform's influence.

[0087] Third-party data agencies include IDC, Gartner, and Statista, which regularly publish reports on the number of users and market share of various technology platforms, from which one can obtain information on the number of platform users. .

[0088] Total Score This represents the combined score of all samples within the sampling platform, or the standardized score. This represents the standardized result of the overall score of all samples within the sampling platform after eliminating differences in the number of samplings.

[0089] Step S6: Calculate the flicker index of the sampling platform based on the standardized score of the sampling platform and the corresponding platform weight, and calculate the multi-platform flicker index based on the flicker index of multiple sampling platforms.

[0090] Step S6 also includes the following sub-steps:

[0091] S6-1, Based on the standardized score and corresponding platform weight of each sampling platform, the flicker index of the sampling platform is calculated using the following formula. :

[0092]

[0093] in, The scintillation index of the sampling platform, The standardized score for the sampling platform, The platform weight of the sampling platform;

[0094] S6-2, Repeat the sampling process from steps S2 to S5 on multiple sampling platforms, and calculate the scintillation index of each sampling platform. ,in It is a positive integer. and ;

[0095] S6-3, the scintillation index of each sampling platform is calculated using the following formula. Perform aggregate calculations to obtain the flicker index for multiple platforms. :

[0096]

[0097] in, For multi-platform flicker index, The scintillation index of the sampling platform, p represents the total number of platforms. and Positive integers.

[0098] It should be noted that the flicker index This represents a comprehensive assessment of the sampling platform's content quality and user base.

[0099] Indicates the scintillation index across multiple sampling platforms The final evaluation index obtained after aggregation calculation is a quantitative representation of the comprehensive mention of a specific thing across multiple large language model platforms.

[0100] Step S7: Calculate the flicker index of each sampling platform based on the standardized score of each sampling platform and its corresponding platform weight;

[0101] Step S7 also includes the following sub-steps:

[0102] S7-1 summarizes the flicker indexes of various sampling platforms, compares the flicker indexes of different platforms horizontally, and analyzes the impact of user groups, content ecosystems, and algorithm characteristics of each platform on the degree of mention of specific things.

[0103] S7-2 uses time series analysis to analyze the flicker index of multiple platforms in the time domain to determine the trend of the degree of mention of a specific thing changing over time.

[0104] S7-3 uses a correlation analysis method based on Pearson correlation coefficient to analyze the correlation between the flicker index and the number of platform users, industry dynamics and market events, and extracts the key factors that affect the degree of mention of specific things;

[0105] S7-4 uses Tableau visualization tools to show users or businesses why the mention rate of a particular item changes across the platform, the trend over time, and the key factors influencing the mention rate of a particular item.

[0106] It should be noted that the analysis of the impact of user groups, content ecosystems, and algorithmic characteristics of various platforms on the frequency of mention of specific topics includes:

[0107] Analyze the impact of user group characteristics such as age, region, and interests on the degree of mention of specific things. For example, some sampling platforms may be more popular among young people, while others may be more popular among professionals.

[0108] The study examines the impact of content type and style on the degree of mention of specific topics across different sampling platforms. For example, social media sampling platforms may place greater emphasis on immediacy and interactivity, while news sampling platforms may place greater emphasis on depth and authority.

[0109] Understand the impact of each sampling platform's content recommendation algorithm on the mention rate of specific items. For example, some sampling platforms may be more inclined to recommend popular content, while others may focus more on personalized recommendations.

[0110] Time series analysis methods specifically include:

[0111] The generation time of the sampling results from each sampling platform is extracted, and time series data is generated by combining the corresponding sampling results. The long-term trend is identified by calculating the moving average of the time series data, and the short-term trend is identified by performing exponential weighted smoothing on the time series data. The long-term trend and short-term trend of the time series data are fitted to obtain the linear trend of the mention level of a specific thing changing over time.

[0112] The Pearson correlation coefficient is a statistical indicator used to measure the degree of linear correlation between two variables. Its value ranges from [−1, 1]. The closer the value is to 1 or -1, the stronger the linear correlation between the two variables. The closer the value is to 0, the weaker the correlation.

[0113] Tableau is a powerful data visualization tool widely used in data analysis and business intelligence. It offers various chart types such as bar charts, line charts, maps, and heatmaps, which can transform complex data into intuitive visualizations, helping users quickly understand and apply the data.

[0114] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for measuring the extent of a thing mention in a large language model, characterized in that, The method comprises the following steps: Step S1, determining the specific matter to be measured and a plurality of sampling platforms according to the needs of a user or an enterprise; Step S2, defining keywords according to the specific matter, sampling on a single sampling platform, recording the number of samplings, the ranking order of single sampling, and the number of keywords; Step S3, calculating the weighted value of single sampling according to the ranking order of single sampling and the number of keywords; Step S4, defining a sample full score, calculating the normalized score of the sampling platform according to the weighted value of single sampling and the sample full score; Step S5, defining a million user constant, combining the million user constant with the number of platform users of the sampling platform to calculate the platform weight of the sampling platform; Step S6, calculating the flash index of the sampling platform according to the normalized score of the sampling platform and the corresponding platform weight, and calculating the multi-platform flash index according to the flash indexes of the plurality of sampling platforms; Step S7, analyzing the multi-platform flash index, and displaying the analysis result to the user or the enterprise in a visual manner; In step S2, the following sub-steps are further included: S2-1, analyzing attributes of the specific thing, defining a keyword and formulating a sampling strategy according to the attributes of the specific thing, the sampling strategy including a query statement, a sampling time interval and a sampling number ; S2-2, selecting a single sampling platform according to the sampling time interval and the sampling times inputting the query statement on the selected sampling platform in sequence and obtaining feedback results; S2-3, define each time from the input query statement to the process of obtaining the feedback result as a sampling, record the ranking order of each sampling and the number of keywords contained in the feedback result obtained by the sampling ; In step S3, the following sub-steps are further included: S3-1, defining ranking order weight coefficients according to user and enterprise needs and keyword combination weight coefficients ; S3-2, a time effectiveness decay model is established, and is specifically as shown in the formula: wherein, is a ranking order weight; S3-3, defining keyword combination weights ; S3-4, compute single sample weighting value , as shown in the formula: ; In step S4, the following sub-steps are further included: S4-1, define sample full score value , as shown in the formula: = 1; S4-2, single sample weighting value Convert to percent score ; S4-3, calculate the total score of the sampling platform ; S4-4, calculating the standardized score of the sampling platform , as shown in the formula: ; In step S5, the following sub-steps are further included: S5-1 Define the million user constant according to the needs of users and enterprises ; S5-2, obtaining the number of platform users of the sampling platform from a third-party data institution ; S5-3, calculating a platform weight of the sampling platform , as shown in the formula: ; In step S6, the following sub-steps are further included: S6-1, according to the standardization score of each sampling platform and the corresponding platform weight, calculating the scintillation index of the sampling platform ; S6-2, repeating the sampling process of steps S2 to S5 on multiple sampling platforms, and calculating the flicker index of each sampling platform wherein 1 , and is a positive integer; S6-3, the flicker index of each sampling platform The aggregated calculation is performed to obtain the multi-platform flicker index , as shown in the formula: .

2. The method for measuring the degree of reference to a matter in a large language model according to claim 1, characterized in that: In step S1, the following sub-steps are further included: S1-1, determining the industry in which the user or the enterprise is located, and determining the needs of the user or the enterprise through market research, and determining the specific matter to be measured according to the needs of the user or the enterprise; S1-2, selecting a plurality of specific large language model platforms as sampling platforms according to the needs of the user or the enterprise and the specific matter to be measured; S1-3, record the user or enterprise's requirements, the specific things to be measured, the sampling platform name, and the total number of selected sampling platforms .

3. The method for measuring the degree of reference to a matter in a large language model according to claim 1, characterized in that: In step S7, the following sub-steps are further included: S7-1, aggregating the flash indexes of the sampling platforms, comparing the flash indexes of different platforms horizontally, and analyzing the influence of the user groups, content ecology, and algorithm characteristics of each platform on the degree of reference to the specific matter; S7-2, analyzing the multi-platform flash indexes in the time domain through a time series analysis method to determine the trend of the degree of reference to the specific matter over time; S7-3, analyzing the correlation between the flash index and the number of platform users, industry dynamics, and market events through a correlation analysis method based on Pearson correlation coefficient, and extracting key factors affecting the degree of reference to the specific matter; S7-4, displaying the reasons for the change of the degree of reference to the specific matter with the platform, the trend of the degree of reference to the specific matter over time, and the key factors affecting the degree of reference to the specific matter to the user or the enterprise through a Tableau visualization tool.

Images