Keyword search optimization system and method based on AI intelligent analysis

Abstract

The application discloses a keyword retrieval optimization system and method based on AI intelligent analysis, and relates to the technical field of keyword recognition.The system comprises a database construction module, a first candidate set screening module, a second candidate set screening module and a verification and retrieval application module;the application builds a database by collecting news and retrieval data, analyzes the redundancy of keyword combinations, screens low-redundancy combinations, selects high-score combinations in combination with coverage and hit rate, and obtains an optimized set through consistency verification.The application replaces manual experience with data quantization, reduces retrieval redundancy, balances coverage and precision, simplifies technical implementation, improves news retrieval efficiency and quality, is suitable for news retrieval scenes, and regularly updates the retrieval keyword library to ensure that the optimization effect is continuously effective.

Description

Technical Field

[0001] This invention relates to the field of keyword recognition technology, specifically to a keyword retrieval optimization system and method based on AI intelligent analysis. Background Technology

[0002] In the field of news information retrieval, keyword retrieval is a core technology, but its efficiency and accuracy have long been limited by traditional manual optimization methods. Current technologies rely heavily on manual experience in keyword combination, lacking scientific quantitative analysis, which easily leads to two key problems: First, keyword combinations contain a large amount of redundant information, with high semantic overlap between different combinations. This results in the need to repeatedly process similar information during the retrieval process, significantly increasing data processing load, reducing retrieval response speed, and failing to meet the needs of real-time news information retrieval. Second, it is difficult to balance the coverage and matching hit rate of keyword combinations. Manually selected combinations often have too narrow a coverage or low hit rate, failing to comprehensively and accurately match target news, leading to omissions or redundancy in search results and affecting the quality of information retrieval.

[0003] While some technologies attempt to optimize keywords through simple data statistics, they lack a systematic redundancy assessment index and a comprehensive analysis system for coverage and hit rate. This makes it impossible to achieve efficient keyword combination screening from multiple dimensions, and still requires the assistance of complex algorithm models, which increases the difficulty and cost of technical implementation and makes it difficult to be widely applied in news retrieval scenarios. Therefore, there is an urgent need for a technical solution that can optimize keyword combinations based on basic data dimensions without the need for complex algorithms. Summary of the Invention

[0004] The purpose of this invention is to provide a keyword retrieval optimization system and method based on AI intelligent analysis to solve the problems mentioned in the background art.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] The keyword retrieval optimization method based on AI intelligent analysis includes the following steps:

[0007] S1. Collect historical news data and historical search records in the target news search scenario, extract all used keyword combinations from the historical search records, integrate the keyword combinations with the corresponding historical search data, and establish a keyword combination database.

[0008] S2. Perform redundancy analysis on each keyword combination in the keyword combination database. Use an AI intelligent model to analyze the proportion of duplicate information contained in each keyword combination and the degree of semantic overlap between keywords within the combination, and construct a redundancy evaluation index. Calculate the redundancy value of each combination, compare the redundancy value with a preset redundancy threshold, and select keyword combinations with redundancy values ​​lower than the preset redundancy threshold to form the first candidate set.

[0009] S3. Conduct coverage and matching hit rate analysis on the keyword combinations in the first candidate set. Use an AI intelligent model to analyze the number of news items corresponding to each keyword in historical news data and the scope of news that each keyword combination can cover, and calculate the coverage of each combination. Extract the number of successful searches and the total number of searches corresponding to each keyword combination from historical search records, and calculate the matching hit rate of each combination. Establish a comprehensive evaluation system of coverage and hit rate, and substitute the coverage and matching hit rate of each combination into the evaluation system for comprehensive scoring. Select keyword combinations with comprehensive scores higher than the preset scoring threshold to form the second candidate set.

[0010] S4. Perform consistency verification on the keyword combinations in the second candidate set, and remove keyword combinations that have abnormal comprehensive scores due to data statistical deviations to obtain the final optimized keyword combination set; store the final optimized keyword combination set in the retrieval keyword library, and call the keyword combinations in the retrieval keyword library to perform retrieval matching when a subsequent news information retrieval request is triggered.

[0011] Furthermore, S1 includes the following:

[0012] The scope of the target news search scenario is determined, and historical news data within this scenario over the past three years is collected. The historical news data includes news titles, text content, publication time, and keyword tags. At the same time, the historical search records of users in this scenario over the past two years are also collected. The historical search records include the keyword combinations used in the search, the search time, the number of clicks on the search results, and the search success indicator.

[0013] The collected historical search records are cleaned to remove invalid records with empty search times or keyword combination character lengths of less than 2, and valid historical search records are retained; all unique keyword combinations are extracted from the valid historical search records to create an initial keyword combination list;

[0014] For each keyword combination in the initial keyword combination list, match the corresponding historical search data to form a keyword combination-related data mapping table; the historical search data includes the total number of searches for that combination, the number of successful searches, and the average number of clicks on the search results;

[0015] Based on the keyword combination-related data mapping table and the collected historical news data, a keyword combination database is constructed; the fields of the database include keyword combination ID, keyword combination content, total number of searches, number of successful searches, average number of clicks on search results, and a set of associated historical news IDs.

[0016] Furthermore, S2 includes the following:

[0017] For each keyword combination Ci in the keyword combination database, extract all keywords kij contained in the combination, where i ranges from 1 to n, n is the total number of keyword combinations, and j ranges from 1 to m, where m is the number of keywords in a single combination. For each keyword combination Ci, extract the domain labels Bi of all corresponding keywords kij. Iterate through all other keyword combinations Cp in the keyword combination database and extract the domain labels Bp corresponding to the keywords kpq contained in them. Similarly, p ranges from 1 to n, and p ≠ i, and q ranges from 1 to m. Calculate the domain label overlap rate L between Ci and Cp, using the formula: L = N(Bi∩Bp) / N(Bi∪Bp), where N(Bi∩Bp) is the number of identical domain labels between keyword combination Ci and keyword combination Cp; N(Bi∪Bp) is the total number of domain labels between keyword combination Ci and keyword combination Cp. When the domain label overlap rate L between keyword combination Ci and keyword combination Cp is greater than or equal to a preset threshold L0, keyword combination Ci and keyword combination Cp are determined to be semantically related combinations.

[0018] Summarize all keyword combinations Cp that are semantically related to keyword combination Ci, count the number of common keywords tip between keyword combination Ci and keyword combination Cp, calculate the keyword overlap rate rip between keyword combination Ci and each keyword combination Cp, and rip=tip / max(m,mp), where mp is the number of keywords contained in keyword combination Cp; take the average of the keyword overlap rates between keyword combination Ci and all related keyword combinations Cp as the proportion of repeated information Ri; obtain the semantic vector Vij of each keyword kij through an AI intelligent model (e.g., based on the Word2Vec pre-trained model), calculate the semantic similarity sju=cos(Vij,Viu) between any two keywords kij and kiu (j≠u) within the combination, where cos() is the cosine similarity function, and take the average of the semantic similarity between all keywords within the combination as the degree of semantic overlap Si;

[0019] Based on the proportion of duplicate information Ri and the degree of semantic overlap Si, a redundancy evaluation index Yi is constructed, and Yi = α × Ri + β × Si, where α and β are weight coefficients, and α + β = 1. α and β are determined by fitting the retrieval efficiency loss rate of redundant combinations in historical data. The mean μ_Y and standard deviation σ_Y of the redundancy evaluation index Yi of all keyword combinations in the keyword combination database are statistically analyzed, and the corresponding redundancy threshold Y0 is calculated, and Y0 = μ_Y - γ1 × σ_Y, where γ1 is an adjustment factor. The redundancy evaluation index Yi of each keyword combination Ci is compared with the redundancy threshold Y0. If Yi < Y0, the keyword combination Ci is included in the first candidate set; otherwise, it is discarded.

[0020] Furthermore, S3 includes the following:

[0021] For each keyword combination Ci in the first candidate set, extract the set of historical news IDs associated with that combination from the keyword combination database, count the number of historical news items Ni contained in that set, and simultaneously count the total number of historical news data N_total in the target news retrieval scenario; calculate the coverage Cov_i of keyword combination Ci, and Cov_i = Ni / N_total; extract the total number of searches Ti and the number of successful searches Ji of keyword combination Ci from the keyword combination database, and calculate the matching hit rate Hi of keyword combination Ci, with the formula: Hi = Ji / Ti;

[0022] A comprehensive evaluation system for coverage and hit rate is established by combining the coverage (Cov_i) and matching hit rate (Hi) of keyword combination Ci. A comprehensive score Zi is calculated, where Zi = w1 × Cov_i + w2 × Hi, and w1 and w2 are weighting coefficients, with w1 + w2 = 1. w1 is determined by the influence coefficient of coverage on retrieval comprehensiveness in historical data, and w2 is determined by the influence coefficient of hit rate on retrieval accuracy. The mean μ1 and standard deviation σ1 of the comprehensive scores Zi of all keyword combinations in the first candidate set are statistically analyzed to calculate the corresponding preset score threshold Z0, where Z0 = μ1 + γ2 × σ1, and γ2 is an adjustment factor. The comprehensive score Zi of each keyword combination Ci is compared with the preset score threshold Z0. If Zi > Z0, the keyword combination Ci is included in the second candidate set; otherwise, it is removed.

[0023] Furthermore, S4 includes the following:

[0024] For all keyword combinations in the second candidate set, extract the comprehensive score Zi for each keyword combination, calculate the mean μ2 and standard deviation σ2 of the comprehensive scores Zi for the second candidate set, and set a consistency check threshold range [Z_min, Z_max], where Z_min = μ2 - 1.5σ_2 and Z_max = μ2 + 1.5σ_2. Check whether the comprehensive score Zi of each keyword combination in the second candidate set is within the range [Z_min, Z_max]: if Zi < Z_min or Zi > Z_max, the combination is considered to have an abnormal comprehensive score due to data statistical bias (such as accidental high hit rate or abnormal coverage caused by missing local news data) and is removed; if Zi is within the range [Z_min, Z_max], the combination is retained.

[0025] Collect all retained keyword combinations to form the final optimized keyword combination set; establish a search keyword database, and store the final optimized keyword combination set in descending order of comprehensive score Zi. The search keyword database is equipped with an automatic update mechanism, and steps S1-S4 are repeated every 3 months to update the combinations in the database; when a user's news information search request is received, extract the core demand keywords in the search request, match keyword combinations with a semantic similarity ≥ 0.7 (based on cosine similarity calculation) with the core demand keywords from the search keyword database, call the combination to perform search matching, and return the search results.

[0026] The keyword retrieval optimization system based on AI intelligent analysis includes: a database construction module, a first candidate set screening module, a second candidate set screening module, and a verification and retrieval application module;

[0027] The database construction module is based on the target news retrieval scenario. It collects historical news data and user search records within a specified time period, cleans invalid search records, extracts unique keyword combinations and matches related data to build a keyword combination database.

[0028] The first candidate set screening module extracts keywords and domain tags for each keyword combination in the database, filters semantically related combinations, calculates the proportion of duplicate information and the degree of semantic overlap, constructs a redundancy index and compares it with the threshold, thereby obtaining the first candidate set.

[0029] The second candidate set screening module calculates the coverage and matching hit rate of the combinations in the first candidate set, establishes a comprehensive evaluation system and calculates the score, and filters the results based on the score threshold to form the second candidate set.

[0030] The verification and retrieval application module eliminates combinations with abnormal scores in the second candidate set through consistency verification, generates the final optimized set, and builds a retrieval keyword library, supporting retrieval request matching and regular updates.

[0031] Furthermore, the database construction module includes a multi-source data acquisition unit and a database integration unit;

[0032] The multi-source data acquisition unit focuses on the target scenario, collecting historical news data including titles, body text, and keyword tags, as well as user search records including search combinations, time, click volume, and success indicators; the database integration unit cleans invalid search records, extracts unique keyword combinations from valid records and matches them with search data, and combines them with historical news data to build a complete keyword combination database.

[0033] Furthermore, the first candidate set screening module includes a semantically relevant screening unit and a redundancy calculation screening unit;

[0034] The semantically relevant filtering unit extracts the keywords and domain tags of each combination, calculates the tag overlap rate with other combinations, and filters out semantically relevant combinations based on the threshold. The redundancy calculation filtering unit calculates the proportion of duplicate information between the target combination and related combinations, as well as the semantic overlap of keywords within the combination, constructs a redundancy index and compares it with the threshold, thereby obtaining the first candidate set.

[0035] Furthermore, the second candidate set screening module includes a coverage hit rate calculation unit and a comprehensive score screening unit;

[0036] The coverage hit rate calculation unit counts the number of news items associated with the combination in the first candidate set and calculates the coverage; it extracts the total number of searches and the number of successful searches for the combination and calculates the matching hit rate; the comprehensive scoring and screening unit sets weight coefficients to construct a comprehensive evaluation system, calculates the combination score, and combines the score threshold to screen out the second candidate set.

[0037] Furthermore, the verification and retrieval application module includes a scoring verification generation unit and a keyword database management unit;

[0038] The scoring verification generation unit determines the scoring verification range of the second candidate set, eliminates abnormal scoring combinations, and collects valid combinations to form the final optimized keyword combination set; the keyword library management unit sorts and stores the final optimized set to build a search keyword library and sets up a periodic update mechanism; when a search request is received, it matches relevant combinations and executes the search, and returns the search results.

[0039] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention constructs a keyword combination database, using basic data such as total search volume, success rate, and coverage as its core, and establishes a multi-dimensional quantitative evaluation system encompassing redundancy, coverage, and hit rate. All screening steps are based on data calculation rather than subjective judgment, significantly improving the scientific rigor and objectivity of keyword combination optimization. This invention filters semantically relevant combinations through domain tag overlap rate, and constructs a redundancy index by combining the proportion of duplicate information and the degree of semantic overlap, accurately eliminating highly redundant combinations; reducing the amount of duplicate information processed during the retrieval process, lowering the system's data computation load, and effectively meeting the speed requirements of real-time news information retrieval. This invention constructs a comprehensive scoring system by quantitatively calculating coverage and matching hit rate, selecting combinations that combine wide coverage and high hit accuracy, avoiding omissions or redundancy in search results, and significantly improving the comprehensiveness and accuracy of user information acquisition. Attached Figure Description

[0040] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0041] Figure 1 This is a flowchart illustrating the keyword retrieval optimization method based on AI intelligent analysis according to the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Please see Figure 1 The present invention provides the following technical solution:

[0044] The keyword retrieval optimization method based on AI intelligent analysis includes the following steps:

[0045] S1. Collect historical news data and historical search records in the target news search scenario, extract all used keyword combinations from the historical search records, integrate the keyword combinations with the corresponding historical search data, and establish a keyword combination database.

[0046] S2. Perform redundancy analysis on each keyword combination in the keyword combination database. Use an AI intelligent model to analyze the proportion of duplicate information contained in each keyword combination and the degree of semantic overlap between keywords within the combination, and construct a redundancy evaluation index. Calculate the redundancy value of each combination, compare the redundancy value with a preset redundancy threshold, and select keyword combinations with redundancy values ​​lower than the preset redundancy threshold to form the first candidate set.

[0047] S3. Conduct coverage and matching hit rate analysis on the keyword combinations in the first candidate set. Use an AI intelligent model to analyze the number of news items corresponding to each keyword in historical news data and the scope of news that each keyword combination can cover, and calculate the coverage of each combination. Extract the number of successful searches and the total number of searches corresponding to each keyword combination from historical search records, and calculate the matching hit rate of each combination. Establish a comprehensive evaluation system of coverage and hit rate, and substitute the coverage and matching hit rate of each combination into the evaluation system for comprehensive scoring. Select keyword combinations with comprehensive scores higher than the preset scoring threshold to form the second candidate set.

[0048] S4. Perform consistency verification on the keyword combinations in the second candidate set, and remove keyword combinations that have abnormal comprehensive scores due to data statistical deviations to obtain the final optimized keyword combination set; store the final optimized keyword combination set in the retrieval keyword library, and call the keyword combinations in the retrieval keyword library to perform retrieval matching when a subsequent news information retrieval request is triggered.

[0049] S1 includes the following:

[0050] The scope of the target news search scenario is determined, and historical news data within this scenario over the past three years is collected. The historical news data includes news titles, text content, publication time, and keyword tags. At the same time, the historical search records of users in this scenario over the past two years are also collected. The historical search records include the keyword combinations used in the search, the search time, the number of clicks on the search results, and the search success indicator.

[0051] The collected historical search records are cleaned to remove invalid records with empty search times or keyword combination character lengths of less than 2, and valid historical search records are retained; all unique keyword combinations are extracted from the valid historical search records to create an initial keyword combination list;

[0052] For each keyword combination in the initial keyword combination list, match the corresponding historical search data to form a keyword combination-related data mapping table; the historical search data includes the total number of searches for that combination, the number of successful searches, and the average number of clicks on the search results;

[0053] Based on the keyword combination-related data mapping table and the collected historical news data, a keyword combination database is constructed; the fields of the database include keyword combination ID, keyword combination content, total number of searches, number of successful searches, average number of clicks on search results, and a set of associated historical news IDs.

[0054] In this embodiment, historical search-related data is matched for each keyword combination in the initial keyword combination list to form a keyword combination-related data mapping table; the specific content is as follows:

[0055] Assuming the target news search scenario is "technology news," 1000 valid historical search records were obtained after data cleaning. From these, 200 unique keyword combinations were extracted to form an initial keyword combination list, including terms such as "artificial intelligence applications," "blockchain technology," and "5G development."

[0056] Take the keyword combination "artificial intelligence application" as an example:

[0057] Traversing the valid historical search records, it was found that this combination appeared a total of 30 times, so the total number of searches was 30.

[0058] Based on the successful search indicator (user marked "relevant news found"), 25 of them were successful searches, so the number of successful searches is 25.

[0059] The click counts for the 30 searches were 5, 8, 10... (30 data points in total). Assuming the arithmetic mean is 7.2, the average click count for the search results is 7.2.

[0060] Similarly, the above statistical process was performed on the remaining 199 keyword combinations, such as "blockchain technology" and "5G development," to ultimately form a mapping table. For example, one record is: "Artificial Intelligence Applications" - 30 - 25 - 7.2.

[0061] S2 includes the following:

[0062] For each keyword combination Ci in the keyword combination database, extract all keywords kij contained in the combination, where i ranges from 1 to n, n is the total number of keyword combinations, and j ranges from 1 to m, where m is the number of keywords in a single combination. For each keyword combination Ci, extract the domain labels Bi of all corresponding keywords kij. Iterate through all other keyword combinations Cp in the keyword combination database and extract the domain labels Bp corresponding to the keywords kpq contained in them. Similarly, p ranges from 1 to n, and p ≠ i, and q ranges from 1 to m. Calculate the domain label overlap rate L between Ci and Cp, using the formula: L = N(Bi∩Bp) / N(Bi∪Bp), where N(Bi∩Bp) is the number of identical domain labels between keyword combination Ci and keyword combination Cp; N(Bi∪Bp) is the total number of domain labels between keyword combination Ci and keyword combination Cp. When the domain label overlap rate L between keyword combination Ci and keyword combination Cp is greater than or equal to a preset threshold L0, keyword combination Ci and keyword combination Cp are determined to be semantically related combinations.

[0063] In this embodiment, assuming that in the context of science and technology news, Ci is "artificial intelligence medical application", and its domain label is ["artificial intelligence", "medical technology", "application scenario"]. When Cp is "AI medical innovation", its domain label is ["artificial intelligence", "medical technology", "innovative application"]. The common domain labels of Ci and Cp are "artificial intelligence" and "medical technology", i.e., N(Bi∩Bp)=2; the total domain label is the union of the two labels ("artificial intelligence", "medical technology", "application scenario", "innovative application"), i.e., N(Bi∪Bp)=4, so the overlap rate L=2 / 4=0.5; assuming the threshold L0=0.3, then Ci and Cp are determined to be a semantically related combination.

[0064] Summarize all keyword combinations Cp that are semantically related to keyword combination Ci, count the number of common keywords tip between keyword combination Ci and keyword combination Cp, calculate the keyword overlap rate rip between keyword combination Ci and each keyword combination Cp, and rip=tip / max(m,mp), where mp is the number of keywords contained in keyword combination Cp; take the average of the keyword overlap rates between keyword combination Ci and all related keyword combinations Cp as the proportion of repeated information Ri; obtain the semantic vector Vij of each keyword kij through an AI intelligent model (e.g., based on the Word2Vec pre-trained model), calculate the semantic similarity sju=cos(Vij,Viu) between any two keywords kij and kiu (j≠u) within the combination, where cos() is the cosine similarity function, and take the average of the semantic similarity between all keywords within the combination as the degree of semantic overlap Si;

[0065] Based on the proportion of duplicate information Ri and the degree of semantic overlap Si, a redundancy evaluation index Yi is constructed, and Yi = α × Ri + β × Si, where α and β are weight coefficients, and α + β = 1. α and β are determined by fitting the retrieval efficiency loss rate of redundant combinations in historical data. The mean μ_Y and standard deviation σ_Y of the redundancy evaluation index Yi of all keyword combinations in the keyword combination database are statistically analyzed, and the corresponding redundancy threshold Y0 is calculated, and Y0 = μ_Y - γ1 × σ_Y, where γ1 is an adjustment factor. The redundancy evaluation index Yi of each keyword combination Ci is compared with the redundancy threshold Y0. If Yi < Y0, the keyword combination Ci is included in the first candidate set; otherwise, it is discarded.

[0066] S3 includes the following:

[0067] For each keyword combination Ci in the first candidate set, extract the set of historical news IDs associated with that combination from the keyword combination database, count the number of historical news items Ni contained in that set, and simultaneously count the total number of historical news data N_total in the target news retrieval scenario; calculate the coverage Cov_i of keyword combination Ci, and Cov_i = Ni / N_total; extract the total number of searches Ti and the number of successful searches Ji of keyword combination Ci from the keyword combination database, and calculate the matching hit rate Hi of keyword combination Ci, with the formula: Hi = Ji / Ti;

[0068] A comprehensive evaluation system for coverage and hit rate is established by combining the coverage (Cov_i) and matching hit rate (Hi) of keyword combination Ci. A comprehensive score Zi is calculated, where Zi = w1 × Cov_i + w2 × Hi, and w1 and w2 are weighting coefficients, with w1 + w2 = 1. w1 is determined by the influence coefficient of coverage on retrieval comprehensiveness in historical data, and w2 is determined by the influence coefficient of hit rate on retrieval accuracy. The mean μ1 and standard deviation σ1 of the comprehensive scores Zi of all keyword combinations in the first candidate set are statistically analyzed to calculate the corresponding preset score threshold Z0, where Z0 = μ1 + γ2 × σ1, and γ2 is an adjustment factor. The comprehensive score Zi of each keyword combination Ci is compared with the preset score threshold Z0. If Zi > Z0, the keyword combination Ci is included in the second candidate set; otherwise, it is removed.

[0069] S4 includes the following:

[0070] For all keyword combinations in the second candidate set, extract the comprehensive score Zi for each keyword combination, calculate the mean μ2 and standard deviation σ2 of the comprehensive scores Zi for the second candidate set, and set a consistency check threshold range [Z_min, Z_max], where Z_min = μ2 - 1.5σ_2 and Z_max = μ2 + 1.5σ_2. Check whether the comprehensive score Zi of each keyword combination in the second candidate set is within the range [Z_min, Z_max]: if Zi < Z_min or Zi > Z_max, the combination is considered to have an abnormal comprehensive score due to data statistical bias (such as accidental high hit rate or abnormal coverage caused by missing local news data) and is removed; if Zi is within the range [Z_min, Z_max], the combination is retained.

[0071] Collect all retained keyword combinations to form the final optimized keyword combination set; establish a search keyword database, and store the final optimized keyword combination set in descending order of comprehensive score Zi. The search keyword database is equipped with an automatic update mechanism, and steps S1-S4 are repeated every 3 months to update the combinations in the database; when a user's news information search request is received, extract the core demand keywords in the search request, match keyword combinations with a semantic similarity ≥ 0.7 (based on cosine similarity calculation) with the core demand keywords from the search keyword database, call the combination to perform search matching, and return the search results.

[0072] The keyword retrieval optimization system based on AI intelligent analysis includes: a database construction module, a first candidate set screening module, a second candidate set screening module, and a verification and retrieval application module;

[0073] The database construction module is based on the target news retrieval scenario. It collects historical news data and user search records within a specified time period, cleans invalid search records, extracts unique keyword combinations and matches related data to build a keyword combination database.

[0074] The first candidate set screening module extracts keywords and domain tags for each keyword combination in the database, filters semantically related combinations, calculates the proportion of duplicate information and the degree of semantic overlap, constructs a redundancy index and compares it with the threshold, thereby obtaining the first candidate set.

[0075] The second candidate set screening module calculates the coverage and matching hit rate of the combinations in the first candidate set, establishes a comprehensive evaluation system and calculates the score, and filters the results based on the score threshold to form the second candidate set.

[0076] The verification and retrieval application module eliminates combinations with abnormal scores in the second candidate set through consistency verification, generates the final optimized set, and builds a retrieval keyword library, supporting retrieval request matching and regular updates.

[0077] The database construction module includes a multi-source data acquisition unit and a database integration unit;

[0078] The multi-source data acquisition unit focuses on the target scenario, collecting historical news data including titles, body text, and keyword tags, as well as user search records including search combinations, time, click volume, and success indicators; the database integration unit cleans invalid search records, extracts unique keyword combinations from valid records and matches them with search data, and combines them with historical news data to build a complete keyword combination database.

[0079] The first candidate set screening module includes a semantically relevant screening unit and a redundant calculation screening unit;

[0080] The semantically relevant filtering unit extracts the keywords and domain tags of each combination, calculates the tag overlap rate with other combinations, and filters out semantically relevant combinations based on the threshold. The redundancy calculation filtering unit calculates the proportion of duplicate information between the target combination and related combinations, as well as the semantic overlap of keywords within the combination, constructs a redundancy index and compares it with the threshold, thereby obtaining the first candidate set.

[0081] The second candidate set screening module includes a coverage hit rate calculation unit and a comprehensive score screening unit;

[0082] The coverage hit rate calculation unit counts the number of news items associated with the combination in the first candidate set and calculates the coverage; it extracts the total number of searches and the number of successful searches for the combination and calculates the matching hit rate; the comprehensive scoring and screening unit sets weight coefficients to construct a comprehensive evaluation system, calculates the combination score, and combines the score threshold to screen out the second candidate set.

[0083] The verification and retrieval application module includes a scoring verification generation unit and a keyword database management unit;

[0084] The scoring verification generation unit determines the scoring verification range of the second candidate set, eliminates abnormal scoring combinations, and collects valid combinations to form the final optimized keyword combination set; the keyword library management unit sorts and stores the final optimized set to build a search keyword library and sets up a periodic update mechanism; when a search request is received, it matches relevant combinations and executes the search, and returns the search results.

[0085] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0086] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A keyword retrieval optimization method based on AI intelligent analysis, characterized by: The method includes the following steps: S1. Collect historical news data and historical search records in the target news search scenario, extract all used keyword combinations from the historical search records, integrate the keyword combinations with the corresponding historical search data, and establish a keyword combination database. S2. Perform redundancy analysis on each keyword combination in the keyword combination database. Use an AI intelligent model to analyze the proportion of duplicate information contained in each keyword combination and the degree of semantic overlap between keywords within the combination, and construct a redundancy evaluation index. Calculate the redundancy value of each combination, compare the redundancy value with a preset redundancy threshold, and select keyword combinations with redundancy values ​​lower than the preset redundancy threshold to form the first candidate set. S2 includes the following: For each keyword combination Ci in the keyword combination database, extract all keywords kij contained in the combination, where i ranges from 1 to n, n is the total number of keyword combinations, and j ranges from 1 to m, where m is the number of keywords in a single combination. For each keyword combination Ci, extract the domain tags Bi of all corresponding keywords kij. Iterate through all other keyword combinations Cp in the keyword combination database and extract the domain tags Bp corresponding to the keywords kpq contained in them. Similarly, p ranges from 1 to n, and p ≠ i, and q ranges from 1 to m. Calculate the domain tag overlap rate L between Ci and Cp, using the formula: L = N(Bi∩Bp) / N(Bi∪Bp), where N(Bi∩Bp) is the number of domain tags that keyword combination Ci and keyword combination Cp share; N(Bi∪Bp) is the total number of domain tags between keyword combination Ci and keyword combination Cp. When the domain tag overlap rate L between keyword combination Ci and keyword combination Cp is greater than or equal to a preset threshold L0, keyword combination Ci and keyword combination Cp are determined to be semantically related combinations. Summarize all keyword combinations Cp that are semantically related to keyword combination Ci, count the number of common keywords tip between keyword combination Ci and keyword combination Cp, calculate the keyword overlap rate rip between keyword combination Ci and each keyword combination Cp, and rip = tip / max(m, mp), where mp is the number of keywords contained in keyword combination Cp; take the average keyword overlap rate between keyword combination Ci and all related keyword combinations Cp as the proportion of repeated information Ri; obtain the semantic vector Vij of each keyword kij through the AI ​​intelligent model, calculate the semantic similarity sju = cos(Vij, Viu) between any two keywords kij and kiu (j≠u) in the combination, where cos() is the cosine similarity function, and take the average semantic similarity between all keywords in the combination as the semantic overlap Si; Based on the proportion of duplicate information Ri and the degree of semantic overlap Si, a redundancy evaluation index Yi is constructed, and Yi = α × Ri + β × Si, where α and β are weight coefficients, and α + β = 1. α and β are determined by fitting the retrieval efficiency loss rate of redundant combinations in historical data. The mean μ_Y and standard deviation σ_Y of the redundancy evaluation index Yi of all keyword combinations in the keyword combination database are statistically analyzed, and the corresponding redundancy threshold Y0 is calculated, and Y0 = μ_Y - γ1 × σ_Y, where γ1 is an adjustment factor. The redundancy evaluation index Yi of each keyword combination Ci is compared with the redundancy threshold Y0. If Yi < Y0, the keyword combination Ci is included in the first candidate set; otherwise, it is removed. S3. Conduct coverage and matching hit rate analysis on the keyword combinations in the first candidate set. Use an AI intelligent model to analyze the number of news items corresponding to each keyword in historical news data and the scope of news that each keyword combination can cover, and calculate the coverage of each combination. Extract the number of successful searches and the total number of searches corresponding to each keyword combination from historical search records, and calculate the matching hit rate of each combination. Establish a comprehensive evaluation system of coverage and hit rate, and substitute the coverage and matching hit rate of each combination into the evaluation system for comprehensive scoring. Select keyword combinations with comprehensive scores higher than the preset scoring threshold to form the second candidate set. S4. Perform consistency verification on the keyword combinations in the second candidate set, and remove keyword combinations that have abnormal comprehensive scores due to data statistical deviations to obtain the final optimized keyword combination set; store the final optimized keyword combination set in the retrieval keyword library, and call the keyword combinations in the retrieval keyword library to perform retrieval matching when a subsequent news information retrieval request is triggered.

2. The keyword retrieval optimization method based on AI intelligent analysis according to claim 1, characterized in that: S1 includes the following: The scope of the target news search scenario is determined, and historical news data within this scenario over the past three years is collected. The historical news data includes news titles, text content, publication time, and keyword tags. At the same time, the historical search records of users in this scenario over the past two years are also collected. The historical search records include the keyword combinations used in the search, the search time, the number of clicks on the search results, and the search success indicator. The collected historical search records are cleaned to remove invalid records with empty search times or keyword combination character lengths of less than 2, and valid historical search records are retained; all unique keyword combinations are extracted from the valid historical search records to create an initial keyword combination list; For each keyword combination in the initial keyword combination list, match the corresponding historical search data to form a keyword combination-related data mapping table; the historical search data includes the total number of searches for that combination, the number of successful searches, and the average number of clicks on the search results; Based on the keyword combination-related data mapping table and the collected historical news data, a keyword combination database is constructed; the fields of the database include keyword combination ID, keyword combination content, total number of searches, number of successful searches, average number of clicks on search results, and a set of associated historical news IDs.

3. The keyword retrieval optimization method based on AI intelligent analysis according to claim 1, characterized in that: S3 includes the following: For each keyword combination Ci in the first candidate set, extract the set of historical news IDs associated with that combination from the keyword combination database, count the number of historical news items Ni contained in that set, and simultaneously count the total number of historical news data N_total in the target news retrieval scenario; calculate the coverage Cov_i of keyword combination Ci, and Cov_i = Ni / N_total; extract the total number of searches Ti and the number of successful searches Ji of keyword combination Ci from the keyword combination database, and calculate the matching hit rate Hi of keyword combination Ci, with the formula: Hi = Ji / Ti; A comprehensive evaluation system for coverage and hit rate is established by combining the coverage (Cov_i) and matching hit rate (Hi) of keyword combination Ci. The comprehensive score Zi is calculated, and Zi = w1 × Cov_i + w2 × Hi, where w1 and w2 are weight coefficients, and w1 + w2 = 1. The mean μ1 and standard deviation σ1 of the comprehensive scores Zi of all keyword combinations in the first candidate set are statistically analyzed to calculate the corresponding preset score threshold Z0, and Z0 = μ1 + γ2 × σ1, where γ2 is an adjustment factor. The comprehensive score Zi of each keyword combination Ci is compared with the preset score threshold Z0. If Zi > Z0, the keyword combination Ci is included in the second candidate set; otherwise, it is removed.

4. The keyword retrieval optimization method based on AI intelligent analysis according to claim 3, characterized in that: S4 includes the following: For all keyword combinations in the second candidate set, extract the comprehensive score Zi for each keyword combination, calculate the mean μ2 and standard deviation σ2 of the comprehensive scores Zi for the second candidate set, and set a consistency check threshold range [Z_min, Z_max], where Z_min = μ2 - 1.5σ_2 and Z_max = μ2 + 1.5σ_2; check whether the comprehensive score Zi of each keyword combination in the second candidate set is within the range [Z_min, Z_max]: if Zi < Z_min or Zi > Z_max, the combination is determined to have an abnormal comprehensive score due to data statistical bias and is removed; if Zi is within the range [Z_min, Z_max], the combination is retained. Collect all retained keyword combinations to form the final optimized keyword combination set; establish a search keyword database, and store the final optimized keyword combination set in descending order of comprehensive score Zi. The search keyword database is equipped with an automatic update mechanism, and steps S1-S4 are repeated every 3 months to update the combinations in the database; when a user's news information search request is received, extract the core demand keywords in the search request, match keyword combinations with a semantic similarity ≥ 0.7 with the core demand keywords from the search keyword database, call the combination to perform search matching, and return the search results.

5. A keyword retrieval optimization system based on AI intelligent analysis, applied to the keyword retrieval optimization method based on AI intelligent analysis as described in any one of claims 1-4, characterized in that: The system includes: a database construction module, a first candidate set filtering module, a second candidate set filtering module, and a verification and retrieval application module; The database construction module, based on the target news retrieval scenario, collects historical news data and user search records within a specified time period, cleans invalid search records, extracts unique keyword combinations, matches related data, and constructs a keyword combination database. The first candidate set screening module extracts keywords and domain tags for each keyword combination in the database, filters semantically related combinations, calculates the proportion of duplicate information and the degree of semantic overlap, constructs a redundancy index and compares it with a threshold, thereby obtaining the first candidate set. The second candidate set screening module calculates the coverage and matching hit rate of the combinations in the first candidate set, establishes a comprehensive evaluation system and calculates the score, and filters the results based on the score threshold to form the second candidate set. The verification and retrieval application module eliminates combinations with abnormal scores in the second candidate set through consistency verification, generates the final optimized set, and constructs a retrieval keyword library, supporting retrieval request matching and regular updates.

6. The keyword retrieval optimization system based on AI intelligent analysis according to claim 5, characterized in that: The database construction module includes a multi-source data acquisition unit and a database integration unit; The multi-source data acquisition unit focuses on the target scenario, collecting historical news data including titles, body text, and keyword tags, as well as user search records including search combinations, time, click volume, and success indicators; the database integration unit cleans invalid search records, extracts non-repeating keyword combinations from valid records and matches them with search data, and combines them with historical news data to construct a complete keyword combination database.

7. The keyword retrieval optimization system based on AI intelligent analysis according to claim 5, characterized in that: The first candidate set filtering module includes a semantically related filtering unit and a redundancy calculation filtering unit; The semantically relevant filtering unit extracts keywords and domain tags for each combination, calculates the tag overlap rate with other combinations, and filters out semantically relevant combinations based on a threshold. The redundancy calculation filtering unit calculates the proportion of duplicate information between the target combination and related combinations, as well as the semantic overlap of keywords within the combination, constructs a redundancy index and compares it with a threshold, thereby obtaining the first candidate set.

8. The keyword retrieval optimization system based on AI intelligent analysis according to claim 5, characterized in that: The second candidate set screening module includes a coverage hit rate calculation unit and a comprehensive score screening unit; The coverage hit rate calculation unit counts the number of news items associated with the combination in the first candidate set and calculates the coverage; it extracts the total number of searches and the number of successful searches for the combination and calculates the matching hit rate; the comprehensive scoring and filtering unit sets weight coefficients to construct a comprehensive evaluation system, calculates the combination score, and combines the score threshold to filter out the second candidate set.

9. The keyword retrieval optimization system based on AI intelligent analysis according to claim 5, characterized in that: The verification and retrieval application module includes a scoring verification generation unit and a keyword database management unit; The scoring verification generation unit determines the scoring verification range of the second candidate set, eliminates abnormal scoring combinations, and collects valid combinations to form the final optimized keyword combination set; the keyword library management unit sorts and stores the final optimized set to build a retrieval keyword library and sets up a periodic update mechanism; when a retrieval request is received, it matches relevant combinations and executes the retrieval, and returns the retrieval results.

Images