An AI-based topic selection ability assessment system based on adaptive evaluation

Abstract

This invention relates to the field of AI-assisted research topic selection technology, and particularly to an AI topic selection ability assessment system based on adaptive evaluation. The system includes a topic sampling module, which extracts topics to form a set of topics to be tested; a literature retrieval module, which retrieves core keywords to form a comparative literature set; a trend analysis module, which determines the research trend cycle of the field based on the comparative literature set; a feature analysis module, which analyzes assessment feature parameters based on each topic, the comparative literature set, and the research trend cycle of the field; a topic anomaly identification module, which determines whether a topic is suspected of being abnormal based on the assessment feature parameters, forming a set of suspected abnormal topics; a misjudgment correction module, which calculates the coverage rate of each topic to form a set of abnormal topics; a system anomaly determination module, which determines whether the target AI topic selection system is abnormal based on the abnormal topic set; and an adaptive optimization module, which automatically adjusts all thresholds in the assessment system. This invention significantly improves the accuracy of AI topic selection ability assessment.

Description

Technical Field

[0001] This invention relates to the field of AI-assisted scientific research topic selection technology, and in particular to an AI topic selection ability assessment system based on adaptive evaluation. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the widespread application of large language models in text generation, AI-assisted research topic selection has become an important tool for improving research efficiency and stimulating innovative thinking. However, the quality of current AI-generated research topics varies greatly, often exhibiting a lack of originality, high repetition with existing research, or even logically inconsistent patchwork. Furthermore, the lack of an efficient, objective, and quantifiable automated evaluation system to continuously monitor and assess the reliability of AI topic selection systems makes it difficult for researchers to trust the results, severely restricting the in-depth application and value realization of this technology in the scientific research field.

[0003] Chinese Patent Publication No. CN119513293A discloses a method and apparatus for recommending research topics based on similarity calculation. This invention relates to a method for recommending research topics based on similarity calculation, comprising: acquiring and preprocessing scientific literature data; extracting keywords from abstracts and titles using the TextRank algorithm and identifying technical keywords through expert identification; vectorizing the keywords using a trained word2vec algorithm; vectorizing user-input keywords and calculating their cosine similarity with each vector in the keyword vector table; selecting the top few scientific literatures and their corresponding scholars and technical keywords, and displaying them to the user through visualization technology. This invention determines the topics of interest to the user by calculating the cosine similarity between keyword vectors, recommending highly relevant literature, scholars, and technologies to help users quickly find high-quality literature, leading figures in the field, and possible research methods related to their topics of interest.

[0004] Therefore, it is evident that the existing technology has the following problems:

[0005] Currently, in the field of AI-assisted scientific research topic selection, there is a lack of an efficient evaluation mechanism that can be automatic, accurate, and adaptive to the research trends in the field. Summary of the Invention

[0006] To address this issue, the present invention provides an AI topic selection ability assessment system based on adaptive evaluation, which overcomes the problem in the current field of AI-assisted scientific research topic selection that the topic selection systems use the same topic selection model for topic recommendation and cannot adjust the model system according to the specific topic selection field, resulting in low evaluation accuracy of the topic selection system.

[0007] To achieve the above objectives, this invention provides an AI topic selection ability assessment system based on adaptive evaluation, comprising:

[0008] The topic selection sampling module is used to randomly select a preset number of topics from the target AI topic selection system within a preset evaluation period to form a set of topics to be tested. The topics include the title and core keywords of the topics.

[0009] The literature retrieval module, based on each topic in the test topic set, calls a semantic retrieval model to retrieve the core keywords of each topic to determine a number of comparative documents to form a comparative document set; the comparative document set includes several comparative documents, the year of the comparative documents, the number of times the comparative documents were retrieved, and the similarity between the comparative documents and the corresponding topics.

[0010] The trend analysis module extracts domain research trend characteristic parameters based on the comparative literature set to determine the domain research trend cycle. The domain research trend characteristic parameters include the number of comparative literature published in each year and the number of comparative literature searches in each year.

[0011] The feature analysis module analyzes the evaluation feature parameters of each topic based on each topic, the corresponding comparative literature set, and the research trend cycle of the field. The evaluation feature parameters include the number of highly matched comparative literatures corresponding to the topic and the semantic coherence of the topic.

[0012] The topic selection anomaly identification module determines whether a topic is suspected of being abnormal based on the evaluation feature parameters of each topic and the research trend cycle of the field, and generates a set of suspected abnormal topics.

[0013] The misjudgment correction module calculates the coverage rate of keywords in related fields for each topic in the suspected abnormal topic set, and corrects the suspected abnormal topic set based on the coverage rate to form an abnormal topic set;

[0014] The system anomaly detection module determines whether the target AI topic selection system is abnormal based on the number of topics selected in the abnormal topic selection set.

[0015] The adaptive optimization module calls the adaptive learning model to dynamically adjust the judgment threshold in the AI ​​topic selection ability assessment system based on the results of historical judgments.

[0016] Furthermore, the trend analysis module determines the research trend cycle of the field based on the difference between the maximum value of the year corresponding to the peak number of publications of each historical document in the comparative literature set and the year corresponding to the peak number of retrievals of comparative literature in the past years and the current year.

[0017] If the difference is less than or equal to a preset threshold, then the domain research trend cycle is determined to be in the domain research active cycle;

[0018] If the difference is greater than a preset threshold, then the domain research trend cycle is determined to be in a domain research decline cycle.

[0019] Furthermore, the feature analysis module determines the number of highly matched comparative documents corresponding to each topic based on the number of comparative documents whose similarity to each topic is greater than a similarity threshold.

[0020] Furthermore, the feature analysis module determines the similarity threshold based on the research trend cycle of the corresponding field for each topic;

[0021] If the research trend cycle of the field is in an active research cycle, then the similarity threshold is determined as the first similarity threshold; if the research trend cycle of the field is in a declining research cycle, then the similarity threshold is determined as the second similarity threshold.

[0022] Furthermore, the feature analysis module calculates the internal similarity of the titles of each topic to determine the semantic coherence of each topic;

[0023] The internal similarity of the titles is determined by calculating the cosine similarity of the titles of each split topic.

[0024] The titles of each split topic include two text segments generated after splitting the title based on preset stop words.

[0025] Furthermore, the topic selection anomaly identification module is used to determine that the topic selection is suspected of being abnormal when the number of highly matched comparative documents corresponding to the topic is greater than the threshold for the number of highly matched comparative documents and the corresponding semantic coherence is less than the threshold for semantic coherence.

[0026] Furthermore, the topic selection anomaly identification module determines the threshold for the number of highly matched comparative documents and the semantic coherence threshold for each topic based on the research trend cycle of the corresponding field.

[0027] Furthermore, the misjudgment correction module calculates the proportion of the core keywords of the selected topic to the core keywords of the selected topic based on the core keywords of the selected topic, and determines it as the coverage rate of the selected topic.

[0028] Furthermore, the misjudgment correction module corrects the suspected abnormal topic set based on the coverage rate, and after removing topics in the suspected abnormal topic set whose coverage rate is greater than a preset coverage rate threshold, an abnormal topic set is generated.

[0029] Furthermore, the system anomaly determination module is used to determine that the target AI topic selection system is abnormal based on the comparison result that the number of topics in the abnormal topic selection set is greater than a preset threshold.

[0030] Compared with existing technologies, the beneficial effects of this invention lie in its proposed AI topic selection ability assessment system based on adaptive evaluation. This system constructs a closed-loop system encompassing sampling, retrieval, trend analysis, feature analysis, anomaly identification, misjudgment correction, system judgment, and adaptive optimization. Through the collaborative work of multiple modules, deep features are extracted from massive amounts of data, and an adaptive mechanism is introduced to achieve reliable evaluation. This enables large-scale, high-frequency automated assessment of AI topic selection ability, significantly reducing the cost and subjective bias of manual evaluation. The process-oriented and modular design ensures the standardization and repeatability of the assessment process. A complete closed loop of "measurement-analysis-feedback-optimization" is formed, which not only diagnoses the current state of the AI ​​system but also drives its continuous improvement through adaptive learning, thereby promoting the entire field of AI research assistance towards a more reliable and trustworthy direction and significantly improving the accuracy of AI topic selection ability assessment.

[0031] In particular, by extracting the number of comparative literature publications and the number of comparative literature searches over the years from the comparative literature collections of each research topic, the trend cycle of the field is determined, which is used to perceive and distinguish the macro environment in which the research is situated. Research fields have their own cycles of activity and decline, and the definition and standards of innovation differ in different cycles. This design solves the drawbacks of the "one-size-fits-all" approach of static evaluation systems. On the one hand, it provides crucial contextual information for all subsequent fine-grained evaluations, laying the foundation for adaptive evaluation. On the other hand, this data-driven objective classification method avoids subjective misjudgments of field trends, making the evaluation results more scientific and fair, and accurately identifying whether it is a reasonable follow-up during a boom or a genuine breakthrough during a period of stagnation.

[0032] Furthermore, by using the "number of highly matched comparative documents" as a core feature and dynamically changing its judgment threshold according to the cyclical trends of the field, the key dimension of "novelty" in AI topic selection is precisely quantified. Novelty is not an absolute concept; in active fields, similarities with cutting-edge work are normal; in declining fields, any high similarity warrants caution. This design solves the problem that fixed thresholds cannot adapt to different academic backgrounds. Its beneficial effect is that it realizes an intelligent novelty evaluation mechanism, thereby significantly improving the accuracy of judgment. During active periods, it protects valuable incremental innovation; during declining periods, it can keenly capture low-level repetitive work, thus effectively driving AI models to generate more disruptive topics in obsolete fields.

[0033] Furthermore, the semantic coherence within the title is calculated to evaluate the "logical rationality" of AI-generated topics. A high-quality topic should have a title that clearly and coherently expresses its core content, rather than being a rigid collection of keywords. This design provides a lightweight, computationally achievable method to simulate the initial human judgment of the logicality of a topic, solving the problem of difficulty in assessing internal consistency based solely on keywords. Its advantages lie in its computational efficiency, the absence of reliance on large-scale deep learning models for complex analysis, ease of implementation, and strong interpretability. It effectively filters out low-quality topics with semantically fragmented titles or those seemingly pieced together from different sources, providing an effective technical filter for identifying AI-generated patchwork content.

[0034] Furthermore, a robust anomaly identification mechanism is constructed by defining the criteria for suspected anomalies (requiring both high matching quantity and low coherence) and dynamically adjusting both thresholds according to the domain cycle. Single-criteria triggering is highly prone to false positives, while the "dual-criteria triggering" mechanism significantly improves the confidence level of the judgment. Simultaneously, the dynamic thresholds ensure the applicability of this mechanism across different domain cycles, greatly reducing the false positive rate. Anomalies are only marked when there is sufficient evidence (both numerous similarities and poor logic), avoiding false positives caused by domain hotspots or expression habits. This makes the final set of suspected anomalies more accurate, providing a high-quality data foundation for subsequent correction and system-level judgment, ensuring the reliability of the entire evaluation process.

[0035] Furthermore, by introducing a "coverage" metric and using it to correct suspected outliers, this approach prevents high-quality research topics that delve into the core and fundamental questions of a research field from being mistakenly identified as anomalous. This design addresses the critical issue of potentially "falsely eliminating" core research in initial screening based on similarity and coherence. Its beneficial effect lies in significantly improving the system's fairness and intelligence. It ensures that the evaluation results not only focus on formal novelty but also respect the depth and value of the content, making the entire system's evaluation of AI topic selection more comprehensive and nuanced.

[0036] Furthermore, the overall status of the AI ​​topic selection system is determined by the proportion of anomalous topic selections. While anomalies in individual topic selections may stem from randomness, a persistently high anomaly rate inevitably indicates a systemic problem. This design provides a reliable system health diagnostic strategy. When the anomaly rate exceeds a threshold, it issues a clear warning, indicating the need for intervention and optimization of the target AI system's prompt word engineering, training data, or the model itself. This effectively transforms micro-level topic selection evaluation results into action signals guiding macro-level system improvement, achieving intelligent operation and maintenance. Attached Figure Description

[0037] Figure 1This is a module connection diagram of an AI topic selection ability assessment system based on adaptive evaluation, as described in an embodiment of the present invention.

[0038] Figure 2 A logical decision diagram for determining the research trend cycle of each topic in the embodiments of the present invention;

[0039] Figure 3 This invention provides a logical decision diagram for identifying potential anomalies in each topic selection.

[0040] Figure 4 A logic diagram for determining anomalies in the target AI topic selection system in an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0042] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0043] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0044] Please see Figures 1-4 As shown, this embodiment of the invention provides an AI topic selection ability assessment system based on adaptive evaluation, comprising:

[0045] The topic selection sampling module is used to randomly select a preset number of topics from the target AI topic selection system within a preset evaluation period to form a set of topics to be tested. The topics include the title of the topic and core keywords.

[0046] The document retrieval module, which is connected to the topic sampling module, is used to call the semantic retrieval model to retrieve the core keywords of each topic in the topic set to be tested, and determine a number of comparative documents to form a comparative document set; the comparative document set includes a number of comparative documents, the year of the comparative documents, the number of times the comparative documents were retrieved, and the similarity between the comparative documents and the corresponding topics.

[0047] The trend analysis module, which is connected to the literature retrieval module, is used to extract field research trend characteristic parameters based on the comparative literature set in order to determine the field research trend cycle. The field research trend characteristic parameters include the number of comparative literature published in each year and the number of comparative literature searches in each year.

[0048] The feature analysis module, which is connected to the trend analysis module, the topic sampling module and the literature retrieval module, is used to analyze the evaluation feature parameters of each topic based on each topic, the corresponding comparative literature set and the research trend cycle of the field. The evaluation feature parameters include the number of highly matched comparative literatures corresponding to the topic and the semantic coherence of the topic.

[0049] The topic selection anomaly identification module is connected to the feature analysis module, the topic selection sampling module, and the trend analysis module. It is used to determine whether the topic selection is suspected of being abnormal based on the evaluation feature parameters of each topic selection and the research trend cycle of the field, and to generate a set of suspected abnormal topics.

[0050] The misjudgment correction module is connected to the topic selection anomaly identification module, the topic selection sampling module, and the trend analysis module. It is used to calculate the coverage rate of keywords in related fields for each topic in the suspected abnormal topic selection set, and to correct the suspected abnormal topic selection set based on the coverage rate to form an abnormal topic selection set.

[0051] The system anomaly determination module, which is connected to the misjudgment correction module, is used to determine whether the target AI topic selection system is abnormal based on the number of topics in the abnormal topic selection set.

[0052] An adaptive optimization module, which is connected to the feature analysis module, the topic selection anomaly identification module, the misjudgment correction module, and the system anomaly judgment module, is used to call the adaptive learning model and dynamically adjust the judgment threshold in the AI ​​topic selection ability assessment system based on the results of historical judgments.

[0053] In this embodiment, a topic sampling module randomly selects a preset number (preferably 100) of topics from the target AI topic selection system within the evaluation period (preferably one week) to ensure the randomness and representativeness of the evaluation. Each topic is structured and stored as {topic ID, title, core keywords}, forming a set of topics to be tested. The literature retrieval module uses the core keywords of each topic as search terms and calls a semantic retrieval model (preferably a BERT-based neural network vector retrieval model) to retrieve relevant papers for each topic. The top N (preferably 50) papers in the search results are extracted, along with their {document ID, publication year, number of searches, and semantic similarity to the topic}, forming a set of comparative literature for each topic. The trend analysis module extracts the number of comparative literature publications and the number of comparative literature searches over the years from the comparative literature set to analyze the research trend cycle in the field. The feature analysis module calculates two core evaluation feature parameters for each topic, including the number of highly matched comparative literatures and semantic coherence. The topic selection anomaly identification module initially screens out potentially anomalous topics based on the core evaluation feature parameters of each topic. The misjudgment correction module calculates the domain keyword coverage of suspected topics to prevent misjudging topics related to "researching core issues" as anomalous. The system anomaly determination module counts the final number of anomalous topics to determine whether the target AI topic selection system is abnormal. The adaptive optimization module automatically optimizes the threshold parameters of each module based on historical data. Through the collaborative work of these modules, a closed-loop process of AI topic sampling, retrieval, feature analysis, anomaly determination, and adaptive optimization is completed, achieving fully automated evaluation of the accuracy of AI topic selection and significantly improving the evaluation precision of AI topic selection capabilities.

[0054] Understandably, in the field of topic recommendation, recommendations are typically made based on the technical fields and keywords of the desired topics provided by users. This involves searching massive literature databases for similar literature and then reordering the resulting paragraphs by recombining and ranking keywords from similar literature in the database. This invention, through precise analysis of topic selection, breaks away from the traditional paradigm of keyword matching and simple reordering. It constructs a deeply integrated evaluation system that combines semantic understanding, trend perception, and dynamic assessment. This allows the invention to effectively identify seemingly reasonable but actually cobbled-together "pseudo-topics," significantly improving the accuracy of evaluating AI's topic selection capabilities.

[0055] Specifically, the trend analysis module determines the research trend cycle of the field based on the difference between the maximum value of the year corresponding to the peak number of publications of each historical document in the comparative literature set and the year corresponding to the peak number of retrievals of comparative literature in the past years and the current year.

[0056] If the difference is less than or equal to a preset threshold, then the domain research trend cycle is determined to be in the domain research active cycle;

[0057] If the difference is greater than a preset threshold, then the domain research trend cycle is determined to be in a domain research decline cycle.

[0058] In this embodiment, the peak year for the number of publications of all comparative literatures corresponding to each topic and the peak year for the number of searches of comparative literatures are extracted. The year with the largest peak is selected to calculate the difference between the largest year and the current year. If the difference is less than or equal to a preset threshold, the research trend cycle of the field is determined to be in the active research cycle of the field. If the difference is greater than the preset threshold, the research trend cycle of the field is determined to be in the decline cycle of the field.

[0059] In this embodiment, the initial threshold for the difference can be set to 3 years to align with the typical cycle of academic research. Research, from topic selection and project initiation to publication and the formation of a field hotspot, usually requires a 2-3 year cycle. 3 years covers the complete stage of hotspot formation and dissemination. The difference threshold is the tuning target of the adaptive optimization module. After each dynamic adjustment, the current adjusted value is used as the preset threshold in each evaluation.

[0060] Specifically, the feature analysis module determines the number of highly matched comparative documents for each topic based on the number of comparative documents whose similarity to a number of comparative documents is greater than a similarity threshold.

[0061] Specifically, the feature analysis module determines the similarity threshold based on the research trend cycle of the corresponding field for each topic;

[0062] If the research trend cycle of the field is in the active research cycle of the field, then the similarity threshold is determined as the first similarity threshold;

[0063] If the research trend cycle of the field is in the decline cycle of the field research, then the similarity threshold is determined as the second similarity threshold.

[0064] In this embodiment, the similarity between several comparative documents and the topic is extracted from the comparative literature set of each topic. The similarity is calculated by the semantic retrieval model in the process of retrieving comparative documents through the core keywords of the topic.

[0065] It can be understood that when the research in the field is in an active cycle, the similarity threshold is set to the first similarity threshold (preferably, the initial value of the first similarity threshold can be 0.75), which is relatively loose. Because there are many new papers during the active cycle, a higher similarity is normal. When the research in the field is in a recession cycle, the similarity threshold is set to the second similarity threshold (preferably, the initial value of the second similarity threshold can be 0.85), which is relatively strict. Because the research field is relatively old during the recession cycle, a high similarity is more likely to mean duplication, and the degree of specialization in the research field is relatively high during the recession cycle, so a higher similarity is selected. Through the adaptive threshold of the field trend cycle, in the active period, the similarity threshold is lenient to avoid killing incremental innovations; in the recession period, the similarity threshold is strict to promote the topic selection system to propose truly novel topics. Both the first similarity threshold and the second similarity threshold are objects for tuning the adaptive optimization module.

[0066] Specifically, the feature analysis module calculates the internal similarity of the titles of each topic selection to determine the semantic coherence of each topic selection.

[0067] Among them, the internal similarity of the title is determined by calculating the cosine similarity of the titles of each topic selection after splitting.

[0068] The titles of each topic selection after splitting include two texts generated by splitting the title based on a preset stop word.

[0069] In this embodiment, the topic selection title is split according to a preset stop word (preferably, "de" can be used) to generate two texts. If the topic selection title does not contain the preset stop word, it is split into two texts by half according to the number of characters. Call the word embedding vector model (preferably, "Word2Vec" can be used), convert the two texts into vectors, and calculate the cosine similarity of the two vectors as the semantic coherence of the topic selection. The semantic coherence is used to check whether the front and back parts of the title are talking about related things. If the title of the topic selection is a patched title, the semantic coherence of its front and back parts will be relatively low.

[0070] Specifically, the topic selection anomaly recognition module is used to determine that the topic selection is suspected of being abnormal when the number of high-matching comparison documents corresponding to the topic selection is greater than the high-matching comparison document quantity threshold and the corresponding semantic coherence is less than the semantic coherence threshold.

[0071] Specifically, the topic selection anomaly recognition module determines the high-matching comparison document quantity threshold and the semantic coherence threshold corresponding to the topic selection according to the field research trend cycle corresponding to each topic selection.

[0072] In this embodiment, if the domain research trend cycle is in an active domain research cycle, then the threshold for the number of highly matched comparative documents is determined as the first threshold for the number of highly matched comparative documents, and the threshold for semantic coherence is determined as the first threshold for semantic coherence; if the domain research trend cycle is in a declining domain research cycle, then the threshold for the number of highly matched comparative documents is determined as the second threshold for the number of highly matched comparative documents, and the threshold for semantic coherence is determined as the second threshold for semantic coherence.

[0073] Understandably, during periods of active research in a field, setting the threshold for the number of highly matched comparative documents to the first threshold for the number of highly matched comparative documents (preferably, the initial value of the first threshold for the number of highly matched comparative documents can be 5) is relatively lenient. This is because there are many new papers during an active period, and the research directions of selected topics are often highly related to many recently published papers, allowing for a large number of highly similar documents. The semantic coherence threshold is set to the first semantic coherence threshold (preferably, the initial value of the first semantic coherence threshold can be 0.6). This is because during an active period, the requirement for logical rigor is slightly lower, and emerging active research fields are evolving rapidly, so the wording of titles may be more exploratory and diverse. Both the first threshold for the number of highly matched comparative documents and the first threshold for semantic coherence are used as tuning targets for the adaptive optimization module.

[0074] When a research field is in a decline phase, the threshold for the number of highly matched comparative documents is set as the second highest matching comparative document threshold (preferably, the initial value of the second highest matching comparative document threshold can be 3), which is relatively strict. This is because during a decline phase, the number of published papers in the research field has already decreased significantly, and the research is very mature and stable. If the chosen topic is highly similar to multiple papers in an outdated field, it is highly likely to be repetitive work. The semantic coherence threshold is set as the second semantic coherence threshold (preferably, the initial value of the second semantic coherence threshold can be 0.7), because during a decline phase, the theoretical framework, methodology, and terminology of a mature field are very clear and stable. Both the second highest matching comparative document threshold and the second semantic coherence threshold are used as tuning targets for the adaptive optimization module.

[0075] In this embodiment, when the number of highly matched comparative documents corresponding to the topic exceeds a threshold for the number of highly matched comparative documents and the corresponding semantic coherence is less than a threshold for semantic coherence, the topic is determined to be potentially abnormal. All topics determined to be potentially abnormal are extracted from the topic set to be tested, generating a potentially abnormal topic set.

[0076] Specifically, the misjudgment correction module calculates the proportion of the core keywords of the selected topic to the core keywords of the selected topic, and determines it as the coverage rate of the selected topic.

[0077] Specifically, the misjudgment correction module corrects the suspected abnormal topic set based on the coverage rate, and after removing topics in the suspected abnormal topic set whose coverage rate is greater than a preset coverage rate threshold, an abnormal topic set is generated.

[0078] In this embodiment, a core keyword library of the relevant field is loaded, the number of core keywords belonging to the core field among the selected topic's core keywords is counted, and the ratio of this number to the total number of core keywords for the selected topic is calculated to determine the coverage rate of the selected topic. It is understood that if all the keywords of a selected topic are the most core terms in the field (i.e., high coverage), then even if it is similar to many other documents, it indicates that it is researching fundamental issues in the field, rather than simply piecing together existing information. This protects excellent selected topics that delve deeply into core areas.

[0079] During an active research cycle in the domain, the coverage threshold is set to a first coverage threshold (preferably, the initial value of the first coverage threshold can be 0.8). Because the domain's core terms are generalized during an active cycle (including traditional core terms and emerging derived core terms), a higher coverage threshold is set to strengthen the "core focus" standard. During a declining research cycle in the domain, the coverage threshold is set to a second coverage threshold (preferably, the initial value of the second coverage threshold can be 0.6). During a declining cycle, the domain's core terms are stable, and most topics are focused on detailed optimization within traditional areas; therefore, a lower coverage threshold is set to protect normal topic selection. The coverage threshold is the tuning target of the adaptive optimization module.

[0080] If the coverage rate is greater than a preset coverage rate threshold, the topic is removed from the suspected abnormal topic set; after removing all topics with coverage rates greater than the threshold from the suspected abnormal topic set, an abnormal topic set is generated.

[0081] Specifically, the system anomaly determination module is used to determine that the target AI topic selection system is abnormal based on the comparison result that the number of topics in the abnormal topic selection set is greater than a preset threshold.

[0082] In this embodiment, the number of topics in the abnormal topic selection set is counted. If the number of topics exceeds a preset threshold, the target AI topic selection system is deemed abnormal. The initial value of the threshold is set to 0.15 times (rounded up) of the topic selection set to be tested, and the threshold is the tuning target of the adaptive optimization module. Statistically, if more than 15% of the topics are problematic, the target AI topic selection system may be stuck in a simple keyword recombination and paragraph rearrangement pattern, lacking true semantic understanding and innovative value judgment capabilities (e.g., the prompts provided to the target AI topic selection system may fail to emphasize key requirements such as "innovation," "cutting-edge," or "feasibility," whether the model training data is biased towards outdated content, or whether the post-processing logic overemphasizes keyword matching while ignoring semantic coherence). Manual intervention for inspection and adjustment is required.

[0083] Specifically, the adaptive optimization module calls an adaptive learning model to automatically adjust all thresholds in the AI ​​topic selection ability assessment system based on the results of historical judgments.

[0084] In this embodiment, the AI ​​topic selection ability assessment system collects all relevant historical data for topic assessments and the corresponding user scores for each topic monthly. With the ultimate goal of maximizing user scores, a quantifiable objective function F(X) is constructed, where X is a parameter vector containing all adjustable thresholds (optimization targets). Using hyperparameter search algorithms such as Bayesian optimization, the optimal parameter configuration X_best that maximizes the objective function F(X) is searched within ±10% of the current values ​​of each threshold. The newly found optimal threshold parameter X_best is automatically updated to each module of the assessment system and applied to the next assessment cycle. Any form of the existing technology can be used for the hyperparameter search algorithm, which will not be elaborated here.

[0085] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. An AI topic selection ability evaluation system based on adaptive evaluation, characterized in that, The method comprises the following steps: a topic sampling module is used to randomly sample a preset number of topics in a target AI topic system within a preset evaluation period to form a set of topics to be evaluated, wherein the topics include the titles and core keywords of the topics; a literature retrieval module is used to call a semantic retrieval model to retrieve the core keywords of each topic in the set of topics to be evaluated to determine a set of comparison literatures to form a set of comparison literatures, wherein the set of comparison literatures includes the number of comparison literatures, the years of the comparison literatures, the number of retrievals of the comparison literatures, and the similarity between the comparison literatures and the corresponding topics; a trend analysis module is used to extract a field research trend characteristic parameter based on the set of comparison literatures to determine a field research trend period, wherein the field research trend characteristic parameter includes the number of comparison literatures published each year and the number of retrievals of comparison literatures each year; a feature analysis module is used to analyze the evaluation characteristic parameters of each topic based on each topic, the set of comparison literatures corresponding to each topic, and the field research trend period, wherein the evaluation characteristic parameters include the number of high-matching comparison literatures corresponding to each topic and the semantic coherence of each topic; a topic anomaly identification module is used to determine whether each topic in the set of topics is suspected to be abnormal based on the evaluation characteristic parameters of each topic and the field research trend period, and to generate a set of suspected abnormal topics; a misjudgment correction module is used to calculate the coverage rate of the keywords related to the field based on each topic in the set of suspected abnormal topics, to correct the set of suspected abnormal topics based on the coverage rate, and to form a set of abnormal topics; a system anomaly determination module is used to determine whether the target AI topic system is abnormal based on the number of topics in the set of abnormal topics; an adaptive optimization module is used to call an adaptive learning model to dynamically adjust the determination threshold in the AI topic capability evaluation system according to the results of historical determinations. 2.The AI topic selection ability evaluation system based on adaptive evaluation according to claim 1, wherein, The trend analysis module determines the field research trend period based on the maximum value of the difference between the year corresponding to the peak value of the number of historical literatures published each year and the year corresponding to the peak value of the number of retrievals of comparison literatures each year and the current year in the set of comparison literatures; if the difference is less than or equal to a preset threshold, it is determined that the field research trend period is in a field research active period; if the difference is greater than the preset threshold, it is determined that the field research trend period is in a field research recession period. 3.The AI adaptive evaluation-based AI topic selection ability evaluation system according to claim 1, characterized in that, The feature analysis module determines the number of high-matching comparison literatures corresponding to each topic according to the number of comparison literatures with a similarity greater than a similarity threshold between each topic and the comparison literatures. 4.The AI topic selection ability evaluation system based on adaptive evaluation according to claim 3, wherein, The feature analysis module determines the similarity threshold according to the field research trend period corresponding to each topic; if the field research trend period is in a field research active period, the similarity threshold is determined to be a first similarity threshold; if the field research trend period is in a field research recession period, the similarity threshold is determined to be a second similarity threshold. 5.The AI adaptive evaluation-based AI topic selection ability evaluation system according to claim 1, wherein, The feature analysis module calculates the semantic coherence of each topic by calculating the internal similarity of the title of each topic; wherein the internal similarity of the title of each topic is determined by calculating the cosine similarity of the title of each topic after splitting; the title of each topic after splitting includes two texts generated by splitting the title based on a preset stop word. 6.The AI topic selection ability evaluation system based on adaptive evaluation according to claim 5, wherein, The selected topic abnormality identification module is configured to determine that the selected topic is suspected to be abnormal when the number of high-matching comparison documents corresponding to the selected topic is greater than a high-matching comparison document number threshold and the semantic coherence corresponding to the selected topic is less than a semantic coherence threshold. 7.The AI topic selection ability evaluation system based on adaptive evaluation according to claim 6, wherein, The selected topic abnormality identification module determines the high-matching comparison document number threshold and the semantic coherence threshold corresponding to each selected topic according to a domain research trend cycle of the selected topic. 8.The AI topic selection ability evaluation system based on adaptive evaluation according to claim 1, wherein, The misjudgment correction module calculates a proportion of domain core keywords in core keywords of the selected topic based on the core keywords of the selected topic, and determines a coverage rate of the selected topic. 9.The AI topic selection ability evaluation system based on adaptive evaluation according to claim 8, wherein, The misjudgment correction module corrects the suspected abnormal selected topic set based on the coverage rate, and generates an abnormal selected topic set after eliminating the selected topic corresponding to the coverage rate greater than a preset coverage rate threshold from the suspected abnormal selected topic set. 10.The AI topic selection ability evaluation system based on adaptive evaluation according to claim 9, wherein, The system abnormality determination module determines that the target AI selected topic system is abnormal based on a comparison result that the number of selected topics in the abnormal selected topic set is greater than a preset number threshold.

Images