Atomic problem generation and selection method and system based on mutual information distillation and computer medium

Abstract

The invention discloses an atomic problem generation and selection method and system based on mutual information distillation, and a computer medium, and belongs to the technical field of information retrieval and natural language processing.The scheme comprises the steps that a mutual information discriminator is trained based on comparative learning to quantify the information association strength of problems and content blocks; generating diversified candidate questions by using a large language model; selecting an overall optimal subset from the candidate problems through a sub-modulus optimization selection algorithm on the basis of a mutual information estimation value and considering the problem coverage degree; and finally, by adopting a negative alignment filtering mechanism, through vector database retrieval and large language model discrimination, actively identifying a generalization problem lacking a determiner, and performing specificity enhancement, so that cross-content block retrieval confusion caused by fuzzy anaphora is avoided. According to the method, the high-quality and high-specificity atomic problem index is constructed for a retrieval enhancement generation (RAG) system, the accuracy of the retrieval system is remarkably improved, false detection is reduced, and the user experience is improved.

Description

Technical Field

[0001] This invention relates to the fields of information retrieval and natural language processing, specifically to a method, system, and computer medium for generating and selecting atomic problems based on mutual information distillation. Background Technology

[0002] Natural language search is an emerging search method that allows users to search using natural language without having to enter specific keywords or grammar. Unlike traditional search engines, natural language search can understand the user's intent and return more relevant search results.

[0003] First, various data related to the question-answering system need to be collected. This data can come from multiple sources such as documents, web pages, and databases, thereby generating atomic questions (i.e., the smallest indivisible semantic question unit). Then, the collected data is cleaned to remove noise, duplicates, and irrelevant information, ensuring data quality and accuracy. The cleaned data is then used to build a knowledge base.

[0004] Traditional methods for generating atomic problems inherently suffer from a core deficiency: the lack of theoretical guidance. Existing methods primarily rely on heuristic rules or simple templates for problem generation. This results in an empirical and superficial generation process, failing to incorporate information theory (such as mutual information) as a quantifiable objective. Consequently, it cannot scientifically assess the strength of the intrinsic connection between the generated problem and its corresponding content block. This leads to the following shortcomings: 1. Severe Question Overlap: Highly similar questions are generated for different but similar content blocks (document fragments). This leads to result confusion and decreased accuracy during retrieval. Users receive a large number of duplicate or similar search results, rather than precise answers.

[0005] Second, insufficient question differentiation: The generated questions lack uniqueness and specificity. This makes it impossible to accurately locate specific content blocks, like a key opening multiple locks, causing the retrieval system to lose its accuracy.

[0006] Third, a lack of a global optimization perspective: Some systems use semantic similarity or keyword matching for filtering, but this is a local greedy selection strategy. Focusing only on the quality of individual questions cannot guarantee the overall quality, diversity, and non-redundancy of the final filtered set of questions.

[0007] Fourth, neglecting cross-document conflicts: Existing methods only focus on the relationship between the question and the current content, without considering whether the question might also highly match the content of other documents. This poses a risk of cross-content block confusion, where a single question might incorrectly recall multiple unrelated pieces of content, leading to retrieval conflicts.

[0008] V. Generalization problem is common: In the process of question generation, existing systems generally fail to effectively identify and handle the generalization problem (Generic Question).

[0009] Definition of generalization problem: The lack of a clear subject or object, or the use of vague pronouns (such as "it," "should," "this"), leading to ambiguous semantic referents. Typical manifestations: The questions "What types are there?" and "How is the detection performed?" are missing subjects. Vague references: "What are the applications of this technology?" or "What are its characteristics?" Case Study: In legal text retrieval scenarios, a generalization problem may arise: Article A is generated: "What is the scope of application of this regulation?" (The reference to "this" is unclear) Article B generated: "What are the penalties?" (Subject missing) This type of generalization problem can lead to matching multiple entries during retrieval, severely impacting retrieval accuracy.

[0010] Therefore, this invention came into being.

[0011] The information disclosed in this background section is intended only to enhance the understanding of the overall background of the invention and should not be construed as an admission or in any way implying that the information constitutes prior art known to those skilled in the art. Summary of the Invention

[0012] Existing atomic problem generation methods, lacking information theory guidance and global optimization mechanisms, generally suffer from high problem overlap, low discriminative power, and generalization problems. This leads to insufficient retrieval accuracy and severe result confusion when applied to retrieval enhancement generation systems, significantly impacting user experience and system efficiency. This invention aims to overcome these shortcomings by providing a method for generating highly specific, low-redundancy atomic problem sets, ultimately improving the accuracy and usability of retrieval systems.

[0013] To achieve the above objectives, a method for generating and selecting atomic problems based on mutual information distillation is provided, characterized by the following steps: Training preparation: Construct a training dataset containing positive and negative sample pairs, and train a mutual information discriminator based on the dataset. The mutual information discriminator is used to calculate the estimated mutual information between the question text and the content block. Candidate question generation: Utilize a large language model to generate multiple candidate atomic questions for at least one content block; Mutual information evaluation: Using the mutual information discriminator, calculate the estimated mutual information between each candidate atomic problem and its corresponding content block; Optimization selection: Based on the mutual information estimate, an optimal subset of problems is selected from the multiple candidate atomic problems through a submodular optimization function. The submodular optimization function is used to maximize the coverage of the selected problem subset to the candidate problem set and ensure the quality of the problems. Negative alignment filtering: For each atomic problem in the optimal problem subset, the similarity of the stored problems is retrieved using a vector database, and a large language model is used to determine whether it is a generalized problem. The generalized problem refers to a problem that lacks a clear subject or object or uses vague pronouns, resulting in unclear semantics. Problems determined to be generalized are specifically enhanced to supplement clear qualifiers, avoiding cross-content block retrieval confusion caused by vague pronouns, so as to obtain the final set of atomic problems. Index building: Establish a mapping index between the final atomic problem set and the corresponding content blocks.

[0014] Preferably, in the training preparation step, the mutual information discriminator adopts a dual-tower neural network architecture, including a first encoder for encoding the question text and a second encoder for encoding the content blocks, and the mutual information estimate is calculated based on the similarity between the encoded vectors output by the first encoder and the second encoder.

[0015] Preferably, in the training dataset, positive samples consist of a question text and its corresponding correct content block, while negative samples consist of the question text and a randomly selected irrelevant content block.

[0016] Preferably, the submodular optimization function in the optimization selection step adopts the form of a facility location function, expressed as: F(S) = Σ_{i∈V} max_{j∈S} [Quality(q_j) · Sim(q_i, q_j)] Where V represents the set of all candidate questions in the current content block, S represents the selected subset of questions, Quality(q_j) = MI(q_j, c) · Length_penalty(q_j) is the question quality score, MI(q_j, c) is the mutual information estimate, Length_penalty(q_j) is the length penalty factor calculated based on the question text length, and Sim(q_i, q_j) is the cosine similarity between questions. This function is solved by a greedy algorithm, selecting the question with the largest marginal gain in each round and adding it to set S, with a theoretical approximation ratio guarantee of (1-1 / e)≈0.632.

[0017] Preferably, the sub-modular optimization is performed on a set of candidate issues for a single content block. The similarity matrix between candidate issues is pre-computed to efficiently execute the greedy algorithm. The selected k issues then enter the negative alignment filtering process for generalization detection and enhancement.

[0018] Preferably, the negative alignment filtering step adopts a two-stage mechanism: the first stage uses a vector database to retrieve the cosine similarity between the current question and the stored questions, and sets a similarity threshold to filter suspected generalized questions; the second stage uses a large language model to determine whether the question lacks a clear subject or object or uses vague pronouns, and for questions determined to be generalized, specificity enhancement is performed by supplementing qualifying words based on content block keywords using LLM.

[0019] Preferably, in the candidate question generation step, the large language model is guided to generate candidate atomic questions of at least two different types, including factual, inferential, applicability, and conditional.

[0020] This invention also provides an atomic problem generation and selection system based on mutual information distillation, comprising: The mutual information discriminator training module is configured to train a mutual information discriminator based on the training dataset of positive and negative sample pairs. The candidate question generation module is configured to call the large language model to generate multiple candidate atomic questions for content blocks. The mutual information evaluation module is configured to call the mutual information discriminator to calculate the estimated mutual information between the candidate atomic problem and the corresponding content block; The sub-modulus optimization selection module is configured to select the optimal subset of problems from candidate atomic problems based on the sub-modulus optimization function. The negative alignment filtering module is configured to identify generalization problems and perform specificity enhancements; and The index storage module is configured to build and store the mapping index between atomic problems and content blocks.

[0021] Preferably, the negative alignment filtering module includes: The generalization problem identification unit uses a vector database to retrieve the cosine similarity between the current problem and the stored problems to identify generalization problems. When the similarity exceeds a preset threshold, a large language model is used to determine whether the problem is a generalization problem. The generalization problem is defined as a problem that lacks a clear subject or object or uses vague pronouns, resulting in unclear semantic direction. The specificity enhancement unit, for questions determined to be generalized, uses a large language model to supplement qualifiers based on keywords and context in the content block, generating specific questions containing clearly defined objects.

[0022] The present invention also provides a computer-readable storage medium storing a computer program that, when executed, implements the above-described method for generating and selecting atomic problems based on mutual information distillation.

[0023] The above technical solution achieves at least one of the following beneficial effects: Improved search accuracy: Quality assurance is achieved through three collaborative steps: mutual information evaluation, optimal selection, and negative alignment filtering. Mutual information estimation ensures the answerability of the question, submodular optimization function guarantees the discriminability of the question through similarity penalty mechanism, and negative alignment filtering improves the clarity of the question through generalized question identification and specificity enhancement.

[0024] Mutual information-guided optimization: By quantifying the degree of information interdependence between questions and content, the specificity of generated questions and the accuracy of retrieval are improved; Negative alignment conflict elimination: Proactively identifies and eliminates problem overlaps across content blocks, reducing false detections; Global optimization selection: Submodular function optimization improves the overall quality of the problem set and avoids the limitations of local greedy selection.

[0025] Improved system efficiency and user experience: Search response optimization: High-quality questions reduce search ambiguity and improve query response performance; Improved user experience: Users can find matching content more accurately, enhancing the user experience; Reduced maintenance costs: Automated problem optimization reduces the workload of manual tuning and improves system maintenance efficiency.

[0026] The technology is highly scalable. Domain adaptability: Information theory methods are universal and can be adapted to different professional fields such as law, medicine, and technology; Scalability: The algorithm complexity is controllable, supporting efficient processing of large-scale document libraries; Modular design: Each component is relatively independent, which facilitates system integration and functional expansion. Attached Figure Description

[0027] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the system architecture of the present invention.

[0028] Figure 2 This is a flowchart of the steps of the present invention.

[0029] Figure 3 This is a schematic diagram of the dual-tower model of the dual-tower neural network architecture in this invention. Detailed Implementation

[0030] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0031] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0032] Reference Figures 1 to 3 As shown, the method for generating and selecting atomic problems based on mutual information distillation includes the following steps: S01 Training Preparation: Mutual Information Discriminator Training.

[0033] The core objective of this stage is to train a discriminator model capable of accurately estimating the mutual information between the question text and the content block. Specifically, a training dataset containing positive and negative sample pairs is constructed, and a mutual information discriminator is trained based on this dataset. This discriminator is used to calculate the estimated mutual information between the question text and the content block.

[0034] Please combine Figure 3 The mutual information discriminator adopts a dual-tower neural network architecture, including a first encoder for encoding the question text and a second encoder for encoding the content blocks. The mutual information estimate is calculated based on the encoding vectors output by the first encoder and the second encoder.

[0035] Encoder architecture: Both encoders are built based on pre-trained language models.

[0036] Pre-trained model selection: This invention uses a Transformer-based bidirectional encoder model (such as BERT, RoBERTa, etc.) instead of an autoregressive generative model (such as GPT, Llama, etc.). The reasons for this selection are as follows: (1) Task adaptability: Bidirectional encoders generate semantic representation vectors of text by simultaneously paying attention to the context, which is naturally suitable for semantic understanding and similarity calculation tasks; while autoregressive models are mainly designed for text generation, and their unidirectional attention mechanism is not conducive to capturing complete bidirectional semantic information.

[0037] (2) Computational efficiency: The bidirectional encoder directly outputs a fixed-dimensional vector representation without generating it token by token; the autoregressive model requires sequential decoding, which has a large computational overhead and is not suitable for vector encoding scenarios.

[0038] (3) Training cost: Bidirectional encoders that have been pre-trained on large-scale corpora can be directly reused, significantly reducing the training data requirements (usually 500-1000 sample pairs are sufficient for fine-tuning) and computational costs.

[0039] Each encoder consists of three components: (1) Pre-trained encoding layer: Reuse the Transformer layer (all or part) of the pre-trained model to convert the text into a deep semantic representation.

[0040] (2) Adaptation layer: The output of the pre-trained model is mapped to a unified low-dimensional vector space (typically 128-512 dimensions) through linear transformation, reducing computation and storage overhead.

[0041] (3) Aggregation layer: Aggregates multiple word vectors of the sequence into a single vector, which can be output using special tags (such as [CLS]) or average pooling.

[0042] The two encoders use the same architecture but have independent parameters, and output the question vector q_vec and the content vector c_vec respectively.

[0043] Mutual information estimation layer: Based on the Mutual Information Neural Estimation (MIS) theory, a discriminant network is used to estimate the mutual information between the question and the content. The specific implementation includes: first, calculating the interaction features of the two vectors (element-wise product of the vectors and cosine similarity); then, concatenating the question vector, content vector, and interaction features to form a fused representation; finally, performing a nonlinear transformation through a multi-layer fully connected network to output the mutual information estimate MI(q_j, c), which is mapped to the [0,1] interval to represent the association strength, where q represents the question text and c represents the content block.

[0044] Training data construction: Positive sample pairs: These are questions and their corresponding correct content blocks, representing high mutual information. Construction methods include generation based on rule templates, generation using large language models, or extraction from existing knowledge bases (such as legal document libraries or technical document libraries). Quality verification is required to ensure a strict correspondence between questions and content blocks and to avoid noise. It is recommended to prepare 500-1000 positive sample pairs for each domain.

[0045] Negative sample pairs: Questions and irrelevant content blocks, representing low mutual information. Construction methods include random sampling from an existing knowledge base, generating questions with mismatched content using a large language model, and intra-batch cross-pairing (correct content from other questions in the same batch). Using diverse negative samples ensures the model learns fine-grained discriminative ability. The ratio of negative samples is controlled between 1:1 and 1:3 to balance the model's discriminative power. For example, if there are 500 positive samples, then there should be 500-1500 negative samples.

[0046] End-to-end training: End-to-end training is performed using a positive-negative sample comparison approach. The input consists of question-content pairs and matching labels (positive sample labels are 1, negative sample labels are 0). The question and content are each encoded into vectors by an encoder, and the matching score is output through a mutual information estimation layer. After calculating the loss, all component parameters are updated simultaneously through backpropagation.

[0047] Using the InfoNCE loss function:

[0048] Where f(q,c) is the mutual information estimate MI(q_j, c), and c + For positive samples (related to the question), cᵢ - The negative samples are irrelevant to the problem, and τ is a temperature parameter (0.05-0.2, default 0.1) used to control the smoothness of the distribution. This loss enhances the scores of positive samples significantly higher than those of negative samples.

[0049] Training strategy: A hierarchical learning rate is adopted, with a smaller learning rate set for the pre-trained encoding layer to retain pre-trained knowledge, and a larger learning rate set for the newly added adaptation layer and mutual information estimation layer to accelerate convergence.

[0050] Evaluation metrics: Classification accuracy for positive and negative samples >85%, mutual information estimate discrimination (mean for positive samples >0.7, mean for negative samples <0.3).

[0051] After training, the mutual information discriminator can effectively calculate the mutual information estimate between any question and content block, providing a quantitative basis for subsequent candidate question quality assessment.

[0052] Exemplary implementation: Legal provisions retrieval scenarios: Data Construction: Positive samples are extracted from legal knowledge bases (such as national legal databases). For example, the question "What is the scope of application of the 'Regulations on the Administration of Construction Safety'?" is paired with corresponding articles to form positive samples. Negative samples are generated by randomly pairing articles (e.g., pairing the above question with unrelated content from the "Tax Collection and Administration Law"). The training data size is approximately 500-1000 positive sample pairs, ensuring coverage of subdomains such as civil and criminal law. Model Training: BERT-base is used as the encoder, with an output vector dimension of 256. During training, the target MI value for positive samples is >0.7, and for negative samples it is <0.3, to distinguish the specificity of the articles.

[0053] API technical documentation scenarios: Data Construction: Positive samples are generated based on open-source API documentation (such as the Swagger specification), for example, pairing the question "What are the required parameters for the OAuth2.0 token interface?" with the corresponding interface description; negative samples are generated by cross-pairing different interface content. The data size is approximately 800 positive sample pairs, covering authentication, data transmission, and other types. Model Training: The encoder uses the RoBERTa model, with a 128-dimensional output dimension for the adaptation layer to optimize the encoding efficiency of technical terms; the training temperature parameter τ=0.1 enhances sensitivity to parameter details.

[0054] Medical Clinic Knowledge Base Scenario: Data Construction: Positive samples are extracted from clinical guidelines (such as WHO guidelines), for example, matching the question "What are the diagnostic criteria for type 2 diabetes?" with corresponding guideline content; negative samples are generated by mixing content from different diseases. The data scale is approximately 1000 positive sample pairs, covering common diseases. Model Training: A domain-adaptive pre-trained model (such as BioBERT) is used, with an output dimension of 512 dimensions to capture fine-grained medical terminology; medical entity recognition features are added during training to improve the accuracy of MI estimation.

[0055] S02: Candidate question generation.

[0056] Specifically, a large language model is used to generate multiple candidate atomic problems for at least one content block.

[0057] The large language model here can be a commercial large language model such as ChatGPT, Claude, or GLM, or a locally deployed open-source large language model such as Llama 2, DeepSeek, or Qwen.

[0058] Prompt design strategy: The large language model is guided to generate diverse candidate atomic questions using prompt words. The prompt words contain the following elements: (1) Input the content block text; (2) Generation requirements: The guided model generation includes at least two different types of questions, including factual, inference, application and conditional questions. Each question is independent and complete and can be answered explicitly from the content block. (3) Quality constraints: The problem statement should be clear and concise, and vague pronouns (such as "it", "that", "this") should be avoided to ensure the specificity of the problem.

[0059] For specific domains, the prompts can be supplemented with domain-specific question types to improve the domain-appropriateness of the generated questions.

[0060] Using the above method, multiple candidate atomic problems are generated for each content block, which then proceed to the next step of mutual information evaluation.

[0061] Prompt design example: In a legal provision retrieval scenario, a prompt word example would be: "Please generate multiple questions of different types based on the following provision content. Questions must be independent and complete, avoid using vague pronouns such as 'it' or 'the,' and ensure that a clear answer can be found in the content. Content: [Provision text]."

[0062] In API technical documentation scenarios, a prompt might look like this: "Based on the following API documentation, please generate several technical questions from different perspectives. The questions should be clear and specific, avoiding vague references, and ensure they can be answered directly from the documentation. Documentation: [API documentation text]"

[0063] In a medical diagnostic knowledge base scenario, a prompt example would be: "Based on the following medical content, please generate multiple questions from different dimensions. The questions must be clear and specific, avoid using pronouns, and ensure that the answers can be obtained from the content. Content: [Medical text]".

[0064] The common feature of the above examples is that the prompts only provide generation requirements and quality constraints. The large language model automatically determines domain features, extracts key elements, and generates diverse questions based on the input content, rather than pre-setting fixed question templates or types.

[0065] S03: Mutual information assessment.

[0066] Specifically, the mutual information discriminator is used to calculate the mutual information estimate between each candidate atomic problem and its corresponding content block.

[0067] For each candidate question q and its corresponding content block c, the mutual information estimate MI(q_j, c) ∈ [0, 1] is obtained by evaluating the mutual information discriminator trained in S01. This value reflects the strength of the information association between the question and the content block.

[0068] To facilitate subsequent sub-module optimization selection, the mutual information estimate is combined with the length penalty factor to calculate the quality score for each candidate problem: Quality(q) = MI(q_j, c) × Length_penalty(q).

[0069] Here, Length_penalty(q) is the length penalty factor, calculated based on the length of the question text. The length penalty mechanism is based on the following principles: concise questions are easier to understand and match, therefore they are given higher weight; excessively long questions may contain redundant information, thus incurring a certain penalty. In specific implementations, piecewise functions, inverse proportional functions, or minimum value functions can be used, and adjustments can be made according to the application scenario and the distribution of question lengths; these are not considered as limitations here.

[0070] The penalty factor is dynamically calculated using an inverse proportional relationship, for example: Length_penalty(q) = 1 / (1 + α · (len(q) - L_target)).

[0071] Parameter description: L_target: target length (e.g., 10 words), in which case the penalty factor is 1; α: penalty intensity coefficient (e.g., 0.05), which controls the penalty magnitude when the length deviates.

[0072] Application examples of mutual information evaluation: In the scenario of legal provision retrieval, for the candidate question "What is the scope of application of the Regulations on Construction Safety Management?" and the corresponding provision content, the mutual information discriminator encodes the question and content as vector representations, calculates the semantic association strength between the two, and outputs a mutual information estimate. A high MI value indicates a strong correlation between the question and the provision content, while a low MI value indicates a weak correlation. A quality score is calculated based on the question length to select questions that are both highly relevant and concisely worded.

[0073] In API technical documentation scenarios, for the candidate question "What are the required parameters of the OAuth2.0 token interface?" and the corresponding interface documentation, the mutual information discriminator evaluates whether the question accurately points to the parameter information of that interface. If the question is too general (such as "What are the parameters?"), the MI value is low; if the question is highly specific and the content matches, the MI value is high. The best candidate questions are retained for further optimization based on the quality score ranking.

[0074] In the context of a medical diagnostic knowledge base, mutual information assessment ensures that the candidate question "What are the diagnostic criteria for type 2 diabetes?" and the corresponding treatment guidelines accurately match the diagnostic information for the disease. If the candidate question set contains multiple similar questions, the mutual information score can quantify the degree of matching between each question and the content, providing a quality basis for sub-model optimization.

[0075] The mutual information evaluation process is the same for the three scenarios mentioned above: input the problem and content into the discriminator trained by S01, calculate the MI value through the neural network, and obtain the quality score by combining the length penalty, so as to provide a quantitative basis for subsequent optimization selection.

[0076] S04: Optimization Selection.

[0077] Specifically, based on the mutual information estimate, an optimal subset of problems is selected from the plurality of candidate atomic problems through a submodular optimization function. The submodular optimization function is used to maximize the coverage of the selected problem subset to the candidate problem set and ensure problem quality.

[0078] The submodal objective function design adopts the facility location submodal function form: F(S) = Σ_{i∈V} max_{j∈S} [Quality(q_j) · Sim(q_i, q_j)] Where V represents the set of all candidate questions in the current content block, S represents the selected subset of questions, Quality(q_j) is the question quality score (calculated from the mutual information estimate and the length penalty factor), and Sim(q_i, q_j) is the semantic similarity between questions, which can be obtained by encoding the question text into a vector representation and then calculating the cosine similarity. The encoding methods include question encoders that reuse the mutual information discriminator, using independent text embedding models, or other semantic encoding methods.

[0079] The semantics of this function are: each candidate problem is represented by the problem most similar to it in the selected set, and the representation value is equal to the product of the quality of the representative and the similarity. The goal is to select a high-quality and diverse subset that can cover all candidate problems.

[0080] This function has the submodular property and satisfies the diminishing marginal returns characteristic, so a greedy algorithm can obtain a theoretical approximation guarantee.

[0081] Greedy algorithm solution A greedy strategy is used to iteratively select problems: in each round, the problem with the largest marginal gain is selected from the remaining candidates and added to the selected set, until a preset number of selections is reached or the marginal gain falls below a set threshold. Marginal gain is defined as the increment in the objective function after adding the problem. Δ(q) = F(S ∪ {q}) - F(S) Where S is the set of currently selected problems, and q is the candidate problem to be evaluated. Termination conditions can be a fixed number limit (e.g., selecting 8-10 problems) or a dynamic threshold limit (stopping when the marginal gain is no longer significant). The algorithm ensures efficient execution by pre-compiling the similarity matrix between candidate problems and employing an incremental update strategy.

[0082] Due to the diminishing marginal returns of the submodular function, the greedy strategy can theoretically guarantee a near-optimal solution for each choice that maximizes the growth of the objective function.

[0083] Application examples of submodulus optimization selection In a legal provision retrieval scenario, suppose a block of content related to a regulation generates 20 candidate questions through S02, and each is evaluated through S03 to obtain a quality score (e.g., Quality values ​​distributed between 0.65 and 0.92). The submodular optimization process is as follows: First, calculate the semantic similarity matrix Sim between each pair of the 20 questions by encoding the questions into vectors and then calculating the cosine similarity (e.g., Sim>0.8 for similar questions, Sim<0.3 for dissimilar questions). Then, apply a greedy algorithm: In the first round, select the question with the highest Quality (e.g., Quality=0.92, marginal gain Δ=0.92); in the second round, calculate the marginal gain of the remaining 19 questions and select the question that maximizes coverage (usually a question with low similarity to the selected questions but high quality, assuming Δ=0.68); continue iterating until the marginal gain drops to a low level (e.g., Δ<0.1), at which point adding more questions has limited effect on improving coverage. This dynamic selection process ensures that the subset is both high-quality (including high-MI value problems) and diverse (the problems cover different aspects), and may ultimately select 8-12 problems, the specific number of which is determined by the complexity of the content and the distribution of problem quality.

[0084] In API technical documentation scenarios, suppose an interface document generates 15 candidate questions with quality scores ranging from 0.58 to 0.89. During submodular optimization, the objective function F(S) comprehensively considers both question quality and coverage: if two questions both ask for parameter information and Sim=0.85, only the one with the higher quality needs to be selected; if one question is about parameters and the other is about return values ​​and Sim=0.25, both may be selected to improve coverage. A greedy algorithm dynamically monitors marginal gain: initially, the selected questions have high marginal gain (e.g., Δ>0.5). As the coverage of the selected set increases, the marginal gain of subsequent questions gradually decreases, automatically stopping when Δ falls below a set threshold. This dynamic mechanism allows simple interfaces to potentially require only 5-7 questions, while complex interfaces may require 10-15 questions, covering different technical elements of the interface (parameters, return values, error codes, etc.) and avoiding duplication and redundancy.

[0085] In a medical diagnostic knowledge base scenario, assuming a disease content block generates 30 candidate questions with quality scores ranging from 0.60 to 0.94, during submodular optimization, the function F(S) automatically balances quality and diversity: if a candidate question contains three similar questions about symptoms (Sim > 0.75), the algorithm selects the one with the highest quality; if it contains three categories of questions about symptoms, diagnosis, and treatment (inter-category Sim < 0.4), each category is selected. Since medical content typically involves multiple dimensions, the marginal gain decreases slowly, and the algorithm may continue selecting until 15-20 questions are chosen, stopping when the marginal gain Δ of new questions is less than 0.08. The final question set guarantees both high mutual information estimates (strongly correlated with content) and coverage of multiple dimensions of the disease, providing a comprehensive and non-redundant question index for the retrieval system.

[0086] The sub-module optimization process is the same for the three scenarios described above: based on the S03 quality score, the similarity between questions is calculated, a greedy algorithm is applied iteratively to select the question with the largest marginal gain, and the marginal gain is dynamically monitored. The process automatically stops when the gain falls below a threshold. This dynamic selection mechanism allows the number of questions to be adaptively determined by the content complexity and candidate quality distribution, rather than a fixed preset number. Therefore, specific preset threshold parameters such as the quality score threshold, similarity threshold, and marginal gain stopping threshold can be adjusted according to the application scenario and content characteristics, and no specific restrictions are imposed here.

[0087] S05: Negative alignment filter.

[0088] For each atomic problem in the optimal problem subset, generalization problems are identified through vector similarity retrieval and large language model discrimination, and the identified generalization problems are enhanced with specificity.

[0089] Identifying generalization problems: Negative alignment filtering aims to prevent issues from mismatching with multiple content blocks. It identifies generalized issues through a two-stage mechanism. Vector similarity filtering: Calculate the similarity between the embedding vector of the current problem and the vectors of problems already stored in the vector database. If multiple highly similar problems exist, they are marked as suspected generalization problems. Large Language Model Judgment: For suspected problems, the large language model is used to determine whether a clear subject or object is missing, whether vague pronouns are used, and whether the semantics are complete when read alone. If it is determined to be a generalization problem, specificity enhancement is performed; if it is determined to be a non-generalization problem, it means that the problem involves knowledge connections between multiple content blocks and belongs to a reasonable cross-content block problem, which is retained.

[0090] Example: The generalized question "What types of it are there?" may match multiple content blocks in different domains and needs to be enhanced into a specific question; while the specific question "What are the complications of type 2 diabetes?" may be highly related to multiple content blocks (symptoms, prevention, treatment, etc.), but because it has a clear subject, it indicates that a complete answer to the question requires multiple knowledge fragments and should be retained.

[0091] Enhanced specificity: Reconstruct problems that are judged to be generalized: Submit the content block and the problem to be processed to the large language model, use the large language model to extract the keywords of the content block and add them to the problem to generate new specific problems.

[0092] This mechanism reduces computational costs while ensuring filtering effectiveness through two-stage discrimination and LLM-assisted reconstruction.

[0093] Example of negative alignment filtering application: In a regulatory text retrieval scenario, assuming S04 optimization outputs 10 questions, negative alignment filtering performs generalization detection on each question. For example, if a question is "What is the scope of application?" (lacking the name of the regulation), vector search finds it highly similar to several other regulation questions already in the database (similarity > 0.85), marking it as a suspected generalization question. Large language model discrimination confirms that the question lacks a clear subject. For specificity enhancement, the keyword "Regulations on Construction Safety Management" is extracted from the text content and reconstructed using LLM as "What is the scope of application of the Regulations on Construction Safety Management?". Finally, the enhanced specificity questions are output and used in index construction.

[0094] In the API technical documentation scenario, assuming S04 outputs 8 questions, negative alignment filtering identifies generalized questions. For example, a question like "What are the error codes?" (lacking an interface name) shows high similarity (>0.80) to questions in other API documentation, potentially causing cross-interface confusion. Large Language Model (LLM) analysis confirms the question lacks a clearly defined object. The specificity enhancement process extracts the keyword "OAuth2.0 token interface" and reconstructs it as "What are the error codes returned by the OAuth2.0 token interface?". For questions with clearly defined objects (e.g., "What are the required parameters of the OAuth2.0 token interface?"), vector retrieval shows low similarity (<0.4) to other API questions, and these are directly passed. If a question like "What are the best practices for microservice architecture?" is highly similar to multiple API documentation content blocks (>0.75), LLM analysis identifies a clear subject, indicating a complete answer requires integrating knowledge fragments from multiple API documentation, constituting a reasonable cross-content block association, and thus retains it.

[0095] In a medical diagnostic knowledge base scenario, assuming S04 outputs 18 questions, negative alignment filtering is used for comprehensive detection. For example, a question like "What are the diagnostic criteria?" (lacking a disease name) shows high similarity (similarity > 0.82) to questions related to multiple diseases, indicating a risk of cross-disease confusion. Large language model (LLM) analysis confirms the lack of a subject. Specific enhancement extracts the keyword "type 2 diabetes" and reconstructs it as "What are the diagnostic criteria for type 2 diabetes?". If a question contains vague pronouns, such as "What are the treatment options for this disease?", it is also identified and enhanced to "What are the treatment options for type 2 diabetes?". For questions like "What are the preventive measures for complications of type 2 diabetes?", although they have high similarity (similarity > 0.78) to multiple content blocks (symptom block, treatment block, nursing block), LLM analysis confirms they have a clear subject and complete semantics, indicating that this question requires integration of multiple knowledge fragments for a complete answer and is a reasonable comprehensive question, thus it is retained.

[0096] The negative alignment filtering process is the same for the three scenarios described above: Vector similarity is performed on each question output by S04, potentially generalized questions are marked, a large language model is used for precise discrimination, keywords are extracted and reconstructed for confirmed generalized questions, and non-generalized but highly similar questions (indicating that multiple knowledge fragments are needed for a complete answer) are retained. This mechanism, through two-stage discrimination and LLM-assisted reconstruction, effectively filters generalized questions while retaining reasonable cross-content block related questions, ensuring high specificity and appropriate knowledge coverage in the final question set.

[0097] S06: Index building.

[0098] Specifically, the final set of atomic problems is stored in a vector database, and a bidirectional mapping index is established between atomic problems and their corresponding content blocks.

[0099] Vector storage and data structures: The question embedding vectors of the final atomic question set are stored in a vector database. The stored representation of an atomic question includes fields such as question identifier, question text, question vector, target content block ID, mutual information score, and quality score, used to support retrieval and quality assessment. The representation structure of a content block includes information such as content block identifier, text content, structured information, metadata, and semantic feature vectors, used to support content management and similarity calculation. Specific field designs can be flexibly adjusted according to the vector database type and application requirements.

[0100] Bidirectional index mapping: Establish a two-way mapping relationship between issues and content blocks, and between content blocks and issues: forward mapping allows for quick location of relevant content blocks based on user queries, while reverse mapping allows for viewing the issue coverage of content blocks.

[0101] Search applications: When a user enters a query, the system encodes the user's question into an embedding vector, retrieves the atomic question with the highest similarity from the vector database, obtains the corresponding content block ID through a forward index, and returns the content block and the matching atomic question for the user's reference. This index structure supports a combination of semantic retrieval and exact matching.

[0102] Application scenario examples The method of this invention can be widely applied to knowledge retrieval scenarios in various fields, and can generate a set of atomic questions with high specificity and low redundancy in each application scenario, significantly improving retrieval accuracy.

[0103] Taking legal provisions, API technical documentation, and medical diagnosis and treatment as examples: In precise retrieval of legal provisions, specific questions containing the name of the regulation (such as "What is the scope of application of the Regulations on Construction Safety Management?") are generated to avoid cross-provisional confusion caused by generalized questions (such as "What is the scope of application of this regulation?"), thus achieving accurate positioning.

[0104] In the API technical documentation intelligent Q&A, highly differentiated questions are generated for different interfaces (such as "What are the error codes for the OAuth2.0 token interface?"), avoiding search conflicts with the documentation of other interfaces and improving the efficiency of developers' queries.

[0105] In the application of medical diagnostic knowledge base, specific questions are generated for the symptoms, diagnostic methods, and treatment plans of different diseases, ensuring that doctors do not mistakenly recall diagnostic information for other diseases when searching for "diagnostic criteria for diabetes".

[0106] This application also provides an atomic problem generation and selection system based on mutual information distillation, including a mutual information discriminator training module, a candidate problem generation module, a mutual information evaluation module, a submodulus optimization selection module, a negative alignment filtering module, and an index storage module. Specifically, the mutual information discriminator training module is configured to train a mutual information discriminator based on a training dataset of positive and negative samples; the candidate problem generation module is configured to call a large language model to generate multiple candidate atomic problems for content blocks; the mutual information evaluation module is configured to call the mutual information discriminator to calculate the mutual information estimate between the candidate atomic problems and the corresponding content blocks; the submodulus optimization selection module is configured to select the optimal subset of problems from the candidate atomic problems based on a submodulus optimization function; the negative alignment filtering module is configured to identify generalization problems and perform specificity enhancement; and the index storage module is configured to construct and store a mapping index between atomic problems and content blocks.

[0107] The mutual information discriminator training module trains a mutual information discriminator based on a training dataset of positive and negative sample pairs. This discriminator is used to calculate the estimated mutual information between the question text and the content blocks. The discriminator employs a dual-tower neural network architecture, such as... Figure 3 As shown, it includes a problem encoder (first encoder) and a content encoder (second encoder). The mutual information estimate is calculated based on the encoded vectors output by the first encoder and the second encoder.

[0108] Encoder architecture: Both encoders are built based on pre-trained language models.

[0109] Pre-trained model selection: This invention uses a Transformer-based bidirectional encoder model (such as BERT, RoBERTa, etc.) instead of an autoregressive generative model (such as GPT, Llama, etc.). The reasons for this selection are as follows: (1) Task adaptability: Bidirectional encoders generate semantic representation vectors of text by simultaneously paying attention to the context, which is naturally suitable for semantic understanding and similarity calculation tasks; while autoregressive models are mainly designed for text generation, and their unidirectional attention mechanism is not conducive to capturing complete bidirectional semantic information.

[0110] (2) Computational efficiency: The bidirectional encoder directly outputs a fixed-dimensional vector representation without generating it token by token; the autoregressive model requires sequential decoding, which has a large computational overhead and is not suitable for vector encoding scenarios.

[0111] (3) Training cost: Bidirectional encoders that have been pre-trained on large-scale corpora can be directly reused, significantly reducing the training data requirements (usually 500-1000 sample pairs are sufficient for fine-tuning) and computational costs.

[0112] Each encoder consists of three components: (1) Pre-trained encoding layer: Reuse the Transformer layer (all or part) of the pre-trained model to convert the text into a deep semantic representation.

[0113] (2) Adaptation layer: The output of the pre-trained model is mapped to a unified low-dimensional vector space (typically 128-512 dimensions) through linear transformation, reducing computation and storage overhead.

[0114] (3) Aggregation layer: Aggregates multiple word vectors of the sequence into a single vector, which can be output using special tags (such as [CLS]) or average pooling.

[0115] The two encoders use the same architecture but have independent parameters, and output the question vector q_vec and the content vector c_vec respectively.

[0116] Mutual information estimation layer: Based on the Mutual Information Neural Estimation (MIS) theory, a discriminant network is used to estimate the mutual information between the question and the content. The specific implementation includes: first, calculating the interaction features of the two vectors (element-wise product of the vectors and cosine similarity); then, concatenating the question vector, content vector, and interaction features to form a fused representation; finally, performing a nonlinear transformation through a multi-layer fully connected network to output the mutual information estimate MI(q_j, c), which is mapped to the [0,1] interval to represent the association strength, where q represents the question text and c represents the content block.

[0117] Training data construction: Positive sample pairs: These are questions and their corresponding correct content blocks, representing high mutual information. Construction methods include generation based on rule templates, generation using large language models, or extraction from existing knowledge bases (such as legal document libraries or technical document libraries). Quality verification is required to ensure a strict correspondence between questions and content blocks and to avoid noise. It is recommended to prepare 500-1000 positive sample pairs for each domain.

[0118] Negative sample pairs: Questions and irrelevant content blocks, representing low mutual information. Construction methods include random sampling from an existing knowledge base, generating questions with mismatched content using a large language model, and intra-batch cross-pairing (correct content from other questions in the same batch). Using diverse negative samples ensures the model learns fine-grained discriminative ability. The ratio of negative samples is controlled between 1:1 and 1:3 to balance the model's discriminative power. For example, if there are 500 positive samples, then there should be 500-1500 negative samples.

[0119] End-to-end training: End-to-end training is performed using a positive-negative sample comparison approach. The input consists of question-content pairs and matching labels (positive sample labels are 1, negative sample labels are 0). The question and content are each encoded into vectors by an encoder, and the matching score is output through a mutual information estimation layer. After calculating the loss, all component parameters are updated simultaneously through backpropagation.

[0120] Using the InfoNCE loss function:

[0121] Where f(q,c) is the mutual information estimate MI(q_j, c), and c + For positive samples (related to the question), cᵢ - The negative samples are irrelevant to the problem, and τ is a temperature parameter (0.05-0.2, default 0.1) used to control the smoothness of the distribution. This loss enhances the scores of positive samples significantly higher than those of negative samples.

[0122] Training strategy: A hierarchical learning rate is adopted, with a smaller learning rate set for the pre-trained encoding layer to retain pre-trained knowledge, and a larger learning rate set for the newly added adaptation layer and mutual information estimation layer to accelerate convergence.

[0123] Evaluation metrics: Classification accuracy for positive and negative samples >85%, mutual information estimate discrimination (mean for positive samples >0.7, mean for negative samples <0.3).

[0124] After training, the mutual information discriminator can effectively calculate the mutual information estimate between any question and content block, providing a quantitative basis for subsequent candidate question quality assessment.

[0125] Furthermore, the candidate question generation module calls the large language model to generate multiple candidate atomic questions for at least one content block.

[0126] The large language model here can be a commercial large language model such as ChatGPT, Claude, or GLM, or a locally deployed open-source large language model such as Llama 2, DeepSeek, or Qwen.

[0127] Prompt design strategy: The large language model is guided to generate diverse candidate atomic questions using prompt words. The prompt words contain the following elements: (1) Input the content block text; (2) Generation requirements: The guided model generation includes at least two different types of questions, including factual, inference, application and conditional questions. Each question is independent and complete and can be answered explicitly from the content block. (3) Quality constraints: The problem statement should be clear and concise, and vague pronouns (such as "it", "that", "this") should be avoided to ensure the specificity of the problem.

[0128] For specific domains, the prompts can be supplemented with domain-specific question types to improve the domain-appropriateness of the generated questions.

[0129] Using the above method, multiple candidate atomic problems are generated for each content block, which then proceed to the next step of mutual information evaluation.

[0130] Furthermore, the mutual information evaluation module calls the mutual information discriminator to calculate the estimated mutual information between each candidate atomic problem and its corresponding content block, and combines it with the length penalty factor to generate a quality score.

[0131] MI value calculation: Input candidate question q and content block c, encode them into vectors q_vec and c_vec through a discriminator, calculate the cosine similarity and output MI(q,c); Batch processing optimization: Batch size 32, GPU acceleration, single problem evaluation time 10-50ms.

[0132] Quality score: The formula is Quality(q) = MI(q,c) × Length_penalty(q), where the length penalty factor Length_penalty(q) = 1 / (1 + a · (len(q) - L_target)), the default parameter L_target=10 (number of words), and a=0.05.

[0133] Threshold application: Filter out problems with an MI value < 0.5 to improve efficiency; high-quality problems have an MI value > 0.7.

[0134] Furthermore, the sub-modular optimization selection module selects the optimal subset of problems from candidate atomic problems based on the sub-modular optimization function, maximizing coverage and ensuring diversity.

[0135] The submodal objective function design adopts the facility location submodal function form: F(S) = Σ_{i∈V} max_{j∈S} [Quality(q_j) · Sim(q_i, q_j)] Where V represents the set of all candidate questions in the current content block, S represents the selected subset of questions, Quality(q_j) is the question quality score (calculated from the mutual information estimate and the length penalty factor), and Sim(q_i, q_j) is the semantic similarity between questions, which can be obtained by encoding the question text into a vector representation and then calculating the cosine similarity. The encoding methods include question encoders that reuse the mutual information discriminator, using independent text embedding models, or other semantic encoding methods.

[0136] The semantics of this function are: each candidate problem is represented by the problem most similar to it in the selected set, and the representation value is equal to the product of the quality of the representative and the similarity. The goal is to select a high-quality and diverse subset that can cover all candidate problems.

[0137] This function has the submodular property and satisfies the diminishing marginal returns characteristic, so a greedy algorithm can obtain a theoretical approximation guarantee.

[0138] Greedy algorithm solution A greedy strategy is used to iteratively select problems: in each round, the problem with the largest marginal gain is selected from the remaining candidates and added to the selected set, until a preset number of selections is reached or the marginal gain falls below a set threshold. Marginal gain is defined as the increment in the objective function after adding the problem. Δ(q) = F(S ∪ {q}) - F(S) Where S is the set of currently selected problems, and q is the candidate problem to be evaluated. Termination conditions can be a fixed number limit (e.g., selecting 8-10 problems) or a dynamic threshold limit (stopping when the marginal gain is no longer significant). The algorithm ensures efficient execution by pre-compiling the similarity matrix between candidate problems and employing an incremental update strategy.

[0139] Due to the diminishing marginal returns of the submodular function, the greedy strategy can theoretically guarantee a near-optimal solution for each choice that maximizes the growth of the objective function.

[0140] Furthermore, the negative alignment filtering module identifies generalization problems and performs specific enhancements to avoid confusion during cross-content block retrieval. Furthermore, the negative alignment filtering module includes a generalization problem identification unit and a specificity enhancement unit.

[0141] The generalization problem identification unit uses a vector database to retrieve the cosine similarity between the current problem and stored problems to identify generalization problems. When the similarity exceeds a preset threshold, a large language model is used to determine whether the problem is a generalization problem. A generalization problem is defined as a problem lacking a clear subject or object, or using vague pronouns that lead to ambiguous semantic meaning. Specifically, Identifying generalization problems: Negative alignment filtering aims to prevent issues from mismatching with multiple content blocks. It identifies generalized issues through a two-stage mechanism. Vector similarity filtering: Calculate the similarity between the embedding vector of the current problem and the vectors of problems already stored in the vector database. If multiple highly similar problems exist, they are marked as suspected generalization problems. Large Language Model Judgment: For suspected problems, the large language model is used to determine whether a clear subject or object is missing, whether vague pronouns are used, and whether the semantics are complete when read alone. If it is determined to be a generalization problem, specificity enhancement is performed; if it is determined to be a non-generalization problem, it means that the problem involves knowledge connections between multiple content blocks and belongs to a reasonable cross-content block problem, which is retained.

[0142] Example: The generalized question "What types of it are there?" may match multiple content blocks in different domains and needs to be enhanced into a specific question; while the specific question "What are the complications of type 2 diabetes?" may be highly related to multiple content blocks (symptoms, prevention, treatment, etc.), but because it has a clear subject, it indicates that a complete answer to the question requires multiple knowledge fragments and should be retained.

[0143] For questions determined to be generalized, the specificity enhancement unit uses a large language model to supplement qualifiers based on the keywords and context of the content block, generating specific questions containing clearly defined objects.

[0144] Specifically, enhanced specificity: Reconstruct problems that are judged to be generalized: Submit the content block and the problem to be processed to the large language model, use the large language model to extract the keywords of the content block and add them to the problem to generate new specific problems.

[0145] This mechanism addresses generalization issues by using LLM to supplement qualifiers based on keywords in the content block. For example, it enhances "What is the scope of application?" to "What is the scope of application of the Regulations on Construction Safety Management?". Through two-stage discrimination and LLM-assisted reconstruction, it reduces computational costs while maintaining filtering effectiveness.

[0146] Finally, the index storage module constructs and stores the mapping index between atomic problems and content blocks. The implementation includes: Vector storage: The final set of atomic problems is stored in a vector database (such as FAISS), with fields including problem ID, text, vector, content block ID, MI value, etc. Bidirectional index: Establishes a forward mapping from issues to content blocks (supporting user queries) and a reverse mapping from content blocks to issues (for monitoring). Retrieval Integration: When a user queries, the query vector is encoded and similar atomic questions are retrieved, and the corresponding content blocks are returned.

[0147] Specifically, vector storage and data structures: The question embedding vectors of the final atomic question set are stored in a vector database. The stored representation of an atomic question includes fields such as question identifier, question text, question vector, target content block ID, mutual information score, and quality score, used to support retrieval and quality assessment. The representation structure of a content block includes information such as content block identifier, text content, structured information, metadata, and semantic feature vectors, used to support content management and similarity calculation. Specific field designs can be flexibly adjusted according to the vector database type and application requirements.

[0148] Bidirectional index mapping: Establish a two-way mapping relationship between issues and content blocks, and between content blocks and issues: forward mapping allows for quick location of relevant content blocks based on user queries, while reverse mapping allows for viewing the issue coverage of content blocks.

[0149] Search applications: When a user enters a query, the system encodes the user's question into an embedding vector, retrieves the atomic question with the highest similarity from the vector database, obtains the corresponding content block ID through a forward index, and returns the content block and the matching atomic question for the user's reference. This index structure supports a combination of semantic retrieval and exact matching.

[0150] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for generating and selecting atomic problems. The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A method for generating and selecting atomic problems based on mutual information distillation, characterized in that, Includes the following steps: Training preparation: Construct a training dataset containing positive and negative sample pairs, and train a mutual information discriminator based on the dataset. The mutual information discriminator is used to calculate the estimated mutual information between the question text and the content block. Candidate question generation: Utilize a large language model to generate multiple candidate atomic questions for at least one content block; Mutual information evaluation: Using the mutual information discriminator, calculate the estimated mutual information between each candidate atomic problem and its corresponding content block; Optimization selection: Based on the mutual information estimate, an optimal subset of problems is selected from the plurality of candidate atomic problems through a submodular optimization function, wherein the submodular optimization function is used to maximize the coverage of the selected subset of problems on the set of candidate atomic problems; Negative alignment filtering: For each atomic problem in the optimal problem subset, the similarity of the stored problems is retrieved using a vector database, and a large language model is used to determine whether it is a generalized problem that lacks a clear subject or object or uses vague pronouns, resulting in unclear semantics. For the generalized problems, clear qualifiers are added to enhance specificity, avoiding cross-content block retrieval confusion caused by vague pronouns, thereby obtaining the final set of atomic problems. Index building: Establish a mapping index between the final atomic problem set and the corresponding content blocks.

2. The method according to claim 1, characterized in that, In the training preparation step, the mutual information discriminator adopts a dual-tower neural network architecture, including a first encoder for encoding the question text and a second encoder for encoding the content blocks. The mutual information estimate is calculated based on the similarity between the encoded vectors output by the first encoder and the second encoder.

3. The method according to claim 1, characterized in that, In the training dataset, a positive sample consists of a question text and its corresponding correct content block, while a negative sample consists of the question text and a randomly selected irrelevant content block.

4. The method according to claim 1, characterized in that, The submodular optimization function in the optimization selection step adopts the facility location function form, expressed as: F(S) = Σ_{i∈V} max_{j∈S} [Quality(q_j) · Sim(q_i,q_j)]; Where V represents the set of all candidate atomic questions in the current content block, S represents the selected subset of questions, q_j represents the selected question, Quality(q_j) represents the quality score of question q_j, which is calculated by multiplying the mutual information estimate MI(q_j, c) with the length penalty factor, and Sim(q_i, q_j) represents the semantic similarity between candidate question q_i and selected question q_j.

5. The method according to claim 4, characterized in that, The quality score Quality(q_j) is calculated as follows: Quality(q_j) = MI(q_j, c) · Length_penalty(q_j), where MI(q_j, c) is the relevance score between the question and the content block output by the mutual information discriminator, and Length_penalty(q_j) is the length penalty factor, which is calculated based on the length of the question text. Concise questions are given higher weights, while lengthy questions are penalized. The question similarity Sim(q_i, q_j) is obtained by calculating the semantic similarity after encoding the question text into a vector representation.

6. The method according to claim 1, characterized in that, The negative alignment filtering step includes two stages: generalization problem identification and specificity enhancement. Generalization problem identification stage: The cosine similarity between the current problem and the stored problems is retrieved using a vector database. When the similarity exceeds a preset threshold, a large language model is used to determine whether the problem is a generalization problem. The generalization problem is defined as a problem that lacks a clear subject or object or uses vague pronouns, resulting in unclear semantic reference. Specificity enhancement stage: For questions judged as generalized, the large language model supplements qualifiers based on the keywords and context of the content block to generate specific questions containing clearly defined objects.

7. The method according to claim 1, characterized in that, In the candidate question generation step, the large language model is guided to generate candidate atomic questions of at least two different types, including factual, inferential, applicable, and conditional questions.

8. An atomic problem generation and selection system based on mutual information distillation, characterized in that, include: The mutual information discriminator training module is configured to train a mutual information discriminator based on the training dataset of positive and negative sample pairs. The candidate question generation module is configured to call the large language model to generate multiple candidate atomic questions for content blocks. The mutual information evaluation module is configured to call the mutual information discriminator to calculate the estimated mutual information between the candidate atomic problem and the corresponding content block; The sub-modulus optimization selection module is configured to select the optimal subset of problems from candidate atomic problems based on the sub-modulus optimization function. The negative alignment filtering module is configured to identify generalization problems and perform specific enhancements. as well as The index storage module is configured to build and store the mapping index between atomic problems and content blocks.

9. The system according to claim 8, characterized in that, The negative alignment filtering module includes: The generalization problem identification unit uses a vector database to retrieve the cosine similarity between the current problem and the stored problems to identify generalization problems. When the similarity exceeds a preset threshold, a large language model is used to determine whether the problem is a generalization problem. The generalization problem is defined as a problem that lacks a clear subject or object or uses vague pronouns, resulting in unclear semantic direction. The specificity enhancement unit, for questions determined to be generalized, uses a large language model to supplement qualifiers based on keywords and context in the content block, generating specific questions containing clearly defined objects.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed, implements the method as described in any one of claims 1-7.

Images