A method and system for candidate set production based on multi-view hints

Abstract

A candidate set production method based on multi-view hints, comprising the following steps: 1) multi-view hint acquisition; multi-dimensional information strongly related to the task is extracted from two types of modalities of image views and text hints; 2) candidate set generation; 2.1) single-modal candidate set generation; 2.2) multi-modal candidate set fusion; single-modal candidate sets are integrated into multi-modal candidate sets by using multi-modal embedding and / or joint attention mechanism, and complementary integration of image and text information is realized; 2.3) answer post-processing; one or more operations of merging, filtering and sorting are performed on the fused multi-modal candidate sets; 3) final answer selection; the final answer selection is determined by confidence evaluation and answer verification on the multi-modal candidate sets. The application introduces a multi-level contrast learning framework; the framework fuses global semantic contrast on the whole level and local feature contrast on the detail level, aiming to significantly reduce the analysis error and improve the reliability of the output result.

Description

Technical Field

[0001] This invention relates to the field of AI applications, and in particular to a method and system for producing candidate sets based on multi-view prompts. Background Technology

[0002] Currently, the generation of candidate sets for multi-view suggestions requires the integration of multimodal information such as images and text. The process faces core challenges including inconsistent data fusion, difficulties in feature differentiation, information redundancy / loss, and the selection of fusion strategies, as detailed below:

[0003] 1. It is difficult to distinguish between common and redundant information in the feature space;

[0004] Relying solely on the consistency and reconstruction goals of the feature space makes it difficult to accurately distinguish common information. This is mainly due to the handling of cross-modal correlations and redundant information that are crucial to the task, including useless or interfering information. If redundant information is misjudged as common information, the candidate set will contain invalid content; if key information is classified as redundant and filtered out, information loss will occur, ultimately affecting the accuracy and effectiveness of the candidate set.

[0005] 2. Multi-view Figure 1 The complexity of consistency constraints

[0006] Multiview Figure 1 Consistency (spatial, geometric, and semantic consistency) is the foundation for candidate set generation, but achieving consistency constraints is challenging. Spatial consistency: the spatial positions of multiple views must be strictly matched. For example, the position / orientation of a multi-camera system needs to be calibrated; otherwise, the candidate set will be spatially misaligned after fusion. Geometric consistency: in tasks such as 3D reconstruction, the geometric structure of multiple views must be consistent. For example, image feature point matching must be accurate; otherwise, the geometric information of the candidate set will be distorted.

[0007] Semantic consistency: In advanced scenarios, such as multimodal question answering, the semantics of different views need to be consistent. For example, the image "car" and the text "car" need to be semantically aligned, otherwise the semantic information of the candidate set will be contradictory.

[0008] 3. Limitations of Cluster Quality Assessment and Optimization

[0009] Multi-view candidate set generation often relies on clustering techniques, but the quality assessment of basal clusters has shortcomings. The "density" of basal clusters only reflects the distribution of the data itself and cannot directly reflect the cluster quality. Traditional evaluation methods, such as entropy and silhouette coefficient, are not well adapted to multi-view scenarios, which can easily lead to bias in candidate set quality assessment and thus affect the accuracy of the final candidate set. Summary of the Invention

[0010] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a candidate set production method and system based on multi-view prompts.

[0011] To solve the above-mentioned technical problems, the technical solution proposed by this invention is: a candidate set production method based on multi-view prompts, comprising the following steps:

[0012] 1) Multi-view cues acquisition: Through basic strategy setting and cues generation, construction of a multi-view cues system, comparative learning to optimize cues and reasoning and EHR text generation, multi-dimensional information strongly related to the task is extracted from two modalities: image view and text cues.

[0013] 2) Candidate set generation;

[0014] 2.1) Single-modal candidate set generation; Candidate answer sets are generated separately for image and text modalities;

[0015] Feature extraction is performed on the image view, and a set of candidate answers is generated by combining the image context;

[0016] The text prompts are analyzed, and a set of candidate answers is generated by combining the text context.

[0017] 2.2) Multimodal candidate set fusion;

[0018] By utilizing multimodal embedding and / or joint attention mechanisms, the image candidate answer set and the text candidate answer set are integrated into a multimodal candidate set, achieving complementary integration of image and text information;

[0019] 2.3) Answer post-processing; perform one or more operations such as merging, filtering, and sorting on the fused multimodal candidate set to improve the quality of the multimodal candidate set;

[0020] 3) Final answer selection: The final answer selection is determined by confidence evaluation and answer verification of the multimodal candidate set generated in step 2).

[0021] In the above-described candidate set production method based on multi-view prompts, preferably, in step 1),

[0022] The basic strategy setting and prompt generation adopts prefix prompts and dynamic prompts to provide the model with task-oriented prior knowledge;

[0023] Prefix hints: Add trainable "prefix" modules to pre-trained language models, allowing the model to quickly adapt to dialogue summarization tasks by fine-tuning the prefix parameters;

[0024] Dynamic prompt generation: Prompts are dynamically generated based on the dialogue context, ensuring that the prompts are strongly relevant to the current dialogue content.

[0025] In the aforementioned candidate set production method based on multi-view prompts, preferably, the pre-trained language model uses prompts from multiple different perspectives to encode or generate the same input, thereby constructing a diverse and complementary candidate solution set.

[0026] In the above-mentioned candidate set production method based on multi-view prompts, preferably, step 1) constructing a multi-view prompt system includes global view prompts and local view prompts to achieve hierarchical extraction of dialogue information;

[0027] Global perspective hint: Used to capture the macro-features of the entire doctor-patient dialogue, providing a global semantic framework for subsequent generation;

[0028] Local perspective cues: Design multiple project-specific cues to generate independent cues for each project in the dialogue, guiding the model to accurately extract local information of the project and ensuring the accuracy of information extraction.

[0029] In the aforementioned candidate set production method based on multi-view prompts, preferably, the macro-features include one or more of the following: dialogue topic, core event, and overall semantic context.

[0030] The above-mentioned candidate set production method based on multi-view prompts preferably divides the global view into multiple local views according to preset angles, and each local view is defined as a project.

[0031] In the above-mentioned candidate set production method based on multi-view prompts, preferably, in step 1), the contrastive learning optimization prompts introduce global contrastive learning and local contrastive learning to improve the discriminativeness and robustness of the prompts;

[0032] Global contrastive learning: The global features of the entire dialogue are compared and learned to distinguish between "effective global information" and "redundant global information" to ensure that the generated summary covers the core semantics of the dialogue;

[0033] Local contrastive learning: Comparative learning is performed on the local features of each "item" to distinguish between "effective local information" and "redundant local information" to ensure that the generated summary contains key details.

[0034] In the above-mentioned candidate set production method based on multi-view prompts, preferably, in step 1), the reasoning and EHR text generation involves: sequentially concatenating prefix prompts, global perspective prompts, local perspective prompts, and the original dialogue text to form the complete input of the pre-trained language model; the pre-trained language model then generates a summary text containing key dialogue information based on the input.

[0035] A candidate set production system based on multi-view prompts is provided to implement the above-mentioned candidate set production method based on multi-view prompts, including a multi-view prompt acquisition unit, a candidate set generation unit, and a final answer selection unit.

[0036] The multi-view prompt acquisition unit extracts multi-dimensional information strongly related to the task from two modalities: image view and text prompt, through basic strategy setting and prompt generation, construction of a multi-view prompt system, comparative learning to optimize prompts, and reasoning and EHR text generation.

[0037] The candidate set generation unit includes a single-modal candidate set generation unit, a multi-modal candidate set fusion unit, and an answer post-processing unit;

[0038] The single-modal candidate set is generated; candidate answer sets are generated separately for image and text modalities.

[0039] The multimodal candidate set fusion unit utilizes multimodal embedding and / or joint attention mechanisms to integrate the image candidate answer set and the text candidate answer set into a multimodal candidate set, thereby achieving complementary integration of image and text information.

[0040] The answer post-processing unit performs one or more operations, such as merging, filtering, and sorting, on the fused multimodal candidate set to improve the quality of the multimodal candidate set.

[0041] The final answer selection unit determines the final answer selection by evaluating the confidence level and verifying the answer in the multimodal candidate set generated by the candidate set generation unit.

[0042] Compared with the prior art, the advantages of the present invention are as follows: The candidate set production method and system based on multi-view prompts of the present invention introduces a multi-level contrastive learning framework; this framework integrates global semantic contrast at the overall level and local feature contrast at the detail level, aiming to significantly reduce parsing errors and improve the reliability of output results. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the process structure of the candidate set production method based on multi-view prompts in Example 1. Detailed Implementation

[0044] To facilitate understanding of the present invention, the present invention will be described more fully and in detail below with reference to preferred embodiments, but the scope of protection of the present invention is not limited to the following specific embodiments.

[0045] It should be noted that when a component is described as being "fixed to, attached to, connected to or connected to" another component, it can be directly fixed to, attached to, connected to or connected to the other component, or it can be indirectly fixed to, attached to, connected to or connected to the other component through other intermediate connectors.

[0046] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by those skilled in the art. The technical terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the scope of the invention.

[0047] Example 1

[0048] like Figure 1 The method for generating a candidate set based on multi-view prompts, as shown, includes the following steps:

[0049] 1) Multi-view cues acquisition: Through basic strategy setting and cues generation, construction of a multi-view cues system, comparative learning to optimize cues and reasoning and EHR text generation, multi-dimensional information strongly related to the task is extracted from two modalities: image view and text cues.

[0050] 2) Candidate set generation;

[0051] 2.1) Single-modal candidate set generation; Candidate answer sets are generated separately for image and text modalities;

[0052] Feature extraction is performed on the image view, such as visual feature encoding; combined with the image context, such as the relationship between images, a set of image candidate answers is generated.

[0053] The system performs text analysis on the text prompts, such as text feature encoding and semantic understanding; and combines the text context, such as document structure and dialogue logic, to generate a set of candidate answers.

[0054] 2.2) Multimodal candidate set fusion;

[0055] By utilizing multimodal embedding and / or joint attention mechanisms, the image candidate answer set and the text candidate answer set are integrated into a multimodal candidate set to achieve complementary integration of image and text information; multimodal embedding includes image-text joint embedding.

[0056] The core of integrating two unimodal candidate sets—image candidate answers and text candidate answers—into a multimodal candidate set is multimodal information fusion and contextual consistency checking: ensuring that candidate answers are logically coherent and consistent within their context. For example, if there is a timeline conflict between image and text candidate answers, it needs to be corrected using contextual rules. Multimodal information fusion utilizes techniques such as multimodal embedding and joint attention mechanisms to achieve complementary integration of image and text information. For example, visual features of images can enhance semantic understanding of text, or semantic information from text can supplement the context of images. Multimodal embedding includes image-text joint embedding.

[0057] 2.3) Answer post-processing; perform one or more operations such as merging, filtering, and sorting on the fused multimodal candidate set to improve the quality of the multimodal candidate set;

[0058] The merged candidate set undergoes refined processing to improve candidate quality. Answer merging combines duplicate or highly similar candidate answers to avoid redundancy. Answer re-ranking re-ranks candidate answers based on indicators such as confidence level and relevance, for example, prioritizing answers with higher confidence levels.

[0059] Answer filtering eliminates candidate answers that are irrelevant to the question, logically contradictory, or of low quality, ensuring the quality of the candidate set.

[0060] In this embodiment, a consistency constraint mechanism based on block diagonal guidance and collaborative representation learning is introduced to solve the multi-view problem. Figure 1 The core of this invention addresses the complexity of consistency constraints. It avoids explicitly constructing and fusing complex global similarity graphs, instead using a constraint model to learn a unified representation with a clear block-diagonal structure, thus implicitly achieving high-order consistency among multiple views. Traditional methods typically require constructing a similarity graph for each view before performing complex graph fusion and constraints, such as Laplace rank constraints, to obtain a unified representation. This invention abandons this cumbersome process by directly introducing a block-diagonal regularization term into the model's optimization objective; guiding the model to learn a unified multi-view representation matrix Z, which approximates an ideal block-diagonal matrix. In an ideal block-diagonal matrix, samples within the same cluster have high similarity (non-zero values), while the similarity between different clusters is zero. This directly encodes the clustering structure, avoiding the indirect acquisition of this structure through complex graph operations.

[0061] 3) Final answer selection: The final answer selection is determined by confidence evaluation and answer verification of the multimodal candidate set generated in step 2).

[0062] In this embodiment, in the final answer selection stage, the most likely correct answer is determined through two steps: confidence evaluation and answer verification. In confidence evaluation, each answer in the candidate set is scored for confidence, and the answer with the highest confidence is selected as a candidate. In answer verification, if there is an additional verification mechanism, such as a knowledge base or external data, the candidate answer is verified to further improve the accuracy. For example, the candidate answer is compared with the domain knowledge base to confirm its rationality.

[0063] In this embodiment, assume a candidate answer a i is generated by the model, and its generation probability is P(ai∣C,vk)P(ai∣C,V k ), where C is the input context (question), and V k is the hint of the k-th view.

[0064] Normalized confidence score: To compare answers of different lengths and views, calculate the normalized log-likelihood of each answer as its confidence score

[0065] where, ∣a i ∣ is the token length of the answer ai, W t is the t-th token, and w<t is all the previous tokens. This score reflects the "average confidence level" of the model in generating this answer.

[0066] In step 1) of this embodiment, for the basic strategy setting and prompt generation, prefix tuning (Prefix-Tuning) and dynamic prompts are adopted to provide task-oriented prior knowledge for the model;

[0067] Prefix tuning (Prefix-Tuning): Add a trainable "prefix" module to the pre-trained language model, such as T5, GPT. By fine-tuning the prefix parameters, the model can quickly adapt to the task of dialogue summary generation; reduce the dependence on large-scale data;

[0068] Dynamic prompt generation: Dynamically generate prompts in combination with the dialogue context to ensure that the prompts are strongly relevant to the current dialogue content.

[0069] In this embodiment, the pre-trained model encodes or generates the same input using multiple prompts from different perspectives, thereby constructing a diverse and complementary set of candidate solutions to improve the robustness and accuracy of the final result. This method breaks through the limitations of traditional single-prompt generation. By introducing a "multi-view" mechanism, the model explores the solution space from multiple dimensions such as semantics, logic, and style.

[0070] To further enhance collaboration and alignment among multiple views, a chunk alignment objective can be introduced during the pre-training phase. This objective learns fine-grained alignment relationships between language and images by predicting whether an image chunk corresponding to a text word is occluded. This cross-modal alignment capability is crucial in multi-view-based cue production methods, ensuring that the model understands and associates the shared semantic core behind cues from different perspectives, thereby generating a complementary and intrinsically consistent set of candidate answers.

[0071] Specifically, a preferred configuration is to adopt a model design philosophy similar to LayoutLMv3. This model simplifies the design through a simple architecture and a unified pre-training objective, enhancing its ease of use in downstream tasks. Its core lies in using a unified text and image masking modeling objective to pre-train the multimodal model; that is, the model simultaneously learns to reconstruct masked word IDs in the language modality and symmetrically reconstructs masked image patch IDs in the image modality. This symmetrical masking reconstruction mechanism enables the model to understand input information from different views (modalities) such as text and vision equally and deeply, laying a unified representational foundation for generating diverse candidate sets.

[0072] In this embodiment, step 1) constructing a multi-view prompting system includes global view prompts (INP) and local view prompts (ItPs) to achieve hierarchical extraction of dialogue information;

[0073] Global Perspective Hints (INP): Used to capture the macro-features of the entire doctor-patient dialogue, such as one or more of the dialogue topic, core events, and overall semantic context, providing a global semantic framework for subsequent generation;

[0074] Local Perspective Cueing (ItPs): Multiple item-specific cues are designed to generate independent cues for each item in the dialogue, guiding the model to accurately extract local information for that item and ensuring the accuracy of information extraction. In this embodiment, the global perspective is divided into multiple local perspectives based on preset angles, and each local perspective is defined as an item. In this embodiment, for example in a doctor-patient dialogue, an item can be one or more of the following: symptom description, examination results, treatment plan, etc.

[0075] In this embodiment, the contrastive learning optimization prompts in step 1) introduce global contrastive learning and local contrastive learning to improve the discriminativeness and robustness of the prompts;

[0076] Global contrastive learning: The global features of the entire dialogue are compared and learned to distinguish between "effective global information" and "redundant global information" to ensure that the generated summary covers the core semantics of the dialogue;

[0077] Local contrastive learning: Comparative learning is performed on the local features of each "item" to distinguish between "effective local information" and "redundant local information" to ensure that the generated summary contains key details.

[0078] This embodiment addresses the difficulty of distinguishing common and redundant information in the feature space by introducing an adaptive information decoupling mechanism based on local contrastive learning. The core principle is to avoid a brute-force separation in the global feature space. Instead, it guides the model to automatically learn to distinguish and strengthen common information crucial to the task by constructing a local contrastive task, while suppressing or ignoring redundant information. The specific model construction can be as follows: For an input sample xx, two different views v1 and v2 are generated through data augmentation such as cropping and color dithering. The model includes a shared encoder fθ and a projection head gϕ. After the encoder extracts features, instead of performing global pooling on the entire feature map, it divides it into multiple local regions or feature blocks.

[0079] In this embodiment, in step 1), reasoning and EHR text generation involve concatenating prefix cues, global perspective cues, local perspective cues, and the original dialogue text in sequence to form the complete input of the pre-trained language model; the pre-trained language model then generates a summary text containing key dialogue information based on the input.

[0080] This embodiment also provides a candidate set production system based on multi-view prompts, including a multi-view prompt acquisition unit, a candidate set generation unit, and a final answer selection unit;

[0081] The multi-view prompt acquisition unit extracts multi-dimensional information strongly related to the task from both image view and text prompt modalities through basic strategy setting and prompt generation, construction of a multi-view prompt system, comparative learning to optimize prompts, and inference and EHR text generation.

[0082] The candidate set generation unit includes a single-modal candidate set generation unit, a multi-modal candidate set fusion unit, and an answer post-processing unit.

[0083] The single-modal candidate set is generated; candidate answer sets are generated for image and text modalities respectively.

[0084] The multimodal candidate set fusion unit utilizes multimodal embedding and / or joint attention mechanisms to integrate the image candidate answer set and the text candidate answer set into a multimodal candidate set, achieving complementary integration of image and text information.

[0085] The answer post-processing unit performs one or more operations, such as merging, filtering, and sorting, on the fused multimodal candidate set to improve its quality.

[0086] The final answer selection unit determines the final answer selection based on the confidence level and answer verification of the multimodal candidate set generated by the candidate set generation unit.

Claims

1. A method for generating a candidate set based on multi-view prompts, characterized in that, Includes the following steps: 1) Multi-view cues acquisition: Through basic strategy setting and cues generation, construction of a multi-view cues system, comparative learning to optimize cues and reasoning and EHR text generation, multi-dimensional information strongly related to the task is extracted from two modalities: image view and text cues. 2) Candidate set generation; 2.1) Single-modal candidate set generation; Candidate answer sets are generated separately for image and text modalities; Feature extraction is performed on the image view, and a set of candidate answers is generated by combining the image context; The text prompts are analyzed, and a set of candidate answers is generated by combining the text context. 2.2) Multimodal candidate set fusion; By utilizing multimodal embedding and / or joint attention mechanisms, the image candidate answer set and the text candidate answer set are integrated into a multimodal candidate set, achieving complementary integration of image and text information; 2.3) Answer post-processing; perform one or more operations such as merging, filtering, and sorting on the fused multimodal candidate set to improve the quality of the multimodal candidate set; 3) Final answer selection: The final answer selection is determined by confidence evaluation and answer verification of the multimodal candidate set generated in step 2).

2. The candidate set production method based on multi-view prompts according to claim 1, characterized in that: In step 1), The basic strategy setting and prompt generation adopts prefix prompts and dynamic prompts to provide the model with task-oriented prior knowledge; Prefix hints: Add trainable prefix modules to pre-trained language models, allowing the model to quickly adapt to dialogue summarization tasks by fine-tuning prefix parameters; Dynamic prompt generation: Prompts are dynamically generated based on the dialogue context, ensuring that the prompts are strongly relevant to the current dialogue content.

3. The candidate set production method based on multi-view prompts according to claim 2, characterized in that: The pre-trained language model uses prompts from multiple different perspectives to encode or generate the same input, thereby constructing a diverse and complementary set of candidate solutions.

4. The candidate set production method based on multi-view prompts according to claim 2, characterized in that: Step 1) constructs a multi-perspective prompting system, including global perspective prompts and local perspective prompts, to achieve hierarchical extraction of dialogue information; Global perspective hint: Used to capture the macro-features of the entire doctor-patient dialogue, providing a global semantic framework for subsequent generation; Local perspective cues: Design multiple project-specific cues to generate independent cues for each project in the dialogue, guiding the model to accurately extract local information of the project and ensuring the accuracy of information extraction.

5. The candidate set production method based on multi-view prompts according to claim 4, characterized in that: The macro-features include one or more of the following: dialogue topic, core event, and overall semantic context.

6. The candidate set production method based on multi-view prompts according to claim 4, characterized in that: The global view is divided into multiple local views based on preset angles, and each local view is defined as a project.

7. The candidate set production method based on multi-view prompts according to claim 4, characterized in that: In step 1), the contrastive learning optimization prompts introduce global contrastive learning and local contrastive learning to improve the discriminativeness and robustness of the prompts. Global contrastive learning: The global features of the entire dialogue are compared and learned to distinguish between effective global information and redundant global information, ensuring that the generated summary covers the core semantics of the dialogue; Local contrastive learning: Comparative learning is performed on the local features of each item to distinguish between effective local information and redundant local information.

8. The candidate set production method based on multi-view prompts according to claim 7, characterized in that: In step 1), reasoning and EHR text generation involve concatenating prefix cues, global perspective cues, local perspective cues, and the original dialogue text in sequence to form the complete input of the pre-trained language model. Based on the input, the pre-trained language model generates a summary text containing dialogue information.

9. A candidate set production system based on multi-view prompts, used to implement the candidate set production method based on multi-view prompts according to any one of claims 1-9, characterized in that: It includes a multi-view prompt acquisition unit, a candidate set generation unit, and a final answer selection unit; The multi-view prompt acquisition unit extracts multi-dimensional information strongly related to the task from two modalities: image view and text prompt, through basic strategy setting and prompt generation, construction of a multi-view prompt system, comparative learning to optimize prompts, and reasoning and EHR text generation. The candidate set generation unit includes a single-modal candidate set generation unit, a multi-modal candidate set fusion unit, and an answer post-processing unit. The single-modal candidate set is generated; candidate answer sets are generated separately for image and text modalities. The multimodal candidate set fusion unit utilizes multimodal embedding and / or joint attention mechanisms to integrate the image candidate answer set and the text candidate answer set into a multimodal candidate set, thereby achieving complementary integration of image and text information. The answer post-processing unit performs one or more operations, such as merging, filtering, and sorting, on the fused multimodal candidate set to improve the quality of the multimodal candidate set. The final answer selection unit determines the final answer selection by evaluating the confidence level and verifying the answer to the multimodal candidate set generated by the candidate set generation unit.

Images