Method and system for quantitative research scheme based on big language model conception

Abstract

The invention discloses a method and a system for conception quantitative research scheme based on a big language model, aiming at challenges and requirements in the whole process of the conception quantitative research scheme of HCI novice researchers, information input by a user is analyzed by adopting the big language model, and retrieval conditions are constructed based on an analysis result to carry out academic index API retrieval. And in combination with an original text injection method, semantic deviation of a vectorization method is avoided, so that it is ensured that the generation scheme has a complete argumentation chain and verifiability, and the problem that evidence is lack of traceability in an existing method is solved. According to the method, the reference pool is established, so that the generation of the quantitative research scheme or the modules in the quantitative research scheme preferentially uses confirmed evidence, and the reliability of the quantitative research scheme is improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to a method and system for conceiving quantitative research schemes based on large language models. Background Technology

[0002] Quantitative research is a core method for ensuring transparency and reproducibility in the field of human-computer interaction. It encompasses hypothesis formulation, experimental design, sample size estimation, statistical testing, and result presentation, demanding high levels of standardization and logical consistency. High-quality research proposals not only promote academic output but also play a crucial role in research training and mentorship. However, novice researchers, lacking systematic domain knowledge and methodological training, often don't know how to begin quantitative research and struggle to identify potential problems in their proposals.

[0003] Traditional methods relying on literature retrieval and manual writing have limited efficiency. While general-purpose large language models can generate text, they often suffer from issues such as outdated knowledge, semantic drift, and insufficient academic traceability, making it difficult to meet the professional needs of quantitative research. Existing auxiliary tools also often lack structured output and systematic evaluation mechanisms for quantitative research. By combining the generation capabilities of large language models with real-time updates to academic databases, a workflow system integrating retrieval, generation, and evaluation can be established. Through engineered prompt templates and context management mechanisms, not only can the output be ensured to conform to academic norms, but sources can also be clearly cited in the research proposal, thereby improving the reliability and traceability of the results.

[0004] US Patent Publication No. US 2025 / 0209273 A1 discloses a research scheme conception method based on a large language model, which has the following characteristics: (1) It adopts a dual-agent architecture: one agent is used to propose or clarify the research question, and the other agent is responsible for generating the corresponding research methods and scheme drafts; (2) It adopts a dual-repository retrieval mechanism, including global repository recall and user repository screening; (3) Iterative process of dynamic verification and method synthesis. The whole process is realized through a multi-step iterative process, and users can interact with the system at key nodes to promote the gradual improvement of the scheme. In addition, the method clarifies the four-step usage process of the system, and defines the specific steps of solution generation by decomposing the core research question into sub-problems, retrieving relevant information based on sub-problems, and integrating the retrieval results to generate candidate schemes. However, the functional boundary of this scheme focuses on the research question proposal, research motivation verification and preliminary solution synthesis, and does not include the quantitative research link in the functional system. It lacks structured process design and technical support for quantitative research scenarios (such as experimental schemes, measurement indicators, data analysis methods, etc.).

[0005] Patent application CN 116991977A discloses a precise domain vector knowledge retrieval method based on a large language model. It proposes splitting candidate documents into subtexts, vectorizing each subtext, and establishing a file vector database. Simultaneously, key information is structured and stored as metadata. During retrieval, matching is first performed based on structured conditions, followed by a secondary vectorized matching, to improve the accuracy and efficiency of unstructured text retrieval. This patent belongs to the field of text data processing technology and aims to address the technical pain points of low accuracy and efficiency in existing unstructured text data retrieval. It provides a precise domain vector knowledge retrieval method and device based on a large language model, suitable for efficient and accurate retrieval of various unstructured text resources (such as PDF, Word, and XML formats). However, this solution mainly focuses on the retrieval module itself and has not yet been specifically applied and verified in academic research scenarios. It is worth noting that the retrieval and analysis of academic papers are crucial for researchers' solution design, deepening their understanding of relevant domain knowledge and providing strong support for the selection and demonstration of methodologies. Summary of the Invention

[0006] This invention provides a method for conceiving quantitative research schemes based on a large language model, which can more accurately and quickly conceive quantitative research schemes.

[0007] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention addresses the challenges and needs faced by novice HCI researchers in the entire process of devising quantitative research schemes. By employing a large language model to parse user input information, constructing search conditions based on the parsing results for academic index API retrieval, and combining this with a text injection method, the generated scheme is guaranteed to have a complete chain of argumentation and verifiability, thus solving the problem of lack of traceable evidence in existing methods.

[0008] This invention improves the reliability of quantitative research protocols by establishing a citation pool, which prioritizes the use of confirmed evidence in the generation of quantitative research protocols or their modules. Attached Figure Description

[0009] Figure 1 A flowchart illustrating a method for devising a quantitative research scheme based on a large language model, provided as a specific embodiment of the present invention; Figure 2 This is a webpage provided for specific embodiments of the present invention, which allows users to obtain recommended literature and view its metadata and quantitative research plans.

[0010] Figure 3 This is a webpage for generating quantitative research schemes, provided as a specific embodiment of the present invention.

[0011] Figure 4This is a comparison chart showing the quality scores of the system constructed by the method provided in a specific embodiment of the present invention and the scheme generated by the benchmark system, as evaluated by experts.

[0012] Figure 5 A box plot comparing the USE scale (availability) scores of the system constructed by the method provided in a specific embodiment of the present invention with those of a benchmark system during the generation of a quantitative research protocol.

[0013] Figure 6 A box plot comparing the UEQ (User Experience) scores of the system constructed by the method provided in a specific embodiment of the present invention with those of a benchmark system during the generation of a quantitative research protocol.

[0014] Figure 7 A box plot comparing the NASA-TLX scale (cognitive load) in the process of generating quantitative research protocols between the system constructed by the method provided in a specific embodiment of the present invention and a benchmark system.

[0015] Figure 8 This is a stacked diagram showing the effectiveness ratings of ten core functions of the system provided by the present invention, provided as a specific embodiment of the invention. Detailed Implementation

[0016] This invention provides a method for devising quantitative research schemes based on large language models, such as... Figure 1 As shown, it includes: S1. The research background and objectives input by the user are parsed using a large language model. Specifically, the user input information is sent to a parser, which is a large language model (LLM Prompt). The user input information includes research background and objectives, including keywords, research interests, target variables, and method preferences. Based on pre-defined prompts, the large language model extracts structured query elements from the user input information. These structured query elements include keywords and research fields. For example, the large model extracts the user's research background, research objectives, and search keywords needed to retrieve relevant papers. For instance, if the user inputs "I want to research..." into the AI, the large model will extract these three data points. The query elements are used to obtain keywords for recommended papers. Because obtaining recommended papers calls the OpenAlex API, some query keywords are required, such as: LLMs, psychology, and optimization algorithms.

[0017] In one embodiment, the pre-set prompt words provided in this embodiment, such as Figure 1 In The set keyword prompt (user suggestion) is: You are an academic literature retrieval expert. Based on the following research background and research purpose, please select 1–3 professional English keywords that are most suitable for retrieving relevant academic literature. Research background: [background] Research purpose: [purpose], please return the results in JSON format. Note: The text in [ ] represents the data to be inserted in the prompt.

[0018] S2. Based on Boolean search conditions, retrieve multiple documents through the Academic Index API. Extract the corresponding research plan from the full text fragments of each retrieved document using the original text injection method, and add it to the model context in the set output format.

[0019] In one specific embodiment, multiple documents are retrieved using the Academic Index API based on Boolean search criteria, including: Boolean search criteria are constructed based on the parsed elements, and the academic indexing API is called. In a specific embodiment, the academic indexing API is OpenAlex. Precise search is performed, matching is performed based on the title, abstract, keywords, and journal of the document, and filtering is performed according to the publication year, citation count, and field.

[0020] In one specific embodiment, the present invention provides a method for performing filtering, comprising: The first layer of filtering: The metadata / quality filtering system uses metadata tags provided by the OpenAlex database for automated quality control. If a paper is marked as is_retracted (retracted paper) or is_paratext (informal paper / non-textual content), the system will automatically exclude it.

[0021] The second layer of filtering: Target domain filtering. Considering the interdisciplinary nature of human-computer interaction, the system utilizes the concepts field of papers in the OpenAlex database. By specifying specific "concept IDs," the system strictly limits the search scope to four strongly related target domains: design, computer science, human-computer interaction, and psychology.

[0022] Third-level filtering: Top conference / top journal filtering (optional). Users can choose to enable the "Top Journals and Conferences" mode. When enabled, the system will impose further restrictions, retaining only papers published in JCR Q1-Q2 journals and CORE A and above conferences.

[0023] Furthermore, the system will combine keyword matching strength and citation volume to calculate a relevance score, and papers with higher relevance scores will be displayed higher in the rankings.

[0024] The specific steps for selecting documents that meet accessibility requirements are as follows: 1. During the search, the system not only matches the title and abstract, but also directly filters open access (OA) articles through the OpenAlex database API.

[0025] 2. The system is connected to the open API of CORE, the world's largest open access academic paper aggregation database. If complete text data that meets the criteria can be successfully retrieved through this API and passes OpenAlex's open access filter, it is determined to "meet accessibility requirements".

[0026] In one specific embodiment, the full-text fragment of each retrieved document is added to the model context in a defined output format using a full-text injection method, including: The method retrieves full-text fragments of selected documents by calling CORE (Concepting Repositories) or other full-text libraries. These fragments are then segmented, and research proposals are extracted from each segment using a large language model based on predefined prompts and a predefined output format. The research proposals from each segment are then integrated to obtain the research proposal for the selected documents. In this specific embodiment, the method incorporates the research proposal into the model context through full-text injection, rather than simply as semantic fragments for vector retrieval. This approach preserves the original text's argumentation chain, facilitating the generation of accurate descriptions of methodological details and experimental settings.

[0027] Due to the excessive length of the paper, it was divided into blocks. The research method was analyzed by segmentation using a large language model, and then prompt words were constructed by splicing them together. The prompt words also needed to be constructed in advance. Finally, the research method was obtained through the large language model.

[0028] In one specific embodiment, such as Figure 1 In The prompts included: "Please help me extract content related to quantitative research from this paper, including but not limited to the following: 1. Research hypothesis: ... 2. Experimental design: ... 3. Data analysis: ... 4. Expected results: ...

[0029] S3. Add the literature selected by the user from the multiple documents to the citation pool. This includes: constructing a structured list based on the metadata, abstract, and full text of the selected documents; creating structured list entries for multiple selected documents and displaying them to the user; receiving the user's selection instruction; and adding the structured list corresponding to the user-selected documents to the citation pool. This citation pool (Citation / Context Pool) is where important documents explicitly selected by the user are placed. The system records the source of the documents, the selection time, and their association with the research plan (e.g., association with a hypothesis or method module). This citation pool is added to the model context when generating quantitative research plans, when the user interacts with the large language model, and when iterating on the quantitative research plan. It can be understood that this citation pool specifically refers to the set of documents selected by the user as supporting evidence.

[0030] In one specific embodiment, such as Figure 2 As shown, Figure 2 In (a) the user enters the research background and objectives in the dialog box; Figure 2 (b) MethodMate recommends relevant literature and generates a list of references; Figure 2 (c) Users can view abstracts of papers of interest and detailed information on the quantitative research content of the paper extracted by MethodMate.

[0031] The metadata provided in specific embodiments of the present invention includes at least one or more of the following: title, author, publication year, publication source, and number of citations.

[0032] S4. In a specific embodiment of the present invention, the research background and objectives, and the model context are based on user input. The model context also includes the dialogue information between the user and the large language model and the reference pool, as well as the output format of the quantitative research plan, which are concatenated into a first prompt word. The concatenated first prompt word is input into the large language model to generate a quantitative research plan. The output format of the quantitative research plan includes multiple modules, which include research hypotheses, experimental design, data analysis, and expected results.

[0033] In one specific embodiment, such as Figure 1 In As shown, the first prompt is: "You are a research expert in the field of Human-Computer Interaction (HCI) with extensive experience in quantitative research methods. Please write a high-quality quantitative research proposal based on the provided research background, research objectives / research questions, and references. The proposal should be clearly structured, logically rigorous, and substantive, avoiding vague statements or jargon. Please strictly follow the structure below and provide a detailed description in each section: 1. Research Hypothesis…, 2. Experimental Design…, 3. Data Analysis…, 4. Results Presentation…." User prompt: The user's research topic is [Custom Topic] (research background, research subjects, and research questions). I will provide you with the following [paper Count] references: [paper Contents].

[0034] Based on the full text of the above references, please generate a detailed quantitative research proposal. The first prompt provided in this embodiment is specifically designed as follows: Input: user research background, user-described topic / problem, citation pool (list of references and key paragraphs), output format requirements (a module consisting of four parts).

[0035] Output format (strictly followed for easy parsing): 1. Research hypotheses (H1, H2..., with related literature listed after each hypothesis).

[0036] 2. Experimental design (subjects, design type, variable definition, materials / tasks, procedures, control items, sample size estimation / statistical power description).

[0037] 3. Data analysis (including subjective data, behavioral data, system record data, data preprocessing, statistical test methods, effect size, etc.).

[0038] 4. Expected results (for each hypothesis, specify the expected chart / table format).

[0039] The specific embodiments of the present invention also provide a traceability log: all generated versions save the names of the references cited at the time of generation and the specific paragraphs cited (if available), which facilitates user / supervisor traceability and review.

[0040] In one specific embodiment, this embodiment also provides a method for evaluating the generated quantitative research plan, including: concatenating the generated quantitative research plan, the research background and objectives input by the user, the model context, and the research plan evaluation template into a second prompt word; evaluating the generated quantitative research plan based on the second prompt word using a large language model to obtain an evaluation result and provide modification suggestions; the model context includes a citation pool and / or the historical dialogue between the user and the AI.

[0041] Specifically, the second prompt is designed as follows: Role-playing: Activate the role of a "rigorous HCI expert / reviewer," providing item-by-item evaluations and improvement suggestions from dimensions such as logical consistency, feasibility, rationality, and completeness. Output: Advantages and highlights of each dimension, item-by-item questions and suggestions, and an evaluation summary of the current module.

[0042] In one embodiment, the second prompt word is, for example... Figure 1 In As shown, the second prompt is: "System prompt: You are the most rigorous, discerning, and experienced HCI expert. Please review the current research plan from the perspectives of logic, rationality, feasibility, and need matching, and provide practical and specific improvement suggestions (avoid generalities). Suggestions should be concise, accurate, and sufficiently professional."

[0043] User prompt: Please systematically evaluate the following research proposal and analyze it from the following three aspects and user needs: 1. Logic: Evaluate the correspondence between research objectives, research hypotheses, evaluation indicators, etc.

[0044] 2. Reasonableness: Evaluate whether the various evaluation indicators, tools, and methods are effective and appropriate.

[0045] 3. Feasibility: Assess whether the user experiment's workload, time, and cost are feasible.

[0046] User Research Requirements Needs matching: Evaluate whether the solution fully meets the user's specific research needs and objectives.

[0047] Specific Research Plan Content: Research background and context: [Research Context], and finally, please summarize and point out the advantages and areas for improvement of this approach.

[0048] Specific embodiments of the present invention also provide a method for optimizing the generated quantitative research scheme or module, including: The generated quantitative research plan or its different modules, evaluation results, modification suggestions, and user-provided improvement goals are used to construct third-party prompts. Based on these third-party prompts, an optimized quantitative research plan or its modules are generated using a large language model, along with the modified content.

[0049] Specifically, the third prompt word is designed as follows: Input: the original quantitative research plan (or any sub-module of the plan's four modules), evaluation results, user-specified improvement goals (such as "reducing complexity", "increasing statistical power" or other detailed iterative aspects input by the user) or system-preset improvement directions (such as "adding more details", "adding detailed descriptions of statistical methods", "optimizing data processing flow").

[0050] Output: Optimized quantitative research protocol (maintaining the four-part structure). The "Protocol Comparison" module lists the content of each module before and after modification, facilitating comparison before and after iteration. Three modification types are indicated by color: red for the original content, green for the modified content, and gray for the unmodified content.

[0051] In one embodiment, the third prompt word is, for example, Figure 1 In As shown, "You are a senior research expert in the field of human-computer interaction. Your task is to optimize an existing research plan based on specific user feedback. Optimization philosophy: Targeted improvement: Focus optimization on selected parts of the research plan directly related to user suggestions. Format requirements: Please return the complete optimized plan in the following format, and the following precise title format must be used: User suggestions: Please optimize the [User-specified section] of the current research plan based on the following user suggestions and needs. User's specific optimization suggestions: [Suggestion]; Current research plan: [Proposal Content]; Focus areas: [Focus Areas]; User's research needs and objectives: [User's Objectives]; Research background and context: [Research Context]."

[0052] When a user initiates a request to "generate a research proposal" or "evaluate a research proposal", the system dynamically constructs a prompt and sends it to the LLM based on the current thread's citation pool, the user's self-described research background and preferences, and template requirements, ensuring that the model output can directly cite or be based on the argument structure of the cited literature.

[0053] In one specific embodiment, such as Figure 3 As shown, Figure 3 (a) in the diagram represents the dialogue between the user and the MethodMate model. The MethodMate model generates quantitative research proposals based on the user's research subjects and selected literature, such as... Figure 3 As shown in (b) of this embodiment, the MethodMate model provides support for multiple rounds of evaluation and iteration of the overall scheme and its four sub-modules, thereby achieving continuous optimization.

[0054] In summary, the modular design provided by the specific embodiments of this invention offers significant advantages. First, it ensures consistency in the output structure, standardizing the generated content and facilitating automatic parsing and export. This consistency not only improves system usability but also reduces the burden on users during subsequent data processing and organization. Second, modularity effectively reduces user prompting costs and supports multi-threaded parallel operation. Users can perform multiple iterations simultaneously on different modules or topics, unlike single-threaded modes which only allow one dialogue at a time, thus avoiding memory confusion caused by long dialogues. Furthermore, the templates contain rich details (such as sample characteristics, grouping methods, experimental procedures, subjective data, behavioral data, system-recorded data, etc.), enabling users to flexibly control the output, improving efficiency and operational flexibility. Overall, modularity not only enhances the system's operability and output quality but also strengthens users' sense of control in research design.

[0055] Modular iteration allows users to initiate iteration requests only for the "Experimental Design" or "Data Analysis" modules. The system runs the improved template within a local context, thereby saving computational resources and reducing unnecessary changes to the overall structure.

[0056] In one specific embodiment, this embodiment uses multiple threads to enable the generation of quantitative research schemes, the evaluation of quantitative research schemes, and the optimization of quantitative research schemes or modules within quantitative research schemes to be carried out simultaneously. Source introductions, method introductions, and statistical method queries can also be carried out simultaneously, and each thread has an independent reference pool and generation history.

[0057] The source information provided in the specific embodiments of this invention is integrated throughout the presentation of the generated solution. Its function is that when the AI ​​proposes a certain data analysis method or experimental design in the solution, the system will automatically locate and display the original text context of the method in the "citation pool" literature. This improves the traceability of the solution and lets users know that the method is not made up by the AI, but is supported by real literature.

[0058] The method described in the specific embodiments of this invention is introduced in the data analysis module, which serves to briefly explain the data analysis methods included in the research plan generated by AI, helping users to quickly master them.

[0059] The statistical method query provided in this specific embodiment of the invention is located in the data analysis module, and its function is to offer in-depth, encyclopedic statistical information. It covers the theoretical background, applicable conditions, statistical steps, and case studies of the statistical method. This allows beginners to avoid having to leave the system to consult other resources when encountering unfamiliar statistical terms (such as T-tests and ANOVA), thus supporting them in making informed decisions.

[0060] In the multi-threaded method provided in this invention embodiment, when designing a quantitative research plan, users often do not create it all at once, but need to iterate and optimize different modules (such as research hypotheses, experimental design, data analysis, etc.) multiple times. The system adopts a "multi-threaded" architecture, which means that users can simultaneously open multiple exploration paths (e.g., thread A attempts "plan evaluation", thread B attempts "method introduction"). In each thread, the system will separately record the literature selected by the user under that path (an independent citation pool) and the record of modifications made in dialogue with the AI ​​(an independent generation history). The advantages of the above design are: 1. Avoiding context pollution: Traditional large model dialogues (such as directly using ChatGPT) are single-threaded, and all modification instructions and cited literature are piled up in the same dialog box, which can easily lead to "memory confusion" of the model, mixing abandoned ideas with subsequent requirements and creating illusions. Independent citation pools and generation history ensure absolute isolation between different exploration directions. 2. Supporting divergent exploration: Allowing novice researchers to safely conduct multi-directional "trial and error". They can pull different literature and generate several distinct research plans for the same research objective without worrying about disrupting the original plan. 3. Visually Presenting Differences to Aid Decision-Making: When users generate different versions of the plan through multiple threads (or iterate on a module multiple times), it's difficult to evaluate which version is better simply by remembering. The history function allows users to visually compare the specific modifications between different versions, helping beginners see more clearly what was added and deleted. 4. Complementary Strengths and Merging the Best: Different threads may perform better in different modules (for example, thread A has a more rigorous experimental design, while thread B's data analysis methods are more suitable). Merging mainly corresponds to users being able to extract the best modules from each thread, like "building blocks," and integrate them into a final, most satisfactory and rigorous quantitative research plan.

[0061] The quantitative research protocol generated by this invention comprises four modules: research hypothesis, experimental design, data analysis, and expected results, providing a clear and rigorous structure. The final exported format is standardized, including PDF and TXT file formats, facilitating user reading and discussion with supervisors.

[0062] On the other hand, specific embodiments of the present invention also propose a quantitative research protocol design system for novice HCI researchers—MethodMate. Based on LLM, this system combines a retrieval, generation, and evaluation closed loop to automatically output structured quantitative research protocols containing research hypotheses, experimental designs, data analysis, and expected results. Existing technologies generally suffer from the following shortcomings: (1) they fail to provide structured support covering the entire quantitative research process; (2) they lack evidence traceability and literature injection mechanisms, making it difficult to guarantee the rigor of the generated protocols; and (3) they do not adequately respond to the needs of novice researchers in methodological argumentation and cross-task collaboration.

[0063] To address these shortcomings, specific embodiments of this invention focus on the challenges and needs faced by novice HCI researchers throughout the entire process of devising quantitative research protocols, proposing the following technical solutions: A two-stage retrieval and injection of original text fragments is employed to avoid semantic biases inherent in vectorization methods, thereby ensuring that the generated protocol possesses a complete chain of arguments and verifiability, solving the problem of lack of traceable evidence in existing methods. Furthermore, through a temporary context pool and a "cite" mechanism, users can selectively manage cited literature, and the model prioritizes the use of confirmed evidence during generation and iteration, enhancing the reliability of the protocol.

[0064] Furthermore, specific embodiments of this invention provide users with structured outputs and actionable modification suggestions through three types of prompt templates: generation, evaluation, and iteration, combined with an "expert role" evaluation mechanism. This solves the problems of disorganized solution structures and insufficient methodological justification for beginners. Moreover, the multi-task coordination and dynamic context management mechanism ensures smooth switching between different tasks and consistency of results.

[0065] To facilitate novice researchers in completing their research design process step-by-step, this invention provides a six-step workflow guidance interaction within the system. In summary, this invention significantly lowers the learning and design threshold for novice researchers through traceable evidence, structured output, and automated evaluation iterations, enabling the generation of high-quality, verifiable quantitative research protocols.

[0066] A specific embodiment of the present invention also provides a system for conceiving quantitative research schemes based on large language models, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they are used to implement the method for conceiving quantitative research schemes based on large language models.

[0067] A specific embodiment of the present invention also provides a system for conceiving quantitative research schemes based on a large language model, namely the MethodMate system, which includes the following main modules: (1) an academic literature retrieval module, which adopts a two-stage retrieval and full-text injection mechanism; (2) a contextual citation pool mechanism to ensure the traceability of research schemes; (3) a template generation, evaluation and iteration module, which provides structured output that conforms to academic norms; and (4) a multi-threaded and modular interaction module, which supports users to switch and optimize between multiple research ideas. The above modules operate collaboratively under a front-end and back-end separation architecture, which can realize a closed-loop process from clarifying research questions and providing literature support to generating and optimizing quantitative research schemes.

[0068] Quantitative research is a core method for ensuring transparency and reproducibility in the field of HCI (Hybrid Clinical Research). It encompasses hypothesis formulation, experimental design, sample size estimation, statistical testing, and results presentation, demanding a high level of standardization and logical consistency. High-quality research protocols not only promote academic output but also play a crucial role in research training and mentorship. However, novice researchers, lacking systematic domain knowledge and methodological training, often don't know how to begin quantitative research and struggle to identify potential problems in their protocols in a timely manner.

[0069] Traditional methods relying on literature retrieval and manual writing have limited efficiency. While general-purpose large language models can generate text, they often suffer from issues such as outdated knowledge, semantic drift, and insufficient academic traceability, making it difficult to meet the professional needs of quantitative research. Existing auxiliary tools also often lack structured output and systematic evaluation mechanisms for quantitative research. By combining the generation capabilities of large language models with real-time updates to academic databases, a workflow system integrating retrieval, generation, and evaluation can be established. Through engineered prompt templates and context management mechanisms, not only can the output be ensured to conform to academic norms, but sources can also be clearly cited in the research proposal, thereby improving the reliability and traceability of the results.

[0070] This invention proposes a six-step workflow for novice HCI researchers and uses it as the core basis for the design and interaction of the MethodMate system. The workflow includes: retrieving relevant literature, focusing on the quantitative research portion of the literature, designing a quantitative research plan, evaluating the plan, iteratively optimizing the plan, and finalizing and confirming the quantitative research plan. The MethodMate system proposed in this invention adopts a closed-loop architecture of "retrieval-generation-evaluation." Its features include: (1) two-stage academic retrieval and full-text injection to ensure knowledge updates and semantic accuracy; (2) a citation pool mechanism to achieve traceability of generated plans; (3) templated research plan generation, evaluation, and iteration to ensure structured and academically sound output; and (4) multi-threaded and modular interaction to reduce cognitive burden and support plan optimization. This system can significantly lower the threshold for novices to enter quantitative research and improve the academic usability and credibility of plans. At the same time, it provides a new technical path for education and training and scientific research practice, and also provides a generalizable design idea for the future development of LLM-based disciplinary tools.

[0071] The MethodMate system provided in this specific embodiment of the invention is a web-based workflow guidance system, employing a front-end and back-end separation architecture: Frontend: Developed based on the Vue.js technology stack, it provides modular functional components covering the entire research process, including a literature search panel, citation pool, quantitative research protocol canvas, analysis and evaluation panel, multi-threaded view and iteration history, supporting visual operation and flexible interaction.

[0072] Backend: The server is built with Node.js and provides efficient and standardized data interaction capabilities through a RESTful API (Representative State Transfer Application Programming Interface). The data layer adopts a dual architecture design of "main database + cache database". It uses a MySQL relational database to store core resources such as recommended literature, research plans and statistical methods, while combining it with a Redis in-memory key-value database to achieve fast reading of high-frequency data and accelerated session response, thereby improving the overall system performance.

[0073] Model layer: This layer calls external LLMs and local / cloud microservice models (which can be replaced with other LLMs that conform to the specified interfaces) to parse user queries, generate suggestions, and provide structured output. The system also connects to academic APIs (such as OpenAlex and CORE) for literature retrieval and full-text crawling.

[0074] Multi-task coordination and dynamic context manager: responsible for passing and maintaining the citation pool (user-selected references), session history, template state, and parallel thread data between modules.

[0075] Deployment and Security: The system supports local deployment on private clouds or on-campus servers, ensuring the privacy and security of scientific research data from the source of data storage, and complying with academic research data management standards.

[0076] Specific Embodiment 1 provided in this invention is a cloud-based SaaS deployment (Web system). Application scenarios and objectives: This service provides an online MethodMate platform for desktop / browser users (novice HCI researchers) in universities and research institutions, supporting an end-to-end workflow from literature retrieval to finalizing quantitative research protocols. Hardware and operating environment (recommended configuration): Frontend: Vue.js 3 + Vite. Backend: Node.js 18, Docker containerized deployment; application server: 4 vCPUs, 16GB RAM; independent deployment of models or API gateways, GPU (NVIDIA T4 / A10) required for local inference; storage: MySQL ≥ 8.0 (structured data), Redis 6 (caching), object storage for PDF / full-text fragments. Third-party services and dependencies: OpenAI or institutionally licensed LLM providers (for generation), OpenAlex and CORE (for academic retrieval and full-text crawling), optional institutional subscriptions (Scopus / Web of Science). Example of key parameters: LLM call: model=GPT-4o; temperature=0.6~0.7; max_tokens=16384; Literature injection: Inject 1,000–2,000 tokens of method or result paragraphs per article; Caching strategy: prompt cache for 24 hours; LLM request rate limit according to service provider quota (example: 60 rpm).

[0077] Program flow (chronological order): S1. Users create a new worker thread and submit research background and research objectives, etc.

[0078] S2. The system calls the LLM parser to extract search keywords and preferences.

[0079] S3. Call OpenAlex to perform a Boolean search based on keywords and use CORE to retrieve the publicly available full text (splittered into paragraphs and stored in fulltext_snippets).

[0080] S4. Users select documents to add to the citation pool; the system records citation metadata (doi, title, author information, publication source, etc.).

[0081] S5. Generate Request: The backend concatenates the user background, reference pool paragraphs, and generated template into a prompt, and calls LLM to generate a structured solution (JSON).

[0082] S6. Save the solution as a new version and display it on the front end.

[0083] S7. User requests evaluation: The system, in the "expert role," invokes the LLM to generate and save an evaluation report.

[0084] S8. Initiate an iteration: The user specifies a module (such as experimental design), and the system generates a new version based on the iteration template and reference pool and records the diff.

[0085] S9. Finalize and export (TXT / PDF).

[0086] Specific Implementation Example 2 – Private Deployment on Institutional Intranet (Offline LLM) Application scenarios and objectives: For research institutions or enterprises with confidentiality requirements for data and models, all components (retrieval, indexing, LLM inference) are deployed on the internal network to avoid external network access.

[0087] Hardware and operating environment (recommended configuration): Local inference cluster: 4–8 nodes (each node equipped with an NVIDIA A100 80GB or equivalent GPU); Kubernetes management, Ceph / S3 storage, PostgreSQL HA cluster.

[0088] Models and retrieval stack: Local LLMs (such as Llama2 / 3, Falcon, etc., subject to license) employ quantization and parallel inference; The system employs a multi-layered hybrid retrieval architecture. First, AI services are used to extract information and keywords related to user research. Then, the OpenAlex API is invoked for external retrieval. Simultaneously, intelligent domain filtering (focusing on fields such as HCI and computer science) and top journal screening mechanisms are integrated. The retrieved papers undergo full-text parsing and structured extraction of research methods through the AI ​​(ChatGPT) service. Finally, the papers are ranked and recommended based on a relevance scoring algorithm (considering factors such as title matching degree, citation count, and journal quality). The entire process uses session-level paper pool management to achieve efficient deduplication and intelligent reuse.

[0089] Example of key parameters: Inference: temperature = 0.6~0.7; max_new_tokens = 16383; context_window = 65k (if the model supports it) Program flow (chronological order): S1. The administrator imports the organization's full-text library and performs paragraph-level splitting and vectorization; S2. The user submits the research objective and triggers local LLM extraction of search elements; S3. The system performs a mixed search and returns relevant paragraphs with metadata; S4. The user adds the selected paragraph to the citation pool; S5. The user initiates "Generate Quantitative Research Plan". The system generates a structured plan based on the local LLM and the injected paragraph and saves the version. S6. Evaluation and iteration are performed locally, and the system clearly marks the location of the cited paragraphs in the evaluation report; S7. The system saves audit logs (generating prompts, model parameters, and reference IDs) for compliance review.

[0090] Additional implementation details: Credibility filtering before citation injection (injecting only paragraphs from journals / conferences or with a similarity threshold ≥0.75); Access control and encrypted storage of reference pool entries.

[0091] Specific Implementation Example 3 – Typical User Scenarios: Suppose Lucy is a first-year master's student specializing in human-computer interaction, developing a system based on a large language model to support quantitative research. She has clearly defined her research background and goals and built a preliminary prototype, but is encountering difficulties in designing a quantitative research protocol. A lack of systematic statistical training makes her uneasy about experimental design. She tries searching for relevant papers, but the process is time-consuming and makes it difficult to choose suitable experimental protocols. She also struggles to understand terms such as t-tests. The results generated by some general-purpose LLM tools she has used lack professionalism and feasibility, further discouraging her confidence.

[0092] In this situation, Lucy switched to MethodMate. She followed the six-step workflow provided by the system: (1) Searching for relevant literature: Lucy describes the research background and purpose in the dialog box, and MethodMate recommends several keywords and returns a list of relevant papers; (2) Focus on the quantitative research section in the literature: She clicks on a highly relevant paper, and MethodMate automatically extracts the quantitative research section of the paper (such as research hypothesis, experimental design, data analysis, expected results) and displays the key points; (3) Design a quantitative research plan: Lucy adds the paper to the citation pool and submits the research objectives. The system generates a structured draft containing "research hypothesis - experimental design - data analysis - expected results" based on the cited literature and user background. (4) Evaluate quantitative research protocols: Lucy uses the “Protocol Evaluation” function, where the system evaluates each protocol item by item in the role of an expert and points out potential problems in logic and methodology; (5) Iterative refinement: She initiates iterations for specific modules based on the evaluation feedback. MethodMate generates updated versions and lists the changes while preserving the reference history. (6) Finalization and confirmation: After several rounds of evaluation and iterative cycles, Lucy derived the final plan as a formal research plan to be discussed with her supervisor.

[0093] Throughout the process, the structured guidance provided by MethodMate helped Lucy discover and correct several initially difficult-to-detect problems (such as sample size estimation, control variable settings, and the selection of statistical tests), and quickly develop a professional and feasible quantitative research plan.

[0094] The system proposed in this invention application has undergone multiple rounds of usability testing and comparative experiments in practical applications, achieving significant results and fully demonstrating its technical value and innovation.

[0095] Firstly, regarding usability, user research validated the system's ease of use and rationality. In the first usability test, the average System Usability Score (SUS) of 6 users reached 87.50, with a 100% task completion rate and no additional requests for assistance, indicating that the system has high usability and complete functional accessibility. After iterative optimization, the average score of 16 users in the second test further improved to 91.89, demonstrating that the technical solution of the specific embodiment of this invention can provide significant support for novice researchers in terms of workflow design, functional effectiveness, and interaction rationality. User feedback indicated that the six-step workflow provided by the system is comprehensive and reasonable, especially the literature retrieval, quantitative research plan generation, and iterative optimization functions, which effectively lower the learning threshold.

[0096] Secondly, in the comparative experiment, our system was systematically compared with a recently developed free-form conversational large language model (the benchmark system). The experimental results show that the research proposals generated by our system are significantly superior to the benchmark system in terms of content quality (logic, feasibility, credibility, completeness, and reproducibility) and presentation quality (clarity of expression, structural completeness, and richness of detail). Statistical tests show that the differences in multiple indicators are significant (see details). Figure 4The solutions output by this system are logically clear, structurally complete, and conform to common academic research formats, helping beginners get started quickly. In contrast, benchmark systems often suffer from logical jumps and insufficient feasibility.

[0097] Furthermore, regarding user subjective experience, this system performs well in terms of usability (usefulness, ease of use, and satisfaction); see [link to relevant documentation]. Figure 5 User experience (all six dimensions: attractiveness, clarity, efficiency, credibility, facilitation, and novelty; see...) Figure 6 ) and cognitive load (all six dimensions: mental workload, physical workload, time workload, performance, effort level, and frustration level; see Figure 7 The system significantly outperformed the benchmark system in all aspects. Participants generally agreed that the system, through workflow guidance, literature tracing, and iterative optimization functions, enabled them to achieve greater efficiency, a stronger sense of control, and higher levels of trust during the design process. Particularly in the experimental design phase, the system's "scheme evaluation" and "scheme iteration" functions provided users with opportunities for continuous optimization, significantly improving the quality of research protocols.

[0098] Finally, in terms of functional effectiveness, users unanimously agreed that the system's three functions—"quantitative research protocol extraction," "source introduction," and "literature recommendation"—are the most crucial, significantly improving their efficiency in understanding and utilizing academic literature (see [link to relevant documentation]). Figure 8 By incorporating a traceability mechanism and a structured six-step workflow, this system helps users complete tasks in a short time.

[0099] The design of a high-quality, verifiable quantitative research scheme addresses the shortcomings of existing technologies in areas such as insufficient evidence traceability, lack of structured support, and imperfect interactive guidance.

[0100] In summary, empirical studies of the specific embodiments of this invention demonstrate that its technical solution not only excels in usability but also significantly outperforms existing methods in terms of solution quality, user experience, and cognitive load. It effectively reduces the learning and design threshold for novice researchers and possesses outstanding technological advancement and application value.

[0101] (The English abbreviations and terms mentioned below are provided with their full English names and Chinese translations, or detailed explanations of the Chinese technical terms, such as: SCIM: a text input platform on the Linux operating system.) MethodMate: The name of the system in a specific embodiment of the present invention, a workflow guidance system based on a large language model, designed to help novice HCI researchers design high-quality quantitative research protocols.

[0102] LLM (Large Language Model): A large-scale natural language processing model based on deep learning (such as GPT-4o, GPT-4 series, etc.), capable of performing tasks such as text generation, reasoning, and information integration.

[0103] HCI (Human-Computer Interaction): The discipline of human-computer interaction studies the interaction behaviors and experiences between humans and computer systems.

[0104] RAG (Retrieval-Augmented Generation): This method enhances the accuracy and timeliness of responses by combining external knowledge base retrieval results with the output of a generative model.

[0105] OpenAlex: An open academic literature database and API that supports academic retrieval based on metadata and keywords.

[0106] CORE: An open-access academic full-text repository that provides full-text download interfaces across journals and conferences.

[0107] Workflow Guidance: A way to guide users through complex tasks by using predefined step-by-step processes and interactive modules, reducing confusion during free exploration.

[0108] Multi-threaded: Allows users to develop multiple research ideas in parallel, with each idea forming an independent thread, supporting parallel comparison and merging.

[0109] Citation / Context Pool: A temporary collection of user-selected references used in the system. This pool is forcibly injected during subsequent generation and evaluation to ensure traceability and consistency.

[0110] Templated Prompts: A predefined framework of prompts to ensure that the model-generated results conform to academic standards and quantitative research format requirements, and to support automatic parsing and subsequent processing.

Claims

1. A method for conceiving quantitative research schemes based on large language models, characterized in that, include: The research background and objectives input by the user are parsed by a large language model. Boolean search conditions are constructed based on the parsed elements. Multiple documents are retrieved through the academic index API based on the Boolean search conditions. The full text fragments of each retrieved document are extracted to extract the corresponding research plan through the original text injection method. The documents selected by the user from the multiple documents are added to the citation pool. The citation pool and the research plan are added to the model context in the set output format. The research background and objectives input by the user, the model context, and the output format of the quantitative research plan are concatenated into the first prompt word. Based on the first prompt word, the quantitative research plan is generated through a large language model.

2. The method for conceiving a quantitative research scheme based on a large language model according to claim 1, characterized in that, The generated quantitative research plan, the user-input research background and objectives, the model context, and the research plan evaluation template are concatenated into a second prompt word. Based on the second prompt word, the generated quantitative research plan is evaluated using a large language model to obtain the evaluation results and provide modification suggestions.

3. The method for devising a quantitative research scheme based on a large language model according to claim 2, characterized in that, The generated quantitative research plan or different modules in the quantitative research plan, evaluation results, modification suggestions and improvement goals given by the user are used to construct a third prompt word. Based on the third prompt word, an optimized quantitative research plan or optimized module is generated through a large language model, and the modified content is marked. The quantitative research plan includes research hypothesis, experimental design, data analysis and expected results.

4. The method for devising a quantitative research scheme based on a large language model according to claim 3, characterized in that, By using multiple threads, the generation, evaluation, and optimization of quantitative research schemes or their internal modules can be carried out simultaneously, with each thread having its own independent reference pool and generation history.

5. The method for devising a quantitative research scheme based on a large language model according to claim 1, characterized in that, The research background and objectives input by the user are parsed using a large language model. The parsed elements are structured query elements. The information input by the user includes keywords, research interests, target variables, and method preferences.

6. The method for devising a quantitative research scheme based on a large language model according to claim 1, characterized in that, Multiple documents were retrieved using the Academic Index API based on Boolean search criteria, including: The system matches documents based on their title, abstract, keywords, and journal, and filters them by publication year, citation count, and field to select documents that meet accessibility criteria.

7. The method for devising a quantitative research scheme based on a large language model according to claim 1, characterized in that, The full-text fragments of each retrieved document are added to the model context in a predefined output format using the original text injection method, including: The full-text fragments of the selected literature are obtained by calling CORE or other full-text libraries. The full-text fragments are divided into blocks, and the research plan is extracted from each block according to the set output format. The research plans of each block are integrated to obtain the research plan of the selected literature.

8. The method for devising a quantitative research scheme based on a large language model according to claim 1, characterized in that, Add the references selected by the user from the aforementioned multiple references to the citation pool, including: A structured list is constructed based on the metadata, abstract, and full text of the selected documents. Multiple selected documents are compiled into structured list entries and displayed to the user. The system receives the user's selection instructions and adds the structured list corresponding to the user's selected documents to the citation pool.

9. A system for conceiving quantitative research schemes based on large language models, characterized in that, The method includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement the method for conceiving a quantitative research scheme based on a large language model as described in any one of claims 1-8.

Images