Method and equipment for constructing large ethical examination model based on instruction set optimization

Abstract

The invention belongs to the field of intelligent medical treatment, and particularly relates to an ethical examination large model construction method and equipment based on instruction set optimization. The method comprises the following steps: acquiring ethical examination knowledge points and an artificially constructed instruction set corresponding to each knowledge point; inputting the knowledge points and the corresponding manually constructed instruction sets into a generative large language model to generate a specified number of simulation instruction sets; the simulation instruction set is subjected to quality evaluation to obtain a quality score, if the quality score is higher than a threshold value, it is considered that the simulation instruction set meets the requirement, and otherwise, it is considered that the simulation instruction set does not meet the requirement; and combining an artificially constructed instruction set and a simulation instruction set meeting requirements to serve as a training instruction set, and inputting the training instruction set into the base model of the large language model for fine tuning training to obtain an ethical examination large model. According to the method, the accurate instruction set suitable for ethical examination is constructed in a quality evaluation and detection mode, so that an accurate result is given when the fine-tuned model is applied to ethical examination.

Description

Technical Field

[0001] The present invention relates to the field of intelligent healthcare, and more particularly, to a method, device, medium, and program product for constructing an ethics review large model optimized based on an instruction set. Background Art

[0002] Ethics review is the main measure to protect the safety and rights of human research subjects, and is also a necessary condition for the standard development of biomedical research involving humans. With the continuous increase in investment in clinical research technological innovation and the development of biotechnology, the improvement of the quality and efficiency of ethics review is an urgent need for the development of clinical research. The construction of ethics review committees in China started relatively late, and the ethics review capabilities of each ethics committee vary, and the review quality is uneven, which greatly restricts the improvement of ethics review efficiency.

[0003] Large language models and their applications in the field of artificial intelligence have become a global research hotspot in science and technology, with the number of parameters increasing from more than one billion to one trillion, significantly improving the ability to capture and understand human language. The success of large language models in multiple fields has also brought research opportunities for developing ethics review assistance systems based on large language models, and then assisting ethics committees with relatively less ethics review experience and training to improve the quality and efficiency of ethics review.

[0004] The patent document "CN202311465845.3" discloses a method and system for assisting decision-making in scientific research ethics review based on association rule mining. The method provides a method and system for assisting decision-making in scientific research ethics review based on association rule mining. The method constructs a data model, uses an association mining algorithm to mine association rules, and compares the current scientific research project data with the mined rules to obtain an auxiliary review conclusion. However, when comparing the method and system for assisting decision-making in scientific research ethics review based on association rule mining with the ethics review technology based on large models, its technical drawbacks may be reflected in the following aspects: (1) Insufficient text understanding and context analysis capabilities: The method based on association rule mining mainly relies on keyword or phrase matching to locate possible ethical issues, which limits its in-depth understanding of complex text contexts; (2) Handling complexity and flexibility: The method based on association rule mining mainly relies on predefined association rules and mining algorithms, and may have certain limitations in dealing with complex and changing scientific research ethics review scenarios. When scientific research ethics issues involve multiple levels, multiple dimensions, or emerging fields, this method may be difficult to comprehensively cover and accurately judge. (3) Data dependence and training costs: Constructing an association rule mining data model requires a large amount of historical data and expert knowledge. If the data quality is not high or there are biases, it may lead to inaccurate mined association rules, thereby affecting the conclusion of ethics review. Summary of the Invention

[0005] In view of the above problems, the present invention provides a method for constructing an ethical review large model based on instruction set optimization, which improves the text understanding and early warning analysis capabilities of the model in the field of ethical review by improving the construction of the instruction set, and is more flexible in complex text processing.

[0006] This application (the first aspect) discloses a method for constructing an ethical review large model based on instruction set optimization, including:

[0007] Obtain the knowledge points of ethical review and the manually constructed instruction sets corresponding to each knowledge point;

[0008] Input the knowledge points and the corresponding manually constructed instruction sets into a generative large language model to generate a specified number of simulated instruction sets, where the content of the simulated instruction sets is different from that of the manually constructed instruction sets but the form is similar;

[0009] Perform quality assessment on the simulated instruction sets to obtain a quality score. If the quality score is higher than the threshold, it is considered that the simulated instruction sets meet the requirements; otherwise, it is considered that the simulated instruction sets do not meet the requirements;

[0010] Merge the manually constructed instruction sets and the simulated instruction sets that meet the requirements as the training instruction sets, and input the training instruction sets into the base model of the large language model for fine-tuning training to obtain an ethical review large model.

[0011] Furthermore, the method for performing quality assessment on the simulated instruction sets to obtain a quality score is as follows: different knowledge points and different instructions have different weights, and adjustment coefficients are used to balance the weights of knowledge points and instructions, and the quality score is calculated;

[0012] Optionally, the calculation method of the quality score is expressed as:

[0013]

[0014] Among them, Q represents the quality score of the instruction set, W is the total number of knowledge points, I is the total number of simulated instructions, z(w) and z(i) are the basic weights of knowledge point w and instruction i respectively; y(w) and y(i) are the application weights of knowledge point w and instruction i respectively; k1 and k2 are adjustment coefficients used to balance the influence of different weights.

[0015] Furthermore, the basic weights of knowledge point w and instruction i are determined according to the importance and frequency of the knowledge points in medical ethical review;

[0016] Optionally, the method for obtaining the basic weights of knowledge point w and instruction i is as follows:

[0017] Step 1: Use a clustering algorithm to group the knowledge points and instructions of the simulated instruction sets;

[0018] Step 2: Automatically extract the importance of knowledge points and instructions in each group using text mining techniques;

[0019] Among them, the basic weight calculation formula is expressed as:

[0020] z(w) = α·ftype(w) + β·ffrequency(w) + γ·fimpact(w)

[0021] z(i) = δ·ftype(i) + ∈·ffrequency(i) + ζ·fdifficulty(i)

[0022] Among them, ftype(w) represents the type of knowledge point, ffrequency(w) represents the frequency of the knowledge point appearing in the group; fimpact(w) represents the degree of influence of the knowledge point on the review result, which can be quantified by the relevance between the review result and the knowledge point; ftype(i) represents the type of knowledge point to which the instruction belongs, and fdifficulty(i) represents the execution difficulty of instruction i; α, β, γ, δ, ∈, ζ are parameters determined by data-driven methods;

[0023] Optionally, the method for obtaining the application weights of knowledge point w and instruction i is as follows:

[0024] Step 1, apply regression analysis to evaluate the specific impact of each knowledge point and instruction on the review result;

[0025] Step 2, use the method of feature selection to select knowledge points and instructions that have a significant impact on the weight;

[0026] Among them, the calculation formula for the application weight is expressed as:

[0027] y(w) = η·fimpact(w) + θ·frelevance(w)

[0028] y(i) = ι·frelevance(i) + κ·fdifficulty(i)

[0029] Among them, fimpact(w) represents the degree of influence of the knowledge point on the review result; fdifficulty(i): the difficulty of executing the instruction; frelevance(i) represents the relevance between the instruction and the review task; η, θ, ι, κ are parameters determined by data-driven methods.

[0030] Further, the method further includes: mixing an artificially constructed instruction set and an instruction set that meets the requirements to obtain a mixed instruction set, selecting a partial instruction set from the mixed instruction set and inputting it into a large language model to generate a specified number of new simulated instruction sets, performing quality evaluation on the new simulated instruction sets to obtain a quality score, and if the quality score is higher than the threshold, it is considered that the new simulated instruction sets meet the requirements, otherwise it is considered that the new simulated instruction sets do not meet the requirements;

[0031] Optionally, when the simulated instruction sets do not meet the requirements, regenerate replacement simulated instruction sets, perform quality evaluation on the replacement simulated instruction sets to obtain a quality score, and if the quality score is higher than the threshold, it is considered that the replacement simulated instruction sets meet the requirements, otherwise it is considered that the replacement simulated instruction sets do not meet the requirements.

[0032] Further, the method further includes: if the number of the training instruction sets does not meet the requirements, continuously generate new simulated instruction sets, and merge the instruction sets with qualified quality evaluation into the mixed instruction set until the training instruction sets obtained after merging the artificially constructed instruction sets and the simulated instruction sets that meet the requirements meet the requirements.

[0033] Further, perform a coverage range score on the instructions in the training instruction sets. If the coverage range score does not meet the requirements, generate targeted simulated instruction sets for the knowledge points lacking coverage until the coverage range score meets the requirements;

[0034] Optionally, the coverage range score is obtained based on the instruction set and the range corresponding to the instructions;

[0035] Optionally, the coverage range score is expressed as:

[0036]

[0037] where C represents the coverage range score of the instruction set, I is the total number of instructions in the instruction set, c(i) is the coverage score of the i-th instruction; S is a normalization factor;

[0038] Optionally, the coverage score of the i-th instruction depends on the types of knowledge points covered by the instruction.

[0039] Further, the generative large language model includes one or more of the following: ChatGPT, GPT4;

[0040] The base model of the large language model includes one or more of the following: ChatGLM2-6B, VIP5, VisualGLM-6B, Mistral-Large-Instruct-2407.

[0041] A method for ethical review based on instruction set optimization, the method includes:

[0042] Obtain the text to be reviewed;

[0043] Input the text to be reviewed into any of the above-mentioned ethical review large language models to obtain an ethical review result.

[0044] The second aspect of this application discloses a construction system for an ethical review large language model optimized based on an instruction set, including:

[0045] An acquisition module 201: Acquire the knowledge points of ethical review and the manually constructed instruction sets corresponding to each knowledge point;

[0046] An instruction set construction module 202: Input the knowledge points and the corresponding manually constructed instruction sets into a generative large language model to generate a specified number of simulated instruction sets, where the content of the simulated instruction sets is different from that of the manually constructed instruction sets but the form is similar;

[0047] An instruction set evaluation and update module 203: Perform quality evaluation on the simulated instruction sets to obtain a quality score. If the quality score is higher than the threshold, it is considered that the simulated instruction sets meet the requirements; otherwise, it is considered that the simulated instruction sets do not meet the requirements;

[0048] A model fine-tuning training module 204: Combine the manually constructed instruction sets and the simulated instruction sets that meet the requirements as training instruction sets, and input the training instruction sets into the base model of the large language model for fine-tuning training to obtain an ethical review large language model.

[0049] The third aspect of this application discloses a computer device, which includes: a memory and a processor; the memory is used to store program instructions; the processor is used to call the program instructions, and when the program instructions are executed, it is used to execute the steps of the above method.

[0050] The fourth aspect of this application discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the above method.

[0051] The fifth aspect of this application discloses a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the steps of the above method.

[0052] This application has the following beneficial effects:

[0053] (1) For the fine-tuning of the large language model, it is necessary to accurately construct the instruction sets in this technical field. This application constructs accurate instruction sets suitable for ethical review through quality evaluation and detection, so that the model after fine-tuning gives accurate results when applied to ethical review;

[0054] (2) Applying the trained large language model for ethical review in this application to ethical review improves the efficiency and accuracy of ethical review: By using large language model technology, the accuracy of ethical review is improved, misjudgments and missed judgments are reduced, and the reliability and effectiveness of the review results are ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following described drawings are only some embodiments of the present invention. For those skilled in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0056] Figure 1 is a schematic flowchart of the method provided in the first aspect of the embodiments of the present invention;

[0057] Figure 2 is a schematic diagram of the program product provided in the second aspect of the embodiments of the present invention;

[0058] Figure 3 is a schematic diagram of the computer device provided in the embodiments of the present invention;

[0059] Figure 4 is a schematic diagram of the architecture of an exemplary computing device provided in the embodiments of the present invention;

[0060] Figure 5 is a schematic diagram of the storage medium provided in the embodiments of the present invention;

[0061] Figure 6 is a schematic flowchart of a process for conducting ethical review based on an ethical review model provided in the embodiments of the present invention;

[0062] Figure 7 is a construction process of an instruction set provided in the embodiments of the present invention;

[0063] Figure 8 is a schematic diagram of a model fine-tuning process provided in the embodiments of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0064] To enable those skilled in the art to better understand the solutions of the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention.

[0065] In some processes described in the specification, claims, and the above-mentioned drawings of the present invention, a plurality of operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order in which they appear herein or may be executed in parallel. The serial numbers of the operations, such as S101, S102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions such as "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent a sequence, and do not limit that "first" and "second" are of different types.

[0066] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present invention.

[0067] Figure 1 It is a schematic flow chart of a method based on evaluating the PATENTNAME method provided by an embodiment of the present invention. Specifically, the method includes the following steps:

[0068] S101: Obtain the knowledge points of ethical review and the artificially constructed instruction sets corresponding to each knowledge point;

[0069] S102: Input the knowledge points and the corresponding artificially constructed instruction sets into a generative large language model to generate a specified number of simulated instruction sets, and the content of the simulated instruction sets is different from that of the artificially constructed instruction sets but similar in form;

[0070] S103: Perform quality evaluation on the simulated instruction sets to obtain a quality score. If the quality score is higher than the threshold, it is considered that the simulated instruction sets meet the requirements, otherwise it is considered that the simulated instruction sets do not meet the requirements;

[0071] S104: Combine the artificially constructed instruction sets and the simulated instruction sets that meet the requirements as the training instruction sets, and input the training instruction sets into the base model of the large language model for fine-tuning training to obtain an ethical review large model.

[0072] The ethical review assistance system based on large language models is a solution integrating advanced artificial intelligence technologies, designed specifically to improve the efficiency, accuracy, and consistency of the ethical review process. This system deeply integrates the expertise of ethical review with the natural language processing capabilities of large language models, achieving full automation from data processing to the generation of review conclusions. It mainly consists of two major components: the ethical review model and the ethical review application. The overall framework of the system is as shown in Figure 6 shown.

[0073] The steps to establish the ethical review model include:

[0074] I. Data Processing

[0075] Conduct a comprehensive cleaning of the input ethical review documents, which includes removing redundant information such as irrelevant text, pictures, tables, etc., as well as correcting spelling mistakes and grammar errors in the text; perform noise reduction processing, identifying and eliminating noise data that is not substantially helpful for ethical review through algorithms; standardize the documents, unifying formats, encodings, and units.

[0076] II. Knowledge Point Rule Extraction

[0077] In the process of ethical review, the accuracy and integrity of knowledge point rules directly affect the reliability of review results. Therefore, the system automatically extracts key ethical rules and review key points from a large number of ethical review cases, regulatory documents, and industry standards through advanced machine learning algorithms.

[0078] These rules not only cover basic ethical principles such as respect, justice, benefit-risk balance, etc., but also are refined for specific types of scientific research projects, subject groups, and ethical issues.

[0079] Through this step, the system constructs a comprehensive and detailed ethical rule library, providing rich knowledge resources for model training and ensuring that the model can accurately understand and apply ethical rules.

[0080] The methods and steps of rule extraction are as follows:

[0081] (1) Preparation stage of knowledge point rule extraction

[0082] Form an expert team: Two secretaries of the ethics committee with more than 10 years of ethical review experience are responsible for building the framework of the informed consent form and the elements that must be included in each part.

[0083] Collect regulatory documents: Sort out and analyze domestic and foreign laws, regulations, and industry standards on medical ethical review.

[0084] (2) Implementation stage of knowledge point rule extraction

[0085] Experience sharing: Invite 3 members of the ethics committee (2 with over 15 years of ethics review experience and 1 with over 10 years of review experience) to list the review principles and key points for each element of the informed consent form based on their experience.

[0086] Data collection: Use machine learning algorithms to automatically extract key ethical rules and review key points from a large number of ethics review cases.

[0087] Rule refinement: Refine and customize ethical rules for specific types of scientific research projects, subject groups, and ethical issues.

[0088] (3) Sorting and summarizing stage of knowledge point rule extraction

[0089] Sorting and summarizing: The secretary sorts and summarizes the review principles and key points listed by all committee members.

[0090] Quality control: Put forward clear quality control requirements around the ethics review work.

[0091] (4) Confirmation and optimization stage of knowledge point rule extraction

[0092] Confirm review key points: Send the sorted rules and key points to the aforementioned 3 committee members and other ethics review committee members with rich review experience (1, with over 10 years of experience) for confirmation.

[0093] Continuous update: With the changes in regulations and practices, regularly update and maintain the instruction set to ensure its timeliness and accuracy.

[0094] (5) Technical implementation of knowledge point rule extraction

[0095] Application of machine learning algorithms: Apply advanced machine learning algorithms, such as natural language processing (NLP) technology, to extract rules from regulatory documents and historical cases.

[0096] Build an ethical rule base: Build a comprehensive and detailed ethical rule base to provide rich knowledge resources for model training.

[0097] (6) Transparency and interpretability of knowledge point rule extraction

[0098] Transparency of the chain of thought: Ensure that the process and logic of rule extraction are transparent, facilitating reviewers to understand the decision-making process of the model.

[0099] Application of ethical principles: Clearly apply ethical principles, such as respect for human rights, protection of privacy, fairness and justice, etc., in the design and operation of the model.

[0100] Examples of the review rules after extraction are shown in Table 1 below.

[0101] Table 1 Key Points of Ethical Review

[0102]

[0103]

[0104] III. Ethical Model Training / Fine-Tuning

[0105] When constructing an ethical review assistance system, an advanced large language model is selected as the core. Using the previously extracted ethical rules and review data, the system will conduct targeted training and fine-tuning on the large language model. In this process, the system not only focuses on the accurate understanding and application of ethical rules by the model, but also pays attention to the model's ability to handle the complexity and diversity of ethical reviews. Through continuous iteration and optimization, the model can gradually adapt to various types of ethical review tasks, improving the accuracy and efficiency of the review. At the same time, the system also has the ability of self-learning and updating, and can continuously optimize the model performance according to new ethical rules and cases.

[0106] The review tasks include conducting ethical reviews on each key point in the text.

[0107] IV. Instruction Set Construction

[0108] Construction of the instruction set for the medical ethical review scenario: In the content review scenario of medical research ethical review, the instruction set of the large model needs to ensure that medical research activities follow ethical principles, protect the rights and interests of subjects, and promote research integrity. For this specific application scenario of medical ethical review, a detailed instruction set is constructed to assist medical ethical review personnel in conducting efficient and accurate ethical evaluations.

[0109] The following are the methods and processes for constructing the instruction set:

[0110] 4.1 Initial Construction of the Instruction Set

[0111] Define the core tasks of medical ethical review: These core tasks are used to identify and evaluate ethical issues in medical practice; ensure that medical research complies with ethical norms and laws and regulations; provide ethical consultations and suggestions; review medical records and research documents; train and guide the ethical behavior of medical team members.

[0112] Initialize the basic instruction set: For each core task, that is, key point, design 50 - 100 manually constructed instructions;

[0113] In some embodiments, the knowledge points are described in easy-to-understand language (readable by junior high school students), and the corresponding instructions are:

[0114] Instruction Example 1: "Explain what this research is about in simple words, as if you were talking to a junior high school student."

[0115] Instruction Example 2: "Write down the purpose of the research, but use very easy-to-understand words and no complex medical terms."

[0116] Instruction Example 3: "Tell us why this research will be helpful to people, in simple and straightforward words."

[0117] Instruction Example 4: "Describe the benefits and possible risks of participating in this research, in a way that is easy to understand at a glance."

[0118] Instruction Example 5: "Tell us in simple language what we need to do if we participate in this research."

[0119] The initialized instructions should cover different ethical issues, scenarios, and cases; the instructions should be clear, specific, and operable. Use relevant laws, regulations, and guidelines in medical ethics as a reference basis for constructing the instruction set; ensure that the instruction set is consistent with current regulations and standards.

[0120] 4.2 Expansion and Enrichment of the Instruction Set

[0121] (1) Automated construction of the instruction set: Select a high-performance large model (such as GPT4) for data augmentation

[0122] (2) Batch processing of the initially constructed instruction set. Each time the large model (such as GPT4) is called, randomly select several instructions as input; let the large model (such as GPT4) imitate the style and structure of these instructions to generate new instructions with different content but similar forms; set the generation quantity, such as generating 100 new instructions per batch.

[0123] (3) Duplicate removal and manual review:

[0124] Perform duplicate removal on the generated instructions to ensure that each instruction is unique;

[0125] Invite medical ethics experts to conduct a manual review of the generated instructions to filter out low-quality, unethical, or ambiguous instructions.

[0126] (4) Mixing and iteration:

[0127] Mix the manually constructed instruction set and the generated instruction set together;

[0128] Randomly select a number of instructions from the mixed dataset as the new input, and call GPT4 again for generation;

[0129] Repeat the above steps, and continuously enrich and expand the instruction set in an iterative manner.

[0130] (5) Quality monitoring and feedback:

[0131] During the iteration process, continuously monitor the quality and diversity of the instruction set;

[0132] In some embodiments, in the initial stage, the instruction set is adjusted and optimized according to the feedback of medical ethics experts, and in the later stage, the instruction set is adjusted and optimized through automatic quality assessment.

[0133] In some embodiments, the instruction set is adjusted and optimized through automatic quality assessment.

[0134] Design an audit calculation formula for monitoring the quality and diversity of the instruction set to evaluate the quality and diversity of the instruction set in the ethical review of medical documents, ensuring that the instruction set is both accurate and comprehensive.

[0135] Formula composition:

[0136]

[0137] Where:

[0138] ·Q is the average quality score of the instruction set.

[0139] ·W is the total number of knowledge points. A knowledge point refers to the review key points extracted by the rules

[0140] ·I is the total number of instructions. An instruction is the review instruction set formed according to the knowledge points.

[0141] ·z(w) and z(i) are the basic weights of knowledge point w and instruction i respectively.

[0142] ·y(w) and y(i) are the application weights of knowledge point w and instruction i respectively.

[0143] ·k1 and k2 are adjustment coefficients used to balance the influence of different weights.

[0144] Determination of weight coefficients:

[0145] ·z(w) and y(w) can be determined according to the importance and frequency of the knowledge points in the medical ethics review.

[0146] ·z(i) and y(i) can be determined according to the coverage, accuracy and practicality of the instructions.

[0147] Implementation steps:

[0148] Determining the weight coefficients: The determination of traditional weight systems is generally completed through a cooperation model with experts. To enhance the automation of determining the weight coefficients of medical ethics review instructions, the basic weights z(w) and z(i), as well as the applied weights y(w) and y(i), are all completed in an automated mode.

[0149] Automated determination of the basic weights z(w) and z(i)

[0150] · Cluster analysis: Use clustering algorithms (such as K-means) to group knowledge points and instructions to identify common patterns or categories.

[0151] · Text mining: Utilize text mining techniques to analyze review feedback and automatically extract the importance of knowledge points and instructions.

[0152] Basic weight calculation formula:

[0153] z(w) = α · ftype(w) + β · ffrequency(w) + γ · fimpact(w)

[0154] z(i) = δ · ftype(i) + ∈ · ffrequency(i) + ζ · fdifficulty(i)

[0155] Where:

[0156] ftype(w): The type of knowledge point, such as privacy protection, informed consent, etc.

[0157] ffrequency(w): The frequency of the knowledge point appearing in the review.

[0158] fimpact(w): The degree of influence of the knowledge point on the review result, which can be quantified by the relevance between the review result and the knowledge point.

[0159] The review result (review conclusion) is a component of the review report. The review report is a detailed document that includes the analysis results of each review point (knowledge point) and the final conclusion (review result) after comprehensive analysis of all results; the review result can be automatically evaluated by setting a threshold for the quality score Q; ftype(i) represents the type of knowledge point to which the instruction belongs, and fdifficulty(i) represents the execution difficulty of instruction i; α, β, γ, δ, ∈, ζ are parameters determined by data-driven methods;

[0160] By analyzing a large amount of historical review data, including review results, application frequencies of knowledge points, execution difficulties of instructions, etc., key factors affecting the weights of knowledge points and instructions are determined. For example, through a regression analysis model, the influence degree of different knowledge points on review results is evaluated, and thus parameters α, β, and γ are automatically calculated to quantify the basic weight z(w) of knowledge points. Similarly, by analyzing the execution frequency of instructions and the feedback from reviewers, parameters δ, ∈, and ζ are determined to quantify the basic weight z(i) of instructions.

[0161] fdifficulty(i): The difficulty of executing the instruction, which may be based on the feedback from reviewers on the execution difficulty.

[0162] frelevance(i): The relevance of the instruction to the review task, which can be measured by the frequency of the instruction being cited in the review.

[0163] α, β, γ, δ, ∈, ζ are parameters determined by a data-driven method.

[0164] Automatic determination of applied weights y(w) and y(i)

[0165] · Regression analysis: Apply regression analysis to evaluate the specific impact of each knowledge point and instruction on review results.

[0166] · Decision tree or random forest: Use decision tree or random forest models to identify which features have a significant impact on the weights.

[0167] Applied weight calculation formula:

[0168] y(w) = η · fimpact(w) + θ · frelevance(w)

[0169] y(i) = ι · frelevance(i) + κ · fdifficulty(i)

[0170] Where:

[0171] ftype(w): The type of knowledge point, such as privacy protection, informed consent, etc.

[0172] ffrequency(w): The frequency of the knowledge point appearing in the review.

[0173] fimpact(w): The degree of influence of the knowledge point on review results, which can be quantified by the relevance between the review result and the knowledge point.

[0174] fdifficulty(i): The difficulty of executing the instruction, which may be based on the feedback from reviewers on the execution difficulty.

[0175] frelevance(i): The relevance of an instruction to a review task, which can be measured by the frequency of the instruction being cited in the review.

[0176] η, θ, ι, κ are parameters determined by a data-driven method.

[0177] Suppose the obtained basic weight z(1) of knowledge point 1 (regarding patient privacy protection) = 0.8, and the applied weight y(1) = 0.9.

[0178] Calculate the relevance coefficient: Use the relevance coefficient calculation method in the formula to calculate the relevance coefficient between each knowledge point and the instruction. For example, for knowledge point 1 and instruction 1 (ensuring the confidentiality of patient data), calculate the relevance coefficient:

[0179]

[0180] Calculate the average quality score: Sum up the relevance coefficients of all knowledge points and instructions, and then divide by the total number of knowledge points and instructions to obtain the average quality score Q. Suppose the sum of all relevance coefficients is 50.0, then:

[0181]

[0182] Review and adjustment: Based on the calculated average quality score Q, review the instruction set to ensure it meets the requirements of medical ethics review, and make adjustments according to the feedback. If Q = 0.25 is considered lower than the expected standard, it may be necessary to add more instructions regarding patient rights protection or adjust the weights of existing instructions.

[0183] Through these steps, the quality and diversity of the medical ethics review instruction set can be effectively monitored and continuously improved.

[0184] 4.3 Final construction of the instruction set

[0185] Determine the scale of the final instruction set:

[0186] According to the actual needs of medical ethics review, determine the quantity and coverage of the final instruction set.

[0187] Consider the characteristics of medical document ethics review, such as the depth, breadth of the review, and the comprehensiveness of the instruction set. Design a calculation formula for the coverage range, which can help us evaluate and determine the coverage range of the instruction set:

[0188]

[0189] Where:

[0190] ·C represents the coverage range score of the instruction set.

[0191] · I is the total number of instructions.

[0192] · c(i) is the coverage score of the i-th instruction, which can be determined according to the types and quantities of ethical issues covered by the instruction.

[0193] · S is a normalization factor used to adjust the coverage score to ensure it is within a reasonable range (e.g., between 0 and 1).

[0194] Determination of the coverage score c(i):

[0195] The coverage score c(i) can be determined based on the following factors:

[0196] Types of ethical issues: The more types of ethical issues covered by the instruction, the higher the score.

[0197] Severity of ethical issues: Covering more severe or more common ethical issues can increase the score.

[0198] Applicability of the instruction: The applicability of the instruction in different medical fields or situations.

[0199] Implementation steps:

[0200] Evaluate the ethical issues covered by the instruction: Determine the types and severity of the ethical issues covered by each instruction i.

[0201] Calculate the coverage score for each instruction: Assign a coverage score c(i) to each instruction according to the above factors.

[0202] Calculate the average coverage score: Sum up the coverage scores of all instructions and divide by the total number of instructions I to obtain the average coverage score.

[0203] Apply the normalization factor: Multiply the average coverage score by the normalization factor S to obtain the final coverage score C.

[0204] In some embodiments, if the coverage score does not meet the requirements, targeted simulation instruction sets are generated for the knowledge points lacking coverage until the coverage score meets the requirements; wherein, the threshold of the quality requirement is determined based on the training objective.

[0205] In some embodiments, weights are evaluated for all knowledge points, and the knowledge points with higher weights are more important in the evaluation of the coverage.

[0206] The types and quantities covered by an instruction are not fixed and are related to the corresponding ethical review documents. Taking the "informed consent" instruction set as an example

[0207] 1. Types of ethical issues

[0208] Transparency: Whether the subjects are fully informed of all relevant information about the research.

[0209] Comprehensibility: Whether the subjects truly understand the information they are told.

[0210] Voluntariness: Whether the subjects' consent is voluntary and not subject to any form of coercion or undue influence.

[0211] Capacity: Whether the subjects have sufficient cognitive ability to make a decision to consent.

[0212] 2. Evaluate the coverage quantity

[0213] Transparency: The instructions detail all the information that needs to be disclosed to the subjects, with a coverage of 100%.

[0214] Comprehensibility: The instructions require the researchers to explain the research information in plain language, with a coverage of 80%.

[0215] Voluntariness: The instructions clearly prohibit any form of coercion or undue influence, with a coverage of 100%.

[0216] Capacity: The instructions require an assessment of the subjects' cognitive ability to ensure that they can understand and consent to participate in the research, with a coverage of 70%.

[0217] Integration and sorting:

[0218] Integrate all the reviewed and optimized instructions together;

[0219] Sort and classify according to the type, difficulty, and importance of the instructions.

[0220] Use the simulated instruction set that meets the requirements after sorting and classification as the final training instruction set.

[0221] Instruction evaluation calculation formula:

[0222] E = α·T + β·D + γ·I + δ·(T×D×I)

[0223] Where:

[0224] E is the overall evaluation score of the instructions.

[0225] T is the type score of the instructions.

[0226] D is the difficulty score of the instructions.

[0227] I is the importance score of the instructions.

[0228] α, β, γ, and δ are weight coefficients used to adjust the relative importance of different factors.

[0229] T×D×I represents the interaction between type, difficulty, and importance, and δ is used to adjust the influence of this interaction.

[0230] Determination of scores:

[0231] Type score T: Score according to the types and scope of the ethical review types covered by the instructions. A multi-level scoring system can be adopted. For example, instructions covering 1 - 2 types are scored 0.5 points, instructions covering 3 - 4 types are scored 0.7 points, and instructions covering 5 or more types are scored 0.9 points.

[0232] Difficulty score D: Score according to the professional knowledge, skills, and resources required to execute the instructions. A reverse scoring system can be adopted, that is, the higher the difficulty, the lower the score, to reflect the impact of execution difficulty on the evaluation of the instructions.

[0233] Importance score I: Score according to the degree of influence of the instructions on the ethical review results. A importance scoring system based on case studies or expert consensus can be adopted.

[0234] Interaction T×D×I Consider the interaction between type, difficulty, and importance, which can reflect the comprehensive effect of the instructions in actual applications.

[0235] Implementation steps:

[0236] Determine the weight coefficients: Collaborate with medical ethics experts to determine the weight coefficients α, β, γ, and δ for type, difficulty, importance, and interaction.

[0237] Evaluate the instructions: Determine the type score T, difficulty score D, and importance score I for each instruction.

[0238] Calculate the interaction: Calculate the interaction between type, difficulty, and importance for each instruction.

[0239] Calculate the overall evaluation score: Calculate the overall evaluation score E for each instruction using the above formula.

[0240] Sorting and classification: Sort and classify the instructions according to the calculated overall evaluation score E.

[0241] Write the usage instructions:

[0242] Write detailed usage instructions for the instruction set, including the interpretation of the instructions, application scenarios, and operation methods, etc.

[0243] Training and promotion:

[0244] Train medical ethics reviewers to familiarize them with and master the usage method of the instruction set;

[0245] Promote the instruction set to relevant medical institutions and ethics review committees to promote the standardization and regularization of medical ethics reviews.

[0246] V. Fine-tune the large language model based on the constructed instruction set to obtain an ethical review model

[0247] Combined with the characteristics of ethical review, the process of fine-tuning the large language model based on the constructed instruction set to obtain an ethical review model involves integrating knowledge points and principles related to ethical review into the model. Key ethical knowledge and examples are extracted from a third-party large model through knowledge distillation technology, and this adaptively adjusted high-quality content is used as training data to enhance the model's understanding and generalization ability of complex ethical review situations;

[0248] The input content includes ethical review-related documents such as medical research proposals, informed consent forms, research methods, and participant data,

[0249] The output is a detailed ethical review report, which covers the ethical evaluation of the research protocol, the identification of potential risks, and recommended measures for the protection of participant rights, thus ensuring that the model's performance in ethical review tasks is more accurate and reliable, while improving the transparency and consistency of the review.

[0250] Introduction of knowledge distillation context examples: Since ethical review is a complex business area that involves not only medical knowledge but also many other aspects of knowledge, methods such as introducing context examples will be adopted to enhance the capabilities of the ethical review model. First, through knowledge distillation, the ethical review content is input into a third-party model with larger model parameters (such as Mistral-Large-Instruct-2407) for further verification. After obtaining generated content that better meets the requirements, the context is adaptively adjusted and introduced as an example into this model. This not only enriches and optimizes the training dataset, improving the quality and diversity of the data; but also enables this model to learn how to generate more compliant content when exposed to higher-quality content, thereby improving its output quality; and also by introducing high-quality data from different sources, this model can learn more patterns and features, which helps to enhance its generalization ability. The flowchart for processing the introduction of high-quality context examples is as follows:

[0251] 5.1. Preparation stage

[0252] (1) Obtain a third-party model

[0253] Performance verification: Evaluate key performance indicators such as the accuracy, recall rate, and F1 score of the third-party model through a benchmark test set.

[0254] Compliance check: Ensure that the third-party model does not use any illegal or non-compliant data for training, and that its output content complies with the relevant laws, regulations, and ethical requirements of medical research ethical review.

[0255] (2) Generated content example

[0256] Use a third - party model to generate content related to medical research ethics: Input queries or prompts related to medical research ethics, and use the third - party model to generate a series of relevant texts or data. Ensure that the generated content covers a wide range of medical research ethics scenarios, such as clinical trial design, informed consent form writing, data privacy protection, etc.

[0257] Screening and pre - processing example:

[0258] Formatting: Convert the generated text or data into a format suitable for input to the large - model for medical research ethics review.

[0259] Desensitization: Remove or replace any parts that may disclose personal privacy or sensitive information to ensure the compliance and security of the examples.

[0260] Quality check: Manually or automatically check the generated examples to ensure they are representative, accurate, and relevant.

[0261] 5.2. Training / fine - tuning stage

[0262] (1) Set up the teacher - student model

[0263] A third - party model as the teacher model: Select a third - party model with superior performance and compliance as the teacher model to provide soft labels or guidance information.

[0264] The large - model for medical research ethics review as the student model: Initialize the parameters of the large - model for medical research ethics review and prepare to receive guidance from the teacher model for training or fine - tuning.

[0265] (2) Teacher model inference

[0266] Perform inference on the screened examples:

[0267] Use the teacher model to perform inference on the screened examples to obtain soft labels (such as probability distributions) or hard labels (such as classification results) for each example.

[0268] Ensure that the inference process is stable and efficient to support the subsequent training process.

[0269] (3) Construct the training dataset

[0270] Combine the filtered examples with their corresponding soft labels to form a training set: Here, the "examples" and "soft labels" correspond to the instruction set and knowledge points. In the context of ethical review, these examples are usually specific cases or scenarios extracted from ethical review-related documents such as medical research proposals, informed consent forms, research methods, and participant data. These examples reflect various situations that may be encountered in the actual ethical review process. The soft labels refer to the expected outputs or evaluation criteria corresponding to these examples, which are usually based on the predefined instruction set and knowledge points. The instruction set provides specific guiding principles for ethical review, while the knowledge points cover the key concepts and criteria in ethical review. The soft labels may include the identification, evaluation, and recommended measures for ethical issues in the examples, and these soft labels reflect the application of the instruction set and knowledge points in actual cases.

[0271] Combine each example with its corresponding soft label or hard label to form a training sample.

[0272] Construct a training data set containing multiple training samples for training the large model for medical research ethical review.

[0273] (4) Train the student model (large model for medical research ethical review)

[0274] Use the training set to train the large model for medical research ethical review:

[0275] Input the training data set into the large model for medical research ethical review for training or fine-tuning.

[0276] Adopt optimization algorithms (such as Adam, SGD, etc.) and learning rate strategies to update the parameters of the model.

[0277] Introduce a loss function:

[0278] Design a loss function to measure the difference between the output of the student model (large model for medical research ethical review) and the soft labels of the teacher model.

[0279] Common loss functions include cross-entropy loss, KL divergence, etc. Minimize the loss function to encourage the student model (large model for medical research ethical review) to imitate the output of the teacher model.

[0280] Optimization of cross-entropy loss

[0281] In view of the characteristics of the training data for medical research ethical review, introduce a classification weighting method for improvement on the basis of the standard cross-entropy loss, and its formula is:

[0282]

[0283] where n is the number of categories, y i is the iProbability of a category (usually a probability distribution for multi-class classification problems), p i is the probability of the i th category predicted by the student model (the large model for medical research ethics review).

[0284] Combined with the actual situation in the training process of the large model for medical research ethics review, the cross-entropy loss function is fused with weighting and label smoothing to construct a weighted label smoothing cross-entropy loss formula:

[0285]

[0286] L represents the total loss.

[0287] w i is the weight of the i th category, used to handle the class imbalance problem.

[0288] y i is the true label. If it is the i th category, then y i = 1; otherwise y i = 0.

[0289] p i is the probability that the model predicts as the i th category.

[0290] λ is the L2 regularization coefficient, used to control the model complexity.

[0291] n is the total number of categories.

[0292] Application of the class weight w i : In medical research ethics review, different ethical issues may have different importance and occurrence frequencies. By assigning different weights w i to different categories of ethical issues, the model can pay more attention to those more important or more frequently occurring ethical issues, thereby improving the prediction accuracy and robustness of the model.

[0293] Label smoothing: In medical research ethics review, due to possible noise or fuzzy boundaries in the data, label smoothing helps the model better handle these complex situations and improve the classification accuracy.

[0294] L2 regularization: In the large model for medical research ethics review, regularization helps improve the generalization ability of the model, enabling the model to maintain good performance when facing new and unseen data.

[0295] Optimization of the loss function: By minimizing the above loss function, the model learns to consider not only the class weights but also the difference between the predicted probability and the true label during prediction. This helps the model to more carefully distinguish different ethical issues during ethical review, improving the accuracy and reliability of the review.

[0296] Optimization of KL divergence

[0297] KL divergence is used to measure the difference between two probability distributions, and its formula is:

[0298]

[0299] where T is the target distribution (usually the soft labels of the teacher model), and P is the predicted distribution of the student model. To avoid numerical stability issues, usually before calculating the KL divergence, a small positive number ∈ (such as 10 i ) is added to the predicted probability p of the student model (the large model for medical research ethics review). -9 )

[0300] Forward propagation (same as cross-entropy loss): Use the student model to process the input data to obtain the predicted probability p i .

[0301] Calculate the KL divergence: Obtain the soft labels t of the teacher model i .

[0302] To avoid division-by-zero errors, replace the predicted probability p i with p i ′ = max(∈, p i ).

[0303] According to the formula of KL divergence, calculate the KL divergence between the predicted probability p i ′ of the student model and the soft labels t i of the teacher model.

[0304] Backward propagation: Calculate the gradient of the KL divergence D KL (T || P) with respect to each layer's parameters of the student model. It involves the chain rule and the derivative of the softmax function (since p i is obtained through softmax). For the derivative of the KL divergence

[0305]

[0306] Pass the gradient layer by layer from the output layer back to the input layer to update the parameter gradients of each layer.

[0307] Parameter update (same as cross-entropy loss): Use an optimization algorithm to update the parameters of the student model according to the gradient.

[0308] 5. Fine-tuning and Optimization

[0309] Adjust model parameters according to training feedback:

[0310] During training, regularly monitor the model's performance (such as accuracy, loss value, etc.) and training status (such as gradient explosion, overfitting, etc.).

[0311] Adjust the model's parameters (such as learning rate, batch size, etc.) according to the monitoring results to optimize the training effect.

[0312] Adopt strategies such as cross-validation and early stopping to avoid overfitting:

[0313] Use cross-validation to evaluate the generalization ability of the model and select appropriate model parameters.

[0314] Implement the early stopping strategy during training. Stop training when the performance on the validation set starts to decline to avoid overfitting.

[0315] 5.3. Evaluation and Validation Phase

[0316] (1) Prepare the validation dataset

[0317] Include various scenarios and examples related to medical research ethics:

[0318] Construct a validation dataset that includes various scenarios and examples related to medical research ethics.

[0319] Ensure that the validation dataset is representative, diverse, and complementary to the training dataset in terms of content.

[0320] (2) Evaluate the model performance

[0321] Use the validation dataset to evaluate the accuracy of the large model for medical research ethics review:

[0322] Input the validation dataset into the trained large model for medical research ethics review for inference and evaluation.

[0323] Calculate key performance indicators such as accuracy, recall rate, and F1 score of the model on the validation dataset.

[0324] 3. Optimization and Adjustment

[0325] Further optimize and adjust the model according to the evaluation results:

[0326] Analyze the performance of the model on the validation dataset and identify existing problems and deficiencies.

[0327] Further optimize and adjust the model according to the analysis results, such as modifying the model structure, adding regularization terms, etc.

[0328] Repeat the process of evaluation and optimization until the model performance reaches a satisfactory level.

[0329] In a specific embodiment, the constructed model is applied for ethical review. The text to be reviewed is input, as shown in Table 1:

[0330] Table 1: Text to be reviewed

[0331]

[0332]

[0333] The results of the ethical review are shown in Table 2. The generated content before fine-tuning the instruction set constructed in this application only includes brief content review, while the generated content after fine-tuning the instruction set constructed in this application covers multiple knowledge points of ethical review.

[0334] Table 2: Results of the ethical review

[0335]

[0336]

[0337] Apply the constructed ethical review model to conduct ethical review processing of the document. The application steps include:

[0338] Obtain the medical text data to be reviewed, input it into the trained large language model for ethical review described above, and output the review results.

[0339] The medical text data is one or more of the following: medical research proposals, informed consent forms, research methods, and participant data, etc., which are relevant documents for ethical review.

[0340] Based on the review results, obtain an ethical review report. The ethical review report is a detailed report, which includes:

[0341] ① Review results: Clearly indicate whether the research proposal meets ethical standards, whether it has been approved, or requires further modification.

[0342] ② Potential risks: List the risks that the research may pose to participants, as well as the controllability and mitigation measures of these risks.

[0343] ③ Suggested measures: Provide specific suggestions to guide the research team on how to improve the research design to better protect the rights and safety of participants.

[0344] Review Conclusion Assembly / Output: Finally, the system will organize the generated review conclusions to form a review report with a clear structure and easy to understand. In this process, the system uses Document Assembly technology to arrange the conclusions of each part in an orderly manner to ensure the integrity and consistency of the report. At the same time, the system also provides a variety of output formats, such as PDF, Word, etc., to meet the needs of different users. Through this step, the system provides users with convenient and efficient ethical review services.

[0345] Figure 3 It is a schematic diagram of a computer device provided by an embodiment of the present invention. As Figure 3 shown, the device 2000 may include: one or more processors 2010, and one or more memories 2020; wherein, computer-readable code is stored in the memory, and when the computer-readable code is run by the one or more processors, the above-mentioned method can be executed.

[0346] The processor in this embodiment may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, operations and logic block diagrams disclosed in the embodiments of the present disclosure. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc., and may be of the X86 architecture or the ARM architecture.

[0347] Generally speaking, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that can be executed by a controller, a microprocessor or other computing devices. When aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flowcharts or using some other graphical representation, it will be understood that the blocks, devices, systems, technologies or methods described herein may be implemented as non-limiting examples in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0348] For example, the method or device according to the embodiments of the present disclosure may also be implemented by means of Figure 4 the architecture of the computing device 3000 shown. As Figure 4As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, and so on. Storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the methods provided by this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 4 the architecture shown is merely exemplary, and when implementing different devices, one or more components in the shown computing device may be omitted according to actual needs. Figure 4

[0349] An embodiment of the present invention also provides a computer-readable storage medium. As Figure 5 shown, it is a schematic diagram of the storage medium 4000 provided by an embodiment of the present invention. Computer-readable instructions 4010 are stored on the computer storage medium 4020. When the computer-readable instructions 4010 are run by a processor, the methods according to the embodiments of this disclosure described with reference to the above figures can be executed. The computer-readable storage medium in the embodiments of this disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memories for the methods described herein are intended to include, but are not limited to, these and any other suitable types of memories. It should be noted that the memories for the methods described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0350] An embodiment of this disclosure also provides a computer program product or a computer program. When the computer program is executed by a processor, the steps of the above method are implemented. As Figure 2 shown, the computer program product or the computer program includes:

[0351] Acquisition module 201: Acquire the knowledge points of ethical review and the manually constructed instruction sets corresponding to each knowledge point;

[0352] Instruction set construction module 202: Input the knowledge points and the corresponding manually constructed instruction sets into a generative large language model to generate a specified number of simulated instruction sets, where the content of the simulated instruction sets is different from that of the manually constructed instruction sets but the form is similar;

[0353] Instruction set evaluation and update module 203: Perform quality evaluation on the simulated instruction sets to obtain a quality score. If the quality score is higher than the threshold, it is considered that the simulated instruction sets meet the requirements; otherwise, it is considered that the simulated instruction sets do not meet the requirements;

[0354] Model fine-tuning training module 204: Combine the manually constructed instruction sets and the simulated instruction sets that meet the requirements as training instruction sets, and input the training instruction sets into the base model of the large language model for fine-tuning training to obtain an ethical review large model.

[0355] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the module, the program segment, or the part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0356] Generally speaking, the various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that can be executed by a controller, a microprocessor, or other computing devices. When aspects of the embodiments of the present disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, devices, systems, technologies, or methods described herein may be implemented as non-limiting examples in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof.

[0357] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0358] In several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling, direct coupling, or communication connection can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be in electrical, mechanical, or other forms.

[0359] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0360] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0361] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art should understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for constructing a large ethical review model based on instruction set optimization, characterized in that: The method comprises: Obtain the knowledge points of ethical review and the manually constructed instruction sets corresponding to each knowledge point; Inputting the knowledge points and the corresponding manually constructed instruction sets into a generative large language model to generate a specified number of simulated instruction sets, wherein the simulated instruction sets are different in content from the manually constructed instruction sets but similar in form; performing a quality assessment on the simulated instruction sets to obtain a quality score, and if the quality score is higher than a threshold, it is considered that the simulated instruction set meets the requirements, otherwise it is considered that the simulated instruction set does not meet the requirements; The artificially constructed instruction set and the simulated instruction set that meets the requirements are combined as a training instruction set, and the training instruction set is input into the base model of the large language model for fine-tuning training to obtain the ethics review large model.

2. The method for constructing a large ethical review model based on instruction set optimization according to claim 1 is characterized in that: The method for performing quality assessment on the simulation instruction set to obtain a quality score is: the quality score is calculated based on different knowledge points in the simulation instruction set, basic weights and application weights of different instructions corresponding to the knowledge points, and adjustment coefficients of the basic weights and application weights; Optionally, the quality score is calculated as follows: Where Q represents the quality score of the instruction set, W is the total number of knowledge points, I is the total number of simulated instructions, z(w) and z(i) are the basic weights of knowledge point w and instruction i, respectively; y(w) and y(i) are the application weights of knowledge point w and instruction i, respectively; k1 and k2 are adjustment coefficients used to balance the impact of different weights.

3. The method for constructing a large ethical review model based on instruction set optimization according to claim 2 is characterized in that: The basic weights of knowledge point w and instruction i are determined according to the importance and frequency of the knowledge point in medical ethics review; Optionally, the method for obtaining the basic weights of the knowledge point w and the instruction i is: Step 1: Group the knowledge points and instructions of the simulation instruction set using a clustering algorithm; Step 2: Use text mining technology to automatically extract the importance of knowledge points and instructions in each group; Among them, the basic weight calculation formula is expressed as: z(w)=α·ftype(w)+β·frequency(w)+γ·fimpact(w) z(i)=δ·ftype(i)+∈·frequency(i)+ζ·fdifficulty(i) Among them, ftype(w) represents the type of knowledge point, ffrequency(w) represents the frequency of knowledge point appearance in the group; fimpact(w) represents the influence of knowledge point on the review result, which can be quantified by the correlation between the review result and the knowledge point; ftype(i) represents the type of knowledge point to which the instruction belongs, and fdifficulty(i) represents the execution difficulty of instruction i; α, β, γ, δ,∈,ζ are parameters determined by data-driven method; Optionally, the method for obtaining the application weights of the knowledge point w and the instruction i is: In step 1, regression analysis was applied to evaluate the specific impact of each knowledge point and instruction on the review results; Step 2: Use feature selection to select knowledge points and instructions that have a significant impact on the weights; The calculation formula of application weight is expressed as: y(w)=eta·fimpact(w)+θ·frelevance(w) y(i)=ι·frelevance(i)+κ·fdifficulty(i) Among them, fimpact(w) represents the influence of the knowledge point on the review result; fdifficulty(i) is the difficulty of executing the instruction; frelevance(i) represents the correlation between the instruction and the reviewed knowledge point; η, θ, ι, κ are parameters determined by data-driven methods.

4. The method for constructing a large ethical review model based on instruction set optimization according to claim 1 is characterized in that: The method further includes: when the simulation instruction set does not meet the requirements, regenerating a replacement simulation instruction set, performing a quality assessment on the replacement simulation instruction set to obtain a quality score, if the quality score is higher than a threshold, the replacement simulation instruction set is considered to meet the requirements, otherwise, the replacement simulation instruction set is considered to not meet the requirements; optionally, mixing the manually constructed instruction set and the simulation instruction set that meets the requirements to obtain a mixed instruction set, selecting a part of the instruction set from the mixed instruction set and inputting it into a large language model to generate a specified number of new simulation instruction sets, performing a quality assessment on the new simulation instruction set to obtain a quality score, if the quality score is higher than a threshold, the new simulation instruction set is considered to meet the requirements, otherwise, the new simulation instruction set is considered to not meet the requirements; Optionally, the method also includes: if the number of training instruction sets does not meet the requirements, continuously generating new simulation instruction sets, and merging the instruction sets that have passed the quality assessment into the mixed instruction set until the training instruction set obtained by merging the manually constructed instruction set and the simulation instruction set that meets the requirements meets the requirements.

5. The method for constructing a large ethical review model based on instruction set optimization according to claim 1 is characterized in that: The training instruction set is scored for coverage. If the coverage score does not meet the requirements, a directed simulation instruction set is generated for the knowledge points that lack coverage to obtain directed simulation instructions. The directed simulation instruction set is quality evaluated to obtain a quality score. If the quality score is higher than a threshold, it is considered that the directed simulation instruction set meets the requirements. Otherwise, it is considered that the directed simulation instruction set does not meet the requirements. The training instruction set and the directed simulation instruction set that meets the requirements are merged as an updated training instruction set, and the updated training instruction set is scored for coverage. This is repeated until the updated training instruction set coverage score meets the requirements. The updated training instruction set that meets the requirements is input into the base model of the large language model for fine-tuning training to obtain the ethics review large model. Optionally, the coverage score is obtained based on the training instruction set and the range corresponding to the instructions; Optionally, the coverage score is expressed as: Where C represents the coverage score of the training instruction set, I is the total number of instructions in the training instruction set, c(i) is the coverage score of the i-th instruction; S is a normalization factor; Optionally, the coverage score of the i-th instruction depends on the type of knowledge points covered by the instruction.

6. The method for constructing a large ethical review model based on instruction set optimization according to claim 1 is characterized in that: The generative large language model includes one or more of the following: ChatGPT, GPT4, Qwen1.5-72B, Yi-1.5-34B; The base model of the large language model includes one or more of the following: ChatGLM2-6B, VIP5, VisualGLM-6B, Mistral-Large-Instruct-2407; Optionally, the threshold is a preset threshold.

7. A method for ethical review based on instruction set optimization, characterized in that: The method comprises: Get the text to be reviewed; The text to be reviewed is input into the ethics review model according to any one of claims 1-6 to obtain the ethics review result.

8. A computer device, characterized in that: The device comprises: a memory and a processor; the memory is used to store a computer program; the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images