Intelligent extraction method of clinical research test abstract keywords based on large model

Abstract

The application provides a kind of clinical research test abstract key word intelligent extraction method based on large model, it is related to data analysis technical field.Adopt parameter inheritance system strategy training network, introduce load balancing loss optimization MoE model.Structured feature embedding is integrated on the basis vector to obtain the input vector, input large model to obtain token level context representation and test level backbone feature, combine task prompt vector input MoE model to obtain test level enhanced feature, respectively input multi-field classification head and multi-label output head to obtain field probability distribution and key word activation probability;Joint dynamic threshold, and then get structured field and key word set.Using the parallel design of pre-trained large model and MoE model and multi-field classification head and multi-label output head ensures the comprehensiveness and accuracy of information.Overcome the problems such as element loss, unstable generalization ability and poor robustness when processing clinical research abstracts in the prior art.

Description

Technical Field

[0001] This invention relates to the field of data analysis technology, and more specifically, to a method for intelligent extraction of keywords from clinical research trial abstracts based on large models. Background Technology

[0002] Existing short text comprehension / keyword methods mostly remain at the level of "topic / a few terms". However, clinical research involves many aspects such as drugs / devices, interventional / observational studies, real-world / registration studies, and multiple disease systems. Clinical research abstracts often contain mixed Chinese and English, Latin abbreviations (AE / SAE, PK / PD, or CQTc, etc.), and thresholds with comparison symbols and units (e.g., decrease ≥20% or absolute value ≥0.8kPa, etc.). Existing abstract keyword extraction methods are unable to reliably cover the multidimensional combination of clinical research elements, such as study stage, study type, randomization / blinding / control, center type, administration and visits, endpoints and safety observations, etc. This leads to problems such as missing elements, unstable cross-domain generalization ability, and poor robustness to mixed language and quantitative conditions in the extracted keywords. Summary of the Invention

[0003] The problem this invention aims to solve is that existing methods suffer from issues such as missing key element extraction, unstable cross-domain generalization ability, and poor robustness to mixed language and quantitative conditions.

[0004] To address the aforementioned problems, in a first aspect, this invention provides a method for intelligent extraction of keywords from clinical trial abstracts based on a large model, comprising:

[0005] The clinical research abstract is segmented into words to obtain the basic vector;

[0006] Based on the basic vector, structured feature embedding is incorporated to obtain the input vector. The structured feature embedding includes language type feature embedding, comparison symbol feature embedding, unit of measurement feature embedding, and numerical change feature embedding.

[0007] Input the input vector into the pre-trained large model to obtain token-level contextual representations and experimental backbone features;

[0008] Input the token-level context representation, experimental-level backbone features, and task cue vectors into the pre-trained MoE model to obtain experimental-level augmented features;

[0009] The experimental enhancement features are input into the multi-field classification head and the multi-label output head respectively to obtain the probability distribution of the fields and the keyword activation probability;

[0010] Based on the probability distribution of the fields, the activation probability of keywords, and the corresponding dynamic thresholds, a set of structured fields and keywords is obtained.

[0011] The large model, MoE model, multi-field classification head, and multi-label output head are pre-trained using clinical research corpora, task corpora, and supervised sample data with parameter inheritance, and the MoE model is trained using a load balancing loss based on probability usage-actual allocation.

[0012] Secondly, this invention also provides an intelligent keyword extraction system for clinical research trial abstracts based on a large model, comprising:

[0013] The abstract processing module is used to segment clinical research abstracts into words to obtain basic vectors;

[0014] The input building module is used to incorporate structured feature embeddings into the basic vector to obtain the input vector. The structured feature embeddings include language type feature embedding, comparison symbol feature embedding, unit of measurement feature embedding, and numerical change feature embedding.

[0015] The feature extraction module is used to input the input vector into the pre-trained large model to obtain token-level context representation and experimental backbone features;

[0016] The feature fusion module is used to input token-level context representation, experimental-level backbone features, and task cue vectors into the pre-trained MoE model to obtain experimental-level enhanced features;

[0017] The multi-head classification module is used to input experimental enhanced features into the multi-field classification head and the multi-label output head respectively to obtain the probability distribution of the fields and the keyword activation probability.

[0018] The output filtering module is used to obtain a set of structured fields and keywords based on the probability distribution of the fields, the activation probability of keywords, and the corresponding dynamic thresholds.

[0019] The pre-training module is used to pre-train the network architecture using parameter inheritance based on clinical research corpora, task corpora, and supervised sample data, and to use a load balancing loss based on probability use-actual allocation to constrain the training of the MoE model. The network architecture includes a large model, a MoE model, a multi-field classification head, and a multi-label output head.

[0020] This invention provides an intelligent keyword extraction method for clinical trial abstracts based on a large model. Compared with existing technologies, it has the following advantages:

[0021] Employing a phased, domain-adaptive pre-training strategy based on parameter inheritance allows the model to better adapt to the specialized corpora and task characteristics of clinical research, improving its accuracy. A load-balanced loss based on probabilistic usage-actual allocation optimizes the internal working mechanism of the MoE model, ensuring effective utilization of each expert subnetwork, thereby enhancing the overall performance and generalization ability of the model. Encoding structured information such as language type, comparison symbols, units of measurement, and numerical variations into feature embeddings effectively addresses the issue of poor robustness to mixed language and quantitative conditions, thus avoiding extraction errors or omissions caused by these complex elements. Utilizing the pre-trained large model and the MoE model significantly improves the coverage of multidimensional and complex elements in clinical research abstracts and enhances cross-domain generalization ability. The large model provides powerful contextual understanding capabilities, enabling a holistic grasp of the abstract's semantics, while the MoE model, through its expert subnetwork architecture, efficiently handles diverse information types in clinical research, allowing for a more comprehensive and in-depth understanding of the multidimensional combinations of elements in the abstract. By employing a parallel design with multi-field classification headers and multi-label output headers, the system achieves simultaneous extraction of structured fields and keywords, ensuring the comprehensiveness and accuracy of information and providing richer and more refined data for the in-depth utilization of clinical research information. This effectively overcomes the problems of missing elements, unstable generalization ability, and poor robustness faced by existing technologies in processing clinical research abstracts. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 A flowchart illustrating a method for intelligent extraction of keywords from clinical trial abstracts based on a large model, provided in this embodiment of the invention;

[0024] Figure 2 This is a schematic diagram of the network architecture and information transmission provided in an embodiment of the present invention;

[0025] Figure 3 A schematic diagram of the structure of the intelligent keyword extraction system for clinical research trial abstracts based on a large model, provided in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments of this application are described clearly and completely. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] In the process of keyword extraction from clinical research abstracts, existing methods fail to effectively handle mixed Chinese and English text, Latin abbreviations, and threshold conditions with comparison symbols and units, resulting in incomplete coverage of multidimensional clinical research elements. Specifically, these methods exhibit problems such as missing element identification, poor cross-domain adaptability, and insufficient robustness to mixed languages ​​and quantitative conditions when parsing content involving drug / device type, study design (e.g., interventional / observational), study environment (e.g., real-world / registration), and multiple disease systems. Consequently, the system is negatively impacted in terms of the accuracy and consistency of field classification and keyword extraction, failing to meet the requirements of standardized processing of clinical research data. For example, when processing an abstract of a clinical trial of an oncology drug, the text contains expressions such as "AE incidence ≥ 5% in cohort A" and "PK parameter ΔAUC > 0.8 kPa". Existing keyword extraction methods fail to correctly interpret "AE" as an abbreviation for adverse event and mishandle the threshold conditions of "≥ 5%" and "> 0.8 kPa", resulting in incomplete extraction of key fields such as safety observation indicators and pharmacokinetic parameters. Furthermore, due to the limited adaptability of the method to mixed Chinese and English and Latin abbreviations, the system is unstable when processing trial summaries from different disease areas, affecting the integrity and reliability of the keyword set.

[0028] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:

[0029] This invention proposes a large-model-driven intelligent keyword extraction method for clinical research ward trial abstracts. It uses a pre-trained large model (such as the BERT model) as the semantic backbone and combines it with a Mixture of Experts (MoE) network to achieve multi-dimensional semantic modeling and automatic keyword extraction of clinical research abstracts. This method aims to solve the problems of insufficient understanding of clinical research abstracts, inaccurate keyword extraction, and poor generalization to multiple diseases and multiple study designs in existing technologies.

[0030] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0031] like Figure 1As shown in the embodiment of this application, a method for intelligent extraction of keywords from clinical trial abstracts based on a large model is provided, including:

[0032] S1: Perform word segmentation on the clinical research abstract to obtain the basic vector.

[0033] S2: Based on the basic vector, structured feature embedding is incorporated to obtain the input vector. The structured feature embedding includes language type feature embedding, comparison symbol feature embedding, unit of measurement feature embedding, and numerical change feature embedding.

[0034] S3: Input the input vector into the pre-trained large model to obtain token-level contextual representations and experimental backbone features.

[0035] S4: Input the token-level context representation, experimental-level backbone features, and task cue vectors into the pre-trained MoE model to obtain experimental-level augmented features.

[0036] S5: Input the experimental enhanced features into the multi-field classification head and the multi-label output head respectively to obtain the probability distribution of the fields and the keyword activation probability.

[0037] S6: Based on the probability distribution of the fields, the activation probability of keywords, and the corresponding dynamic thresholds, obtain the set of structured fields and keywords.

[0038] The large model, MoE model, multi-field classification head, and multi-label output head are pre-trained using parameter inheritance with clinical research corpora, task corpora, and supervised sample data. A load balancing loss based on probability usage-actual allocation is used to constrain the MoE model's training. Parameter inheritance pre-training means that during pre-training, the model inherits the parameters trained in the previous stage as initial values ​​when using different corpora at different stages, thereby accelerating training and improving model performance. First, a basic model is trained on a general corpus to learn common language knowledge. Then, the parameters of this basic model are used as initial values ​​for further training on clinical research corpora, followed by training on task corpora, and finally fine-tuning on supervised sample data. This process gradually adapts to the professional knowledge and specific tasks of clinical research, ensuring the model's professionalism and accuracy in processing clinical research abstracts. During the training of the MoE model, a basic load balancing loss can be adopted. For example, by penalizing the variance of the usage frequency among expert subnetworks, load balancing can be promoted so that each expert subnetwork in the MoE model can be used evenly, so as to avoid some expert subnetworks being overused while others are underutilized, thereby improving the overall efficiency and performance of the model.

[0039] In this optional embodiment, a parameter inheritance-based pre-training strategy and a load-balanced loss based on probability usage-actual allocation are used to constrain the training of the MoE model, ensuring its professionalism, efficiency, and stability. This phased, domain-adaptive pre-training allows the model to better adapt to the professional corpora and task characteristics of clinical research, improving its accuracy. The load-balanced loss optimizes the internal working mechanism of the MoE model, ensuring that each expert subnetwork is effectively utilized, thereby improving the overall performance and generalization ability of the model. By introducing structured feature embeddings, the problem of poor robustness to mixed languages ​​and quantitative conditions is effectively solved. Unlike traditional methods that rely solely on word vectors, this method explicitly encodes structured information such as language type, comparison symbols, units of measurement, and numerical changes into feature embeddings. This enables the model to more accurately understand the comparative meaning of "≥" and the unit attribute of "%" in "decline ≥1.5%", as well as the specific language type of "SAE", thus avoiding extraction errors or omissions caused by these complex elements. By utilizing pre-trained large-scale models and the MoE model, the ability to cover multidimensional and complex elements in clinical research abstracts and its cross-domain generalization capabilities are significantly improved. The large-scale model provides powerful contextual understanding, enabling it to grasp the semantics of the abstract as a whole, such as recognizing the meaning of "Phase III" in the context of "Phase III clinical trial." The MoE model, through its expert sub-network architecture, can efficiently handle diverse information types in clinical research; for example, one expert may focus on processing trial design information, while another may focus on processing safety data. This multi-expert collaborative processing mechanism allows the model to understand the multidimensional combination elements in the abstract more comprehensively and deeply, such as simultaneously identifying research phase, research type, drug name, and adverse events, far exceeding the capabilities of existing methods that only reach the "topic / few terms" level. Through the parallel design of multi-field classification heads and multi-label output heads, the simultaneous extraction of structured fields and keywords is achieved, ensuring the comprehensiveness and accuracy of the information. Compared to single-task methods that only extract keywords or perform classification, this method can simultaneously output structured fields such as "Research Phase: Phase III" and keywords such as "Drug X" and "HbA1c," providing richer and more refined data for the in-depth utilization of clinical research information. In summary, by combining structured feature embedding, large-scale models with MoE models, multi-head classification, and optimized pre-training strategies, this method effectively overcomes the problems of missing elements, unstable generalization ability, and poor robustness faced by existing technologies in processing clinical research abstracts, providing an efficient, accurate, and robust solution for the intelligent extraction of clinical research information.

[0040] like Figure 2 The network architecture of this application includes a large model, a MoE model, a multi-field classification header, and a multi-label output header. The information transmission process is as follows:

[0041] 1. Input: The clinical research abstract or brief research protocol text is used as input. This type of text typically includes structured information such as research background, indications and target population, research objectives, research type and phase, research design, sample size and inclusion / exclusion criteria, dosing and visit / follow-up arrangements, endpoint definition, and key safety observation points. Addressing the common issues in clinical research texts such as mixed Chinese and English, Latin abbreviations, and complex quantitative expressions, the original text undergoes standardized preprocessing. This includes segmentation by paragraph and semantic unit, identification and standardization of Chinese and English terms and Latin abbreviations, and structured labeling of units of measurement and comparison symbols. This provides a more stable input representation for subsequent semantic encoding and structured element extraction.

[0042] 2. Large-Scale Semantic Encoding: This large-scale model is further pre-trained on a large-scale corpus of Chinese and English electronic medical records, clinical trial registration entries, guidelines, and literature, enabling it to better adapt to the clinical research context. Through the constructed clinical domain pre-trained semantic backbone model (i.e., the large-scale model), the input text is segmented and encoded. When segmenting and encoding the preprocessed clinical research text, not only can conventional token-level contextual representations and overall pooled representations be obtained, but also, through specialized hybrid linguistic and quantitative feature embedding, Chinese and English terms, Latin abbreviations, numerical ranges, comparison symbols, and key units of measurement are jointly encoded. This forms trial-level backbone features that simultaneously reflect the overall semantics of the trial, the distribution of elements, and quantitative constraints. This allows the model to perform fine-grained modeling of complex descriptions in clinical research texts, forming trial-level backbone features.

[0043] 3. MoE Hybrid Expert Modeling: Multiple parallel expert subnetworks are constructed based on trial-level backbone features. These subnetworks employ a multilayer perceptron or lightweight Transformer architecture and are divided according to professional specialization: expert subnetworks for drug / device, interventional / observational, real-world / registration research models, and expert subnetworks for different disease systems such as digestive, cardiovascular, oncology, and endocrinology. Specialized expert subnetworks focusing on endpoint types and follow-up models are also included to represent imaging endpoints, laboratory endpoints, PRO scale endpoints, and differences between long and short follow-up periods. The router network scores each expert subnetwork based on semantic features (cue words such as "phase IV" and their context) in the input clinical study abstract. Several of the most relevant expert subnetworks are selected for forward computation, and a weighted combination is used to output the expert-enhanced representation (i.e., trial-level enhanced features) for the current study abstract. By using this router network and expert subnetwork partitioning strategy, it is possible to achieve specialized modeling for multiple disease domains and various research design patterns based on a single trial-level backbone feature. This approach balances performance for both high-incidence indications and indications with relatively scarce samples, overcoming the problem of uneven accuracy in multi-task and multi-domain scenarios of traditional general text models.

[0044] 4. Multi-task Output and Keyword Decoding: In terms of output layer design, a multi-task decoding structure is adopted. The experimental enhancement features output by MoE are simultaneously input into multiple classification heads (multi-field classification heads) and a set of multi-label binary output heads (multi-label output heads). Additionally, sequence labeling or fragment extraction sub-modules can be configured to meet the needs of keyword extraction from clinical research abstracts. The multi-field classification head is used to predict structured discrete fields, including key design features such as study type, disease system and indication category, study stage (Phase I / II / III / IV, exploratory or confirmatory), randomization method, blinding settings, control type, single-center or multi-center attributes, and whether it belongs to a real-world study. The multi-label output head is used to predict elements and keyword sets of high concern in clinical research wards, including whether visits and long-term follow-up are involved, whether data sources include imaging data, wearable device data, laboratory indicators, or questionnaires, the clinical domain of the endpoint (e.g., liver fibrosis, cardiovascular events, tumor remission, survival outcomes), and key safety monitoring organ systems. Meanwhile, by performing feature modeling on descriptions containing quantitative thresholds, we can identify trigger words and constraint structures corresponding to quantitative constraints such as "MRE decreases by ≥20% from baseline", "ALT decreases below ULN", and "HbA1c decreases by ≥1%", thus supplementing key information that traditional methods are prone to miss or misjudge from both keyword and structured element levels.

[0045] 5. Reasoning and Application: For newly input clinical research abstracts or brief research protocols, the model automatically performs preprocessing, semantic encoding, expert routing, and multi-task decoding processes. It outputs structured field predictions including research type, indication, research stage, and design pattern, as well as a set of keywords and structured elements such as visit / follow-up arrangements, endpoints and safety factors, data sources, and quantitative constraints. Simultaneously, it generates corresponding trial-level embedding vectors. By introducing a hybrid expert network and multi-task decoding mechanism onto a unified pre-trained semantic backbone, the model enhances the capabilities of clinical research abstracts in terms of structured representation, multi-dimensional semantic understanding, multi-disease / multi-design adaptation, and medical hybrid language and quantitative condition modeling. This effectively overcomes the shortcomings of existing technologies in clinical research ward scenarios, such as insufficient structure, inadequate adaptation to heterogeneity, and poor robustness in handling abbreviations and quantitative conditions.

[0046] 6. Training and Validation: During training and validation, based on the provided real-world data examples, the study type, indication, study stage, design features, visit and follow-up patterns, endpoints and safety factors, data sources, and keyword sets were systematically labeled. Joint loss was used to train the BERT semantic backbone, MoE expert network, and all output heads end-to-end. To address the uneven distribution of different disease systems and study design categories, task or category weighting and focus loss mechanisms were introduced to improve the ability to identify minority indications and rare design patterns. For mixed language, abbreviations, and complex quantitative conditional expressions, corresponding data augmentation and contrastive learning strategies were designed to improve the model's robustness to new terms, new expressions, and evolving medical vocabulary. During the validation set evaluation phase, the accuracy and F1 score of each structured field, as well as the recall and precision of keywords and feature domains, were monitored simultaneously. The optimal model parameters were selected based on multi-task comprehensive indicators to ensure an overall balance and synergistic optimization between semantic understanding and keyword extraction.

[0047] The following is a detailed description of each step.

[0048] S1: Perform word segmentation on the clinical research abstract to obtain the basic vector.

[0049] Specifically, word segmentation and input representation construction are performed on clinical research abstracts to adapt them to clinical scenarios. For the common issues in abstracts such as mixed Chinese and English, Latin abbreviations, and quantitative conditions containing comparison symbols and units of measurement, WordPiece can be used to obtain a token sequence (i.e., a basic vector). Alternatively, a word segmenter based on dictionary matching or a statistical model can be used to segment the Chinese text, breaking down "clinical research abstract" into "clinical," "research," and "abstract." For English text, word segmentation can be performed based on spaces and punctuation marks. Subsequently, each word is converted into a basic vector, which initially carries the basic semantic information of the word itself.

[0050] S2: Based on the basic vector, structured feature embedding is incorporated to obtain the input vector. The structured feature embedding includes language type feature embedding, comparison symbol feature embedding, unit of measurement feature embedding, and numerical change feature embedding.

[0051] Specifically, the base vector undergoes regular word embedding and position embedding. Building upon this, structured feature embeddings tailored to clinical research summaries are introduced to construct the input vector, enhancing the model's robustness to mixed linguistic and quantitative constraints. The structured feature embeddings aim to supplement specific information that may be missing from the base vector and is crucial for understanding the clinical research summary. These structured feature embeddings include at least: LangType feature embedding: distinguishing between Chinese tokens, English tokens, Latin abbreviation tokens, number tokens, and unit tokens; Comparator feature embedding: encoding comparison symbols such as "≥, ≤, >, <, ="; Unit feature embedding: encoding common clinical units of measurement such as kPa, mg / kg, U / L, mmHg, and %. NumericSpan feature embedding: encoding "threshold / interval / direction of change," including at least lower bound, upper bound, absolute change, and relative change types. These structured feature embeddings can be concatenated or summed with the base vector to form an input vector containing richer information. By using the explicit embedding described above, the model can distinguish elements such as indicator name, comparison symbol, value, unit, and direction at the input layer, thereby reducing the risk of unit mismatch, direction misjudgment, and missed detection when dealing with conditions such as "the endpoint is a decrease of ≥20% in MRE from the baseline or an absolute decrease of ≥0.8 kPa".

[0052] S3: Input the input vector into the pre-trained large model to obtain token-level contextual representations and experimental backbone features.

[0053] Specifically, the constructed input vector is fed into the pre-trained large model to obtain a token-level context representation. And the pooled representation obtained from the [CLS] position or pooling. The large model is pre-trained on a large amount of general text data, enabling it to understand the deep semantics of language. When an input vector is fed into the large model, it generates a token-level contextual representation for each token, capturing the semantic information of the token within its context. Simultaneously, the large model extracts an experimental core feature from the entire input sequence. This step specifically includes the following:

[0054] S310: Input the input vector into the pre-trained large model to obtain token-level context representation and pooling features.

[0055] Specifically, large models (such as those based on the Transformer architecture) process input vectors through multi-layer self-attention mechanisms and feedforward networks, generating a high-dimensional vector for each input token. This vector encodes the relationship between the token and its context, i.e., a token-level context representation. Simultaneously, large models can employ various pooling strategies to generate pooling features. For example, the output vector corresponding to the first specific token in the sequence (such as the [CLS] token) can be extracted as a pooling feature, as this token is typically designed to aggregate the semantic information of the entire sequence. In another implementation, pooling features can be obtained by performing average pooling or max pooling on all token-level context representations to obtain a fixed-length vector as the pooling feature, thereby summarizing the semantics of the entire sequence.

[0056] S320: Concatenate the token-level context representation and pooling features to obtain the experimental-level backbone features.

[0057] Specifically, in order to obtain a test-level semantic backbone representation that reflects the distribution and importance weights of each element in the entire test summary, this method uses token-level representation. The vector is obtained by performing average pooling or attention-weighted pooling. and pooled The main features of the experimental stage are formed by splicing together. :

[0058] ;

[0059] in, As experimental backbone features for clinical research, they are used both as global semantic input for routers and as shared input for multi-field classification headers and multi-label output headers, to ensure that structured field prediction and keyword set prediction are optimized collaboratively in a unified semantic space.

[0060] In this step, a large-scale clinical research model based on pre-trained BERT can be constructed to perform token-level and trial-level semantic representations on the input clinical research abstracts, obtaining stable and reusable clinical research embedding vectors. This fusion of multi-granularity features avoids the limitations of single feature representations, enabling the subsequent MoE model to receive an enhanced feature representation that integrates multi-granularity information. This allows for a more accurate understanding of the complex semantics of clinical research abstracts and provides a more solid foundation for keyword extraction and field classification tasks.

[0061] S4: Input the token-level context representation, experimental-level backbone features, and task cue vectors into the pre-trained MoE model to obtain experimental-level augmented features.

[0062] Specifically, the task cue vector can be a pre-defined, fixed vector related to the current keyword extraction and field classification tasks, used to guide the MoE model to focus its attention on the specific task. The MoE model can employ a routing mechanism, for example, evenly distributing all inputs (a concatenation of token-level contextual representations, experimental backbone features, and the task cue vector) across all expert subnetworks for processing, and then performing a pre-defined weighted average of the outputs of all expert subnetworks to obtain the experimental augmented features. Alternatively, it can be implemented using the following specific methods.

[0063] S410: Input the token-level context representation into multiple expert subnetworks within the MoE model to obtain the output of each expert subnetwork.

[0064] Specifically, token-level contextual representations are input in parallel into multiple expert subnetworks within the MoE model. Each expert subnetwork processes this representation independently, thereby extracting different feature representations from multiple professional perspectives. Each expert is a multilayer perceptron or a lightweight Transformer variant, and their parameters are not shared. The expert subnetworks are categorized according to their expertise based on clinical research ward usage scenarios, including at least: research mode experts: drugs / devices, interventional / observational, real-world / registration, etc.; disease system experts: gastroenterology, cardiovascular, oncology, endocrinology, etc.; and acquisition module triggering experts: imaging endpoints, laboratory indicator endpoints, PRO scale endpoints, ECG QTc, liver stiffness, etc.

[0065] Token-level context representation of a given sample at time t The experts produced the following output results:

[0066] ;

[0067] in, This represents the processing of the e-th expert subnetwork.

[0068] S420: The token-level context representation, experimental backbone features, and task cue vector are concatenated and input into the router network within the MoE model to obtain the calling probability of each expert.

[0069] Specifically, the router is designed to perform expert scoring on each token. To enhance the joint judgment of "local cues + global context" in clinical research abstracts, the router input not only uses the token-level context representation at time t, but also... It also splices together the main features of the experimental stage. And the task hint vector (which can be constructed from pooled vectors or field task hint vectors) forms the routing input:

[0070] ;

[0071] in, This represents the task prompt vector. This indicates a splicing operation.

[0072] Router outputs logits:

[0073] ;

[0074] in, It is a learnable parameter matrix. This represents the score of the e-th expert subnetwork.

[0075] The call probability is obtained through the softmax function:

[0076] ;

[0077] in, Let represent the score of the j-th expert subnetwork, and exp() is the natural exponential function.

[0078] S430: For each token-level context representation, retain a specified number of expert subnetworks with the highest call probability, and set the call probability of the remaining expert subnetworks to zero.

[0079] Specifically, to balance computational efficiency and selective focus, only the token with the highest probability is retained for each token. A group of experts For non The expert probability is set to zero and then normalized again:

[0080]

[0081] in, This means that the highest call probability is selected after sorting all expert subnetworks by their call probabilities. A group of experts; For indicator functions, when the condition " "Take 1 if it is true, otherwise take 0; therefore, This represents the calling probability of the e-th expert subnetwork after top-k truncation of the original calling probabilities. That is, only the calling probabilities of the top-k expert subnetworks are retained, and the calling probabilities of the remaining experts are set to 0, in order to balance computational efficiency and selective focus.

[0082] S440: Based on the calling probability of the retained expert subnetwork and the sum of the calling probabilities of all expert subnetworks, obtain the fusion weight of each retained expert subnetwork.

[0083] ;

[0084] in, This represents the normalized fusion weights used to fuse the outputs of various experts after truncation. This represents the calling probability of the j-th expert subnetwork after top-k truncation of the original calling probabilities. This represents the summation of the probabilities of calling all experts after truncation; It is a very small positive constant used to prevent numerical instability caused by a denominator of 0; through this renormalization, the sum of weights in the expert set is approximately 1, thereby ensuring the stability of the weighted summation scale during fusion.

[0085] S450: Based on the fusion weights and the output of the retained expert subnetwork, experimental-level enhancement features are obtained.

[0086] Specifically, the experimental enhancement features are:

[0087] ;

[0088] in, This represents the token representation enhanced by a hybrid expert mechanism. This represents the output of the e-th expert network (usually a vector). This expression represents the summation over all experts; therefore, it signifies the weighted fusion of the outputs of multiple experts according to their respective fusion weights to obtain the final expert-enhanced token representation, which is then used for subsequent pooling to obtain experimental-level enhanced features. It also serves as the shared input for downstream multi-task heads. A hybrid expert network is introduced, and a router adaptively assigns different abstract modes based on research type, indication, and text style to achieve differentiated and refined modeling for various abstract modes, including digestive system diseases, cardiovascular diseases, oncology, and multicenter real-world studies.

[0089] S5: Input the experimental enhanced features into the multi-field classification head and the multi-label output head respectively to obtain the probability distribution of the fields and the keyword activation probability.

[0090] Specifically, one or more fully connected networks are constructed as multi-field classification heads and multi-label output heads. Therefore, the multi-field classification head and multi-label output head are actually trained fully connected network layers. These two heads work in parallel to extract discriminative information for their respective tasks from experimental augmentation features. For example... Figure 2As shown, each structured field corresponds to an independent set of fully connected networks, and the parameters of the fully connected networks corresponding to each field are not shared. Experimental enhancement features are mapped to the probability distribution of each structured field through independent fully connected branches, achieving simultaneous classification of multiple fields. All keyword tags share a set of fully connected networks, and experimental enhancement features are mapped to the activation probabilities of all keyword tags through the fully connected networks, achieving simultaneous extraction of multiple keywords in a single abstract.

[0091] S6: Based on the probability distribution of the fields, the activation probability of keywords, and the corresponding dynamic thresholds, obtain the set of structured fields and keywords.

[0092] Specifically, you can simply select the field with the highest probability in the field probability distribution as the structured field of the final output. For keyword extraction, you can preset a fixed threshold, such as 0.5. When the activation probability of a keyword is greater than or equal to this fixed threshold, it is included in the keyword set.

[0093] S610: Determine the structured fields of the final output based on the probability distribution of the fields.

[0094] Specifically, a preset probability threshold can be set; when the probability value of a field exceeds this threshold, it is determined as the final output structured field. Alternatively, a maximum probability selection strategy can be adopted, directly selecting the field with the highest probability value as the final output structured field. Furthermore, predefined business rules or domain knowledge can be combined to perform post-processing on the probability distribution to improve the accuracy of determining the structured fields.

[0095] S620: When the keyword activation probability is greater than or equal to the dynamic threshold, output the keyword corresponding to the keyword activation probability to obtain the keyword set.

[0096] Specifically, a dynamic threshold is an adjustable, non-fixed value that can be optimized based on different task requirements, dataset characteristics, or model performance. For example, this dynamic threshold can be obtained during model training or validation by optimizing a specific evaluation metric (such as Macro-F1); alternatively, it can be automatically calculated based on statistical analysis or machine learning methods, according to the characteristics of keywords or their distribution in the corpus. Furthermore, different dynamic thresholds can be set based on the importance of keywords or their context in different fields.

[0097] For determining structured fields, the most suitable fields are identified from their probability distributions based on preset rules or strategies and used as output. Simultaneously, for keyword extraction, a dynamic threshold is introduced instead of a single fixed threshold, making keyword selection more flexible and precise. When the activation probability of a keyword reaches or exceeds the dynamic threshold, the keyword is considered valid and included in the final keyword set. This approach allows the model to adaptively adjust the keyword extraction criteria according to different task requirements or data characteristics, thereby improving the accuracy and recall of keyword extraction. It effectively solves the problem of how to transform the probabilistic information output by the model into accurate and practical structured fields and keyword sets, ensuring the reliability of the intelligent keyword extraction results for clinical research abstracts.

[0098] In an optional embodiment of this application, the large model, MoE model, multi-field classification head, and multi-label output head are pre-trained using a parameter inheritance model with clinical research domain corpora, task corpora, and supervised sample data. A load balancing loss based on probability usage-actual allocation is used to constrain the training of the MoE model. To enhance the model's ability to understand elements such as "research stage / research type / follow-up / endpoint / safety organ system / abbreviation / unit / symbol" in the context of clinical research, a four-stage training strategy is adopted. The parameters of the large model are... (Includes Embedding and Transformer encoder parameters); Multi-task header parameters are multi-field classification header parameters. and multi-label output header parameters The parameters of the MoE model are: (Router parameters) With expert network parameters ); the corpus for stage B is The corpus for stage C task is... Stage D has supervised data. ,in, Represents a digest token sequence. This indicates a multi-category label for a field (which can be a collection of multiple fields). This represents keyword multi-label annotation (0 / 1 vector).

[0099] To ensure visualization of training progress, the following training and evaluation criteria are explicitly adopted in this embodiment, with the verification principle being: the best model is saved based on the optimal comprehensive multi-task metric on the validation set. Output metrics include: Accuracy and Macro-F1 (average classification performance or F1 score for the field) for field tasks; Micro-F1 (overall extraction performance), Macro-F1 (average label extraction performance), and probability ranking capabilities (Micro-ROC-AUC and Macro-PR-AUC) for keyword multi-label tasks; MoE expert utilization (expert_usage) and average load balancer loss (avg_load_balance_loss) are also output to verify whether expert collapse has occurred. All of the above metrics must be recorded on both the validation and test sets during the evaluation phase.

[0100] The pre-training methods for the large model, MoE model, multi-field classification head, and multi-label output head include:

[0101] Phase A: Build the network architecture using the general large model, the general MoE model, and two general fully connected layers, and initialize the parameters in the network architecture.

[0102] Specifically, the general large model is initialized.

[0103] ;

[0104] in, The public parameters come from a general pre-trained model. This stage does not involve the training objective; it only assigns the public parameters to the large model in the network architecture.

[0105] Phase B: Continue pre-training the large model using MLM with corpora from the clinical research domain to adapt the large model to clinical research expressions.

[0106] Specifically, Domain Continuing Pre-training (DAPT) involves continuing MLM pre-training on a mixed Chinese-English corpus of desensitized clinical trial registration entries, trial abstracts, guidelines / literature, etc., to adapt the model to the expression style of clinical research.

[0107] domain corpus MLM (Masked Language Modeling) defines random masking operations. Get the set of masked locations. :

[0108] ;

[0109] ,in A token-level contextual representation of an input text sequence (clinical trial summary). N is the sequence length; This means that the random mask operation selects several positions according to a predetermined mask ratio and strategy, and perturbs the tokens at these positions according to the BERT replacement rules. Indicates to implement The sequence obtained after the operation is the masked sequence of the actual input to the model.

[0110] ,in, This represents the set of indexes of the masked locations; For token location index ( ); Represents the original sequence In position The real token at the location; set This represents the set of indices n for all tokens that are masked at position n, indicating that the MLM loss is calculated only at these masked positions.

[0111] MLM conditional probability:

[0112] ;

[0113] in The parameter is The conditional probability distribution given by the pre-trained language model; This refers to all trainable parameters of the large model (including Embedding layer and Transformer encoder weights, etc.). The model represents the input After encoding, in position The hidden state vector obtained at point (context representation, dimension 1) ); The output layer linear mapping matrix; This is the output layer bias vector; logits vector Normalization is performed on the probability distribution of the vocabulary, thus obtaining... In a given Predicted probability under given conditions.

[0114] Domain-based pre-training loss (cross-entropy / negative log-likelihood):

[0115] ;

[0116] in, For the Domain Continuing Pre-training (DAPT) phase of MLM training loss; Represents domain-specific corpora (such as anonymized clinical trial registration entries, trial summaries, guidelines / documents, etc., unannotated text in a mix of Chinese and English); Indicates sample Sampling from the domain corpus; This indicates taking the expectation of the domain corpus samples; This indicates that only the set of masked locations is considered. Index in Summation; This is the logarithm of the model's predicted probability for the real token; the negative sign indicates minimizing the negative log-likelihood, thereby driving the model to learn the language and terminology expressed in the domain context.

[0117] Optimization goal:

[0118] ;

[0119] in, This represents the optimal model parameters obtained after training. Indicates all optional parameters Searching for the loss function The parameter solution with the minimum value is selected; in actual implementation, iterative updates are performed using optimization algorithms such as gradient descent or AdamW. This allows the loss from pre-training in the domain to continue to decrease, thus making the model more suitable for the expression methods of the clinical research domain.

[0120] Phase C: Adaptive pre-training of the large model obtained in Phase B using the task corpus, so that the large model converges to the task representation distribution.

[0121] Specifically, Task-Adaptive Pre-training (TAPT) involves lightweight pre-training on the task's summary corpus, enabling the model to further converge to the task representation distribution.

[0122] Isomorphic with stage B, only the corpus has changed :

[0123] ;

[0124] in, θ represents the MLM training loss during the Task Adaptive Pre-training (TAPT) phase; θ is the set of trainable parameters for the pre-trained large model. The task corpus refers to the set of summary texts that are consistent with the input distribution of the final processing task in this application. This indicates that sample x was sampled from the task corpus; This indicates the expectation of the task corpus samples.

[0125] Stage C takes the optimal parameters from Stage B and then optimizes them further:

[0126] ;

[0127] in This indicates that the model parameters will be updated to the new parameter values; This indicates searching in the parameter space for the loss function. The parameter solution that takes the minimum value.

[0128] Phase D: Each part of the network architecture obtained in Phase C is trained step-by-step alternately using supervised sample data. The parameters of the network architecture are fine-tuned using the total loss of the network architecture until the total loss converges to the expected range. Specifically, during the step-by-step individual training of the MoE model, a load balancing loss based on probability usage-actual allocation is used to constrain the MoE model's training until both the total loss and the load balancing loss converge to the expected range. Through multi-task joint loss (total loss), the model is forced to learn the constraints between elements (e.g., research phase and PK / PD, liver disease and liver stiffness), thereby improving element completeness and the structured nature suitable for CRF / collection scheme generation.

[0129] Specifically, the large model, MoE model, multi-field classification head, and multi-label output head are trained end-to-end, and the multi-field classification head and multi-label output head are fine-tuned.

[0130] Suppose there are N clinical research abstract samples in the training set, denoted as Ni.

[0131] ;

[0132] in, Indicates the first A sequence of tokens in a digest (i.e., the i-th sample); The structured discrete field label set corresponding to the abstract (e.g., research stage, research type, indication category, center type, randomization method, blinding setting, control type, whether it is a real-world study, etc.) can be represented as a one-hot or category index of multiple fields; Let L represent the set of multi-tag keywords corresponding to this abstract. , This indicates that the l-th keyword is activated in the abstract.

[0133] Model for input After semantic encoding and hybrid expert modeling, two types of predictions are output: 1) For each discrete field m, a category probability distribution is output, with the conditional probability of different categories given by softmax; 2) For the entire keyword set, a multi-label prediction vector is output, with the probability of each keyword being activated given by sigmoid. Simultaneously, experimental enhancement features derived from the aggregation of hybrid experts are used. As an embedding vector for clinical research, it is used for downstream similar study retrieval, CRF template recommendation, and collection scheme generation.

[0134] To prevent the overuse of a few experts from causing the degradation or even ineffectiveness of other experts, this method introduces load balancing loss during the training phase, performing joint regularization on the "probabilistic use" and "actual allocation" of experts within a batch. The batch size is defined as... The sequence length is The number of experts is Specifically, the load balancing loss based on probability-based usage-actual allocation is a weighted sum of the expert usage basic balancing loss, the actual allocation constraint loss, and the sparsity consistency loss.

[0135] The expert used the basic equilibrium loss as

[0136] ;

[0137] ;

[0138] in, This indicates that the experts use a basic equilibrium loss to suppress expert collapse caused by the router concentrating a large number of tokens on a small number of experts; B is the size of the training samples, T is the length of the token-level context representation, and E is the total number of expert subnetworks. Let b be the routing probability assigned to the e-th expert subnetwork at position t; ∈{0,1} represents the attention mask, where 1 represents the real token in the token-level context representation and 0 represents the mask of the padding position in the token-level context representation. This correction ensures that only the real token is counted, so that the probability usage definition is consistent with the actual training / inference. Let represent the probability utilization rate of the e-th expert subnetwork within a training sample; Var(·) represents the variance operation over the expert dimension, that is, calculating the dispersion of each expert's utilization rate as a component. Therefore, when the utilization rates of each expert are approximately uniformly distributed, Approaching a constant of 1, variance Approaching 0 Smaller; when a small number of experts are overused, As differences among different experts increase, variance rises. This increases the penalty for extreme biases during training and promotes a more balanced expert load.

[0139] The actual allocation constraint loss is

[0140] ;

[0141] ;

[0142] in, For indicator functions, if the first... Sample at position Top-k expert collection If the network includes the e-th expert subnetwork, set the value to 1; otherwise, set the value to 0. Let represent the actual average frequency of the e-th expert subnetwork being selected into the top-k expert set, which conforms to the meaning of "hard utilization / actual allocation" and can effectively detect and suppress expert collapse; C represents the expected capacity. ;k represents the number of expert subnetworks in the top-k expert set. This indicates that flexibility is allowed (e.g., 0.1); This represents the activation function.

[0143] The sparse consistency loss is

[0144] ;

[0145] ;

[0146] in, Let be the fusion weight of the e-th expert subnetwork at position t for the b-th sample. This represents the average expert weight distribution vector within the sample. ; This represents the average expert weight corresponding to the e-th expert subnetwork in the average expert weight distribution vector; Represents a uniformly distributed vector. Then the uniform distribution corresponding to the e-th expert subnetwork ; This represents the KL divergence.

[0147] Load balancing loss is

[0148] ;

[0149] in, These are adjustable weights corresponding to the three losses, used to strike a trade-off between "expert balance" and "necessary expertise". A clinical scenario hybrid expert (MoE) model is introduced after the large model, using top-k routing to allow different experts to learn the "local semantic subspace" of different disease systems / research models, and load balancing loss is added to suppress expert collapse, so that effective expert capacity can be obtained even in low-sample domains, improving the stability of segmented Macro-F1.

[0150] Enhanced features at the experimental level To unify the input, structured field prediction and keyword extraction are jointly optimized using a multi-field classification head and a multi-label output head, thereby achieving multi-dimensional semantic understanding of clinical research abstracts. For each discrete field (study stage, study type, indication category, center type, randomization method, blinding setting, control type, and whether it is a real-world study, etc.), a one- or multi-layer fully connected network is constructed as the classification head, outputting the probability distribution of the field.

[0151] ;

[0152] in, Indicates the first The first field (or the first) A linear classification weight matrix (for each classification task) is used to classify samples. Experimental-level enhancement features Mapped to the logits score vector for each category of this field, for the field The classification weights are used to map the shared input to scores (logits) for each category of the field. This represents the bias vector corresponding to this field, used for learnable shifting of the logits for each category, also for this field. The bias term also corresponds to the field. For each category (each category has a bias used to translate the logit of that category), therefore After obtaining the category score vector, the probability distribution of this field in each category is output after passing through the softmax function. .

[0153] Supervised learning is performed based on the standard cross-entropy loss, resulting in the cross-entropy loss of the multi-field classification head.

[0154] ;

[0155] in, This indicates the actual distribution.

[0156] A complete set of keywords reflecting multidimensional semantic features such as follow-up patterns, data sources, endpoints, and safe organ systems. A multi-label output head is constructed, and the multi-label prediction vector is obtained by applying an affine transformation and sigmoid activation.

[0157] ;

[0158] in, This indicates the complete set of keywords. The predicted probability vector, It is a dimension-wise sigmoid function; The weight matrix for the multi-label output head targets each label in the entire keyword set. Each keyword label has a weight row, which is used to determine whether the label should be activated from the shared input. This is a bias vector, applied to each label in the entire keyword set, with one bias per label used to shift the logit of that label; therefore Indicates sample The predicted probability of the s-th keyword tag being activated (to 1) can be used to transform the probability vector into the final multi-tag keyword set output by combining a threshold strategy.

[0159] in It is a dimension-wise sigmoid function. For the sample The Middle The keyword activation probability of each keyword. The multi-label part is trained using binary cross-entropy loss, and the binary cross-entropy loss of the multi-label output head is...

[0160] ;

[0161] in, Keywords In the sample The true activation marker on. By simultaneously optimizing and Within a unified semantic and expert space, it collaboratively learns structured elements such as "trial phase, type, indication, and design features" and keyword sets such as "follow-up / visit, imaging and wearable data, endpoint and safety organ" to improve the problem of structured fields and keywords operating independently.

[0162] Combining multi-field classification, multi-tag keywords, and MoE load balancing, the total loss is constructed as follows:

[0163] ;

[0164] in, This represents the cross-entropy loss of the multi-field classification header. This represents the binary cross-entropy loss for multi-label output headers. This represents the load balancing loss of the MoE model. This represents the balancing weight coefficient. A trainable loss function system is formed, unifying multi-field classification loss, multi-label binary cross-entropy loss, and MoE load balancing loss into a single objective function, supporting continuous optimization and iterative updates on clinical research datasets. During training, AdamW is used in conjunction with linear warmup learning rate scheduling to ensure stable convergence under multi-task, multi-expert composite structures.

[0165] In addition, the pre-trained network architecture was tested using validation and test sets, and the model evaluation metrics output by the network architecture were recorded. The model evaluation metrics included Accuracy and Macro-F1 for field tasks; Micro-F1, Macro-F1, Micro-ROC-AUC, and Macro-PR-AUC for multi-label keyword tasks; and expert utilization and mean load balancing loss within the MoE model.

[0166] During the reasoning phase, for any new input clinical research summary The model sequentially performs semantic encoding, MoE expert routing, and multi-task decoding to obtain the predicted distribution of each discrete field. and multi-label prediction vectors ;Predicting vectors for multiple labels The Each dimension sets a specific dynamic threshold. In validation set testing (before S6), the dynamic threshold is searched to maximize the Macro-F1 for each label, adapting to the actual clinical needs of high-precision structured fields and key elements of high recall.

[0167] The large model is structurally modified with "clinical mixed language and quantitative conditional augmentation" to explicitly encode comparison symbols, numerical ranges, units, abbreviation types, etc., into learnable features. At the same time, "data augmentation + threshold search" is used on the training side, and the optimal threshold is selected for each label individually through threshold search to avoid sacrificing the recall of a few labels due to a uniform threshold.

[0168] In summary, compared with existing technologies, it has the following beneficial effects:

[0169] 1. It has advantages in the accuracy and completeness of keyword extraction from clinical research abstracts.

[0170] This invention employs a large model as the semantic backbone and combines it with hybrid expert-enhanced representation to more fully characterize the key information and implicit semantic relationships in the abstract. In evaluations using simulation data and real samples, the classification accuracy of each structured field and the recall of keywords are superior to those using only a single BERT model or traditional rule-based methods. It also demonstrates good recognition capabilities for information involving complex quantitative conditions and descriptions across sentences and paragraphs.

[0171] 2. It has good generalization ability in multiple disease areas and various research design scenarios.

[0172] By introducing a hybrid expert structure, this invention learns the differences in characteristics of different disease domains and research design patterns within a unified model framework. It can maintain relatively stable understanding performance in various scenarios such as liver disease, digestive system, cardiovascular, oncology, and multi-center real-world studies, and alleviate the significant performance degradation of traditional single models in specific diseases or specific design types.

[0173] 3. It exhibits strong robustness to mixed languages ​​and medical abbreviations.

[0174] Based on domain adaptation and continuous pre-training with a large-scale medical corpus, the model has high semantic sensitivity to mixed Chinese and English expressions and abbreviations such as “MAFLD liver cirrhosis”, “APRI / FIB-4”, “AE / SAE”, “MRI-PDFF”, and “CAP”. It can more clearly distinguish different categories of endpoint indicators, safety observation domains, imaging indicators and laboratory indicators at the keyword level, thereby improving the clinical interpretability of keyword extraction results.

[0175] 4. It can provide higher quality input for downstream CRF design and data acquisition scheme generation.

[0176] The clinical research embedding vector and corresponding structured keyword set output by this invention can be directly used as input features for modules such as CRF template recommendation, similar study retrieval, and automatic generation of data collection plans. This helps reduce the workload of manually reviewing each research plan, improves the consistency and completeness of CRF design, and provides basic support for the intelligent management of subsequent data in clinical research wards.

[0177] 5. It has good deployment and response characteristics under resource-constrained conditions.

[0178] By optimizing the model structure and inference process, and combining it with quantization strategies, this invention reduces computational and storage overhead while ensuring text understanding and keyword extraction performance. It can achieve low-latency inference on a single GPU or mid-to-high-end CPU server, meeting the comprehensive requirements of hospital clinical research centers and clinical research wards in terms of privacy protection, security, and response time.

[0179] The above method is applied to a specific example, using a clinical research abstract of a digestive system disease as an example, to illustrate the specific application process of the method of the present invention.

[0180] One study, titled "A Phase IV Interventional-Device Clinical Trial in Patients with Active Ulcerative Colitis Receiving Standard Treatment," includes the following abstract: 1) Study Subjects: Patients with active ulcerative colitis receiving standard treatment; 2) Study Type and Phase: Phase IV, interventional-device clinical trial, using a non-randomized, open-label, standard-treatment controlled design; 3) Dosing and Follow-up: Fixed-dose parallel group, inhaled administration, according to routine clinical medication records, approximately 277 subjects were enrolled, with a screening period and a baseline period, and follow-up until approximately 104 weeks after administration; 4) Study Objectives: The primary objective was to assess the real-world safety of the investigational device / drug; secondary objectives included assessing the medication modality and resource utilization; exploratory objectives involved real-world efficacy; 5) Endpoints: Primary endpoints included clinical remission / clinical remission rate and incidence of adverse events / serious adverse events (AE / SAE); secondary endpoints included endoscopic mucosal healing rate; exploratory endpoints included fecal inflammatory markers or imaging indicators, etc.

[0181] The execution process of the method in this application is as follows:

[0182] 1. The BERT semantic encoding module performs word segmentation and encoding on the above clinical research text summary to obtain experimental-level backbone features. .

[0183] 2. MoE routers are based on the characteristics of the experimental backbone. The sample was primarily assigned to several expert sub-networks specializing in real-world safety research and digestive system diseases, and experimental-level enhancement features were formed through top-k weighted combination. .

[0184] 3. Multi-field classification header output. Research area: Digestive disease; Indication: Active ulcerative colitis; Research stage: Phase IV; Research type: Interventional-device clinical trial; Follow-up characteristics: Long-term follow-up (104 weeks); Randomization and blinding: Non-randomized, open-label; Control type: Standard treatment control.

[0185] 4. Multi-label keyword header output. The predicted keyword set includes (examples): "including follow-up"; "including safety endpoints"; "including clinical remission endpoints"; "including endoscopic mucosal healing endpoints"; "including fecal inflammatory markers"; "including imaging indicators" and "key safety organs: liver, kidney, cardiovascular, digestive system", etc.

[0186] The clinical research embedding vector of this sample, along with the corresponding set of structured fields and keywords, can be used as input for downstream CRF template recommendation and related clinical research data collection scheme generation algorithms, enabling automatic conversion from natural language summaries to structured elements.

[0187] like Figure 3As shown in the embodiment of this application, a clinical research trial abstract keyword intelligent extraction system based on a large model is provided, comprising:

[0188] Abstract processing module 10 is used to perform word segmentation on clinical research abstracts to obtain basic vectors.

[0189] The input building module 20 is used to incorporate structured feature embeddings into the basic vector to obtain the input vector. The structured feature embeddings include language type feature embedding, comparison symbol feature embedding, unit of measurement feature embedding, and numerical change feature embedding.

[0190] The feature extraction module 30 is used to input the input vector into the pre-trained large model to obtain token-level context representation and experimental backbone features.

[0191] The feature fusion module 40 is used to input the token-level context representation, experimental-level backbone features, and task cue vectors into the pre-trained MoE model to obtain experimental-level enhanced features.

[0192] The multi-head classification module 50 is used to input experimental enhanced features into the multi-field classification head and the multi-label output head respectively to obtain the probability distribution of the fields and the keyword activation probability.

[0193] The output filtering module 60 is used to obtain a set of structured fields and keywords based on the probability distribution of the fields, the activation probability of keywords, and the corresponding dynamic thresholds.

[0194] The pre-training module 70 is used to perform parameter inheritance pre-training on the network architecture using clinical research domain corpora, task corpora and supervised sample data, and to use a probability-based use-actual allocation load balancing loss to constrain the training of the MoE model. The network architecture includes a large model, a MoE model, a multi-field classification head and a multi-label output head.

[0195] In this optional embodiment, by modularly combining structured feature embedding with the large model and the MoE model, complex elements such as mixed Chinese and English, Latin abbreviations, and thresholds with comparison symbols and units in clinical research abstracts are effectively handled. This achieves stable coverage of multi-dimensional combinations of clinical research elements, improves cross-domain generalization ability, and enhances robustness to mixed language and quantitative conditions. Specifically, the pre-training module 70 adopts a parameter inheritance strategy, allowing the network architecture to sequentially adapt to clinical research domain corpora, task corpora, and supervised sample data. At the same time, it utilizes a load-balanced loss constraint MoE model based on probability usage-actual allocation to ensure balanced utilization of expert sub-networks, significantly improving the model's generalization stability in cross-domain scenarios. The language type feature embedding integrated in the input construction module 20 can accurately distinguish between Chinese, English, and Latin abbreviations (such as AE / SAE), the comparison symbol feature embedding can identify the semantic relationships of symbols such as "≥" and "<", the unit of measurement feature embedding accurately parses unit information such as "%" and "kPa", and the numerical change feature embedding captures trend descriptions such as "decreasing" and "increasing". Together, these measures address the robustness deficiencies of existing methods in handling mixed language and quantitative conditions. The feature extraction module 30 and feature fusion module 40 obtain token-level contextual representations through a large model to understand local semantics. Combined with experimental-level backbone features and the multi-expert collaborative mechanism of the MoE model, they achieve global coverage of multi-dimensional elements such as research stage, research type, and random / blind / control methods, avoiding the problem of missing elements. Finally, the output filtering module 60 outputs the predicted structured fields and keyword sets.

[0196] This application provides an electronic device including a memory and a processor; the memory is used to store a computer program; the processor is used to implement the above-described intelligent keyword extraction method for clinical research trial abstracts based on a large model when executing the computer program.

[0197] This application provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the intelligent keyword extraction method for clinical research trial abstracts based on a large model, as described above.

[0198] In this embodiment, the beneficial effects of electronic devices and computer-readable storage media are similar to those of the above-described intelligent keyword extraction method for clinical trial abstracts based on large models, and will not be repeated here.

[0199] The present invention describes electronic devices that can serve as servers or clients of this application, which are examples of hardware devices that can be applied to various aspects of this application. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital assistant devices, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the application described and / or claimed herein.

[0200] Electronic devices include a computing unit that can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) or loaded from a storage unit into random access memory (RAM). The RAM can also store various programs and data required for device operation. The computing unit, ROM, and RAM are interconnected via a bus. Input / output (I / O) interfaces are also connected to the bus.

[0201] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. In this application, the separately described modules may or may not be physically separate. Some or all of the modules can be selected to achieve the purpose of the embodiments of this application according to actual needs. Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0202] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0203] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for intelligent extraction of keywords from clinical trial abstracts based on a large model, characterized in that, include: The clinical research abstract is segmented into words to obtain the basic vector; Based on the basic vector, structured feature embedding is incorporated to obtain the input vector. The structured feature embedding includes language type feature embedding, comparison symbol feature embedding, unit of measurement feature embedding, and numerical change feature embedding. Input the input vector into the pre-trained large model to obtain token-level contextual representations and experimental backbone features; Input the token-level context representation, experimental-level backbone features, and task cue vectors into the pre-trained MoE model to obtain experimental-level augmented features; The experimental enhancement features are input into the multi-field classification head and the multi-label output head respectively to obtain the probability distribution of the fields and the keyword activation probability; Based on the probability distribution of the fields, the activation probability of keywords, and the corresponding dynamic thresholds, a set of structured fields and keywords is obtained. The large model, MoE model, multi-field classification head, and multi-label output head are pre-trained using clinical research corpora, task corpora, and supervised sample data with parameter inheritance, and the MoE model is trained using a load balancing loss based on probability usage-actual allocation.

2. The intelligent keyword extraction method for clinical trial abstracts based on a large model as described in claim 1, characterized in that, The process of inputting the input vector into the pre-trained large model to obtain token-level contextual representations and experimental backbone features includes: Input the input vector into the pre-trained large model to obtain token-level context representation and pooling features; By concatenating the token-level context representation and pooling features, we obtain the experimental-level backbone features.

3. The intelligent keyword extraction method for clinical trial abstracts based on a large model as described in claim 1, characterized in that, The process of inputting token-level context representation, experimental-level backbone features, and task cue vectors into the MoE model to obtain experimental-level enhanced features includes: The token-level context representation is input into multiple expert subnetworks within the MoE model to obtain the output of each expert subnetwork. The token-level context representation, experimental backbone features, and task cue vector are concatenated and input into the router network within the MoE model to obtain the calling probability of each expert. For each token-level context representation, retain a specified number of expert subnetworks with the highest call probability, and set the call probability of the remaining expert subnetworks to zero. The fusion weight of each retained expert subnetwork is obtained based on the calling probability of the retained expert subnetwork and the sum of the calling probabilities of all expert subnetworks. Based on the fusion weights and the output of the retained expert subnetworks, experimental-level augmentation features are obtained.

4. The intelligent keyword extraction method for clinical trial abstracts based on a large model as described in claim 1, characterized in that, The process of obtaining the structured field and keyword set based on the probability distribution of the fields, the keyword activation probability, and the corresponding dynamic threshold includes: Based on the probability distribution of the fields, determine the structured fields of the final output; When the keyword activation probability is greater than or equal to the dynamic threshold, the keyword corresponding to the keyword activation probability is output to obtain the keyword set.

5. The intelligent keyword extraction method for clinical trial abstracts based on a large model as described in claim 1, characterized in that, The pre-training methods for the large model, MoE model, multi-field classification head, and multi-label output head include: Phase A: Build the network architecture using the general large model, the general MoE model, and the general fully connected layer, and initialize the parameters in the network architecture; Phase B: Continue pre-training the large model using MLM with corpora from the clinical research domain to adapt the large model to clinical research expressions; Phase C: Adaptive pre-training of the large model obtained in Phase B using the task corpus, so that the large model converges to the task representation distribution; Phase D: Each part of the network architecture obtained in Phase C is trained step-by-step alternately using supervised sample data, and the parameters of the network architecture are fine-tuned using the total loss of the network architecture until the total loss converges to the expected range. In particular, when training the MoE model step by step, the load balancing loss based on probability use-actual allocation is also used to constrain the training of the MoE model until both the total loss and the load balancing loss converge to the expected range.

6. The intelligent keyword extraction method for clinical trial abstracts based on a large model as described in claim 5, characterized in that, The load balancing loss based on probabilistic usage-actual allocation is a weighted sum of the expert usage basic balancing loss, the actual allocation constraint loss, and the sparsity consistency loss. The expert used the basic equilibrium loss as ； ； Where B is the size of the training samples, T is the length of the token-level context representation, and E is the total number of expert subnetworks. Let b be the routing probability assigned to the e-th expert subnetwork at position t; ∈{0,1} represents the attention mask, where 1 represents the real token in the token-level context representation and 0 represents the mask of the padding position in the token-level context representation; represents the probability utilization rate of the e-th expert subnetwork within a training sample; Var(·) represents the variance operation on the expert dimension; The actual allocation constraint loss is ； ； in, For indicator functions, if the first... Sample at position Top-k expert collection If the network includes the e-th expert subnetwork, set the value to 1; otherwise, set the value to 0. C represents the actual average frequency of the e-th expert subnetwork being selected into the top-k expert set; C represents the expected capacity. ;k represents the number of expert subnetworks in the top-k expert set. This indicates that flexibility is allowed; Indicates the activation function; The sparse consistency loss is ； ； in, Let be the fusion weight of the e-th expert subnetwork at position t for the b-th sample. This represents the average expert weight distribution vector within the sample. ; This represents the average expert weight corresponding to the e-th expert subnetwork in the average expert weight distribution vector; Represents a uniformly distributed vector. ; This represents the KL divergence.

7. The intelligent keyword extraction method for clinical trial abstracts based on a large model as described in claim 5, characterized in that, The total loss is ； in, This represents the cross-entropy loss of the multi-field classification header. This represents the binary cross-entropy loss for multi-label output headers. This represents the load balancing loss of the MoE model. This represents the balancing weight coefficient.

8. The intelligent keyword extraction method for clinical trial abstracts based on a large model as described in claim 1, characterized in that, The pre-trained network architecture was tested using validation and test sets, and the model evaluation metrics output by the network architecture were recorded. The model evaluation metrics included Accuracy and Macro-F1 for field tasks; Micro-F1, Macro-F1, Micro-ROC-AUC, and Macro-PR-AUC for multi-label keyword tasks; and expert utilization and mean load balancing loss within the MoE model.

9. The intelligent keyword extraction method for clinical trial abstracts based on a large model as described in claim 8, characterized in that, Before obtaining the structured fields and keyword set based on the probability distribution of the fields, the keyword activation probability, and the corresponding dynamic threshold, the process also includes: In the validation set test, the dynamic threshold is obtained by searching to maximize the Macro-F1 value for each label.

10. A smart keyword extraction system for clinical research trial abstracts based on a large model, characterized in that, include: The abstract processing module is used to segment clinical research abstracts into words to obtain basic vectors; The input building module is used to incorporate structured feature embeddings into the basic vector to obtain the input vector. The structured feature embeddings include language type feature embedding, comparison symbol feature embedding, unit of measurement feature embedding, and numerical change feature embedding. The feature extraction module is used to input the input vector into the pre-trained large model to obtain token-level context representation and experimental backbone features; The feature fusion module is used to input token-level context representation, experimental-level backbone features, and task cue vectors into the pre-trained MoE model to obtain experimental-level enhanced features; The multi-head classification module is used to input experimental enhanced features into the multi-field classification head and the multi-label output head respectively to obtain the probability distribution of the fields and the keyword activation probability. The output filtering module is used to obtain a set of structured fields and keywords based on the probability distribution of the fields, the activation probability of keywords, and the corresponding dynamic thresholds. The pre-training module is used to pre-train the network architecture using parameter inheritance based on clinical research corpora, task corpora, and supervised sample data, and to use a load balancing loss based on probability use-actual allocation to constrain the training of the MoE model. The network architecture includes a large model, a MoE model, a multi-field classification head, and a multi-label output head.

Images