Children medical big model construction method and device based on PRI hybrid expert system and program product

Abstract

The invention relates to the field of intelligent medical treatment, in particular to a child medical large model construction method and device based on a PRI mixed expert system and a program product. The method includes: acquiring a child clinical sample data set; transmitting the clinical sample data set to a neural network of a PRI hybrid expert system module for training to obtain a child big model of the PRI hybrid expert system; the neural network of the PRI hybrid expert system module comprises an embedding layer, a coding layer, a hybrid expert module, a decoding layer and an output layer, and the hybrid expert module comprises an expert module and a dynamic routing module for semantic reasoning. The application has good clinical value.

Description

Technical Field

[0001] This application relates to the field of intelligent healthcare, specifically to a method, device, program product, and computer-readable storage medium for constructing a large-scale pediatric medical model based on the PRI hybrid expert system. Background Technology

[0002] The diagnosis of childhood diseases has long faced unique challenges, including vague symptom descriptions (infants and young children cannot accurately describe their discomfort), highly heterogeneous etiologies (a complex interplay of genetic, infectious, and environmental factors), and a high proportion of rare diseases. Traditional expert systems primarily rely on manually constructed rule bases and clinical guidelines (such as pediatric treatment guidelines). While these provide standardized diagnostic pathways, they also have drawbacks. For example, knowledge rigidity and generalization are insufficient; expert systems require pre-defined disease rules, making it difficult to cover the dynamics of childhood development and phenotypic variations of rare diseases (such as immune dysregulation syndromes caused by dual gene mutations). When faced with unstructured data (such as vague complaints from parents or image texts), logical reasoning becomes rigid, easily overlooking comorbidities; updates are lagging and data is fragmented. Guidelines relying on manual input are updated slowly (e.g., the discovery cycle for new pathogens or gene mutations is long), and they cannot dynamically integrate multi-source information (such as real-time epidemiological data or individualized gene sequences), resulting in low diagnostic rates for long-tail cases. Therefore, current research is shifting towards large-scale model systems with a hybrid expert (MoE) architecture to achieve collaborative reasoning based on multimodal data (genes, images, medical record texts). However, the MoE expert system still faces challenges, such as the contradiction between computing power and real-time performance, the need for real-time calculation of expert model weights for dynamic routing, and the high cost of deployment at the grassroots level due to the scale of hundreds of billions of parameters, making it difficult for small medical institutions to afford the local computing power requirements. Summary of the Invention

[0003] To address the aforementioned problems, this invention provides a method for constructing a large-scale pediatric medical model based on a PRI hybrid expert system. The PRI mitigation expert system obtained through the PRI expert system construction method of this invention significantly reduces the total amount of parameter computation while maintaining excellent model performance. Specifically, it includes: Obtain a dataset of pediatric clinical samples; The clinical sample dataset is fed into the neural network of the PRI hybrid expert system module for training to obtain a large pediatric model of the PRI hybrid expert system; The neural network of the PRI hybrid expert system module includes an embedding layer, an encoding layer, a hybrid expert module, a decoding layer, and an output layer. The hybrid expert module is a module that includes an expert module and a dynamic routing module for semantic reasoning.

[0004] Optionally, the hybrid expert module comprises N identical network layers stacked together, where N is a natural number greater than 2. Each network layer includes a first module and a second module, which are connected in series. The first and second modules each include an expert module and a dynamic routing module. Each expert module includes K experts, where K is a natural number greater than 1. The number of expert modules in the first module is less than the number of expert modules in the second module. After receiving input data, the first module selects an expert module through the dynamic routing module to perform feature extraction and obtain the output of the first module. The output of the first module is then sent to the second module. The dynamic routing module selects an expert module to perform semantic reasoning and obtains the output of the current network layer. The output of the current network layer is then sent to the next layer until the output of the expert system is obtained.

[0005] Optionally, the expert modules in the first and second modules further include fixed expert modules. One expert is fixed as the fixed expert module, and the remaining experts are dynamic expert modules. After receiving the input data, the first module executes the fixed expert module and the dynamic expert module in parallel. The dynamic expert module selects an expert module through the dynamic routing module to perform feature extraction and obtain the first dynamic expert output result. The fixed expert module performs feature processing on the input data through the fixed expert to obtain the first fixed expert output result. The first dynamic expert output result and the first fixed expert output result are fused and then input to the second module. The second module executes the fixed expert module and the dynamic expert module in parallel to obtain the second dynamic expert output result and the second fixed expert output result. The second dynamic expert output result and the second fixed expert output result are fused to obtain the output of the current network layer.

[0006] Optionally, the first module further includes a feature extraction module, which is connected in parallel with the expert module. After receiving data, the first module sends the data to the feature extraction module and the expert module respectively. The feature extraction module extracts features from the received data to obtain feature output. The expert module selects an expert and performs feature extraction to obtain the expert module output. The feature output and the expert module output are then sent to the second module. The second module obtains the output of the current network layer through the dynamic routing module and the expert module.

[0007] Optionally, the expert module of the first module further includes a feature extraction module. The feature extraction module is parallel to the expert. After receiving data, the first module selects an expert through the dynamic routing module to perform feature extraction to obtain the expert output, or obtains the expert output through dynamic experts and fixed experts. The received data is then processed by the feature extraction module to extract features to obtain feature output. The feature output and the expert output are then fused to obtain the output of the first expert module.

[0008] Optionally, the network layers determine whether to share parameters through calculation. The specific process is as follows: calculate the block influence degree of the network layer to obtain the block influence score; determine whether the network layers are similar based on the block influence score, calculate the difference in the block influence scores of the network layers, and when the difference is greater than a preset threshold, determine that the network layers are similar network layers and share parameters with the similar network layers; otherwise, the network layer obtains the parameters of the current network layer by performing feature processing on the input data.

[0009] Optionally, the first and second modules further include a self-attention layer and a normalization layer. After receiving data, the first module sequentially passes through the self-attention layer and the normalization layer, and then selects an expert module through the dynamic routing module to extract features. The feature data is then processed by the normalization layer to obtain the output of the first module. The output of the first module is then input to the second module, which sequentially passes through the self-attention layer and the normalization layer, and then selects an expert module through the dynamic routing module to perform semantic reasoning to obtain reasoning features. The reasoning features are then processed by the normalization layer to obtain the output of the current network layer. The output of the current network layer is then input to the next layer until the output of the expert system is obtained.

[0010] Optionally, the pediatric clinical sample dataset may include one or more of the following: basic information data, disease diagnosis data, laboratory test data, biochemical test data, treatment data, and question and answer data.

[0011] Optionally, the embedding layer performs word embedding through parallel standard word embedding channels and medical concept channels to obtain embedding vectors, and the embedding vectors are input to the encoding layer for encoding; the word embedding of the medical concept channel is obtained through the hierarchical position information in the knowledge graph; Optionally, the word embedding generation process of the medical concept channel is as follows: obtain pediatric clinical sample data and convert it into a word sequence; query the entity corresponding to each word in the word sequence in the pediatric knowledge graph to obtain the hierarchical encoding vector of the entity; and map the hierarchical encoding vector to the embedding vector of the medical concept channel. Optionally, the hierarchical encoding vector is a medical classification hierarchical encoding vector; Optionally, the query also includes entity determination. When a word is queried in the pediatric knowledge graph, it is determined whether the query result is an entity. If the query result is an entity, the hierarchical encoding vector of the entity is obtained. If the query result is not an entity, the hierarchical encoding vector is set to zero vector. Optionally, the standard word embedding channel and the medical concept channel are used to obtain standard word embeddings and medical concept embeddings. The gating weights are calculated through a dynamic gating network, and the standard word embeddings and medical concept embeddings are fused through the gating weights to obtain an embedding vector. Optionally, the dynamic gating network calculates gating weights by weighting standard word embeddings, medical concept embeddings, and the contextual importance of lexical sequences.

[0012] Optionally, the neural network model is a Transformer, and the hybrid expert module is connected to the feedforward neural network in the Transformer. The multi-head attention in the Transformer obtains the output of the multi-head attention layer through hierarchical perception calculation, and the output of the multi-head attention layer is input to the feedforward neural network and the hybrid expert module. The hierarchical perception calculation is to group the multi-head attention, and each group of attention heads obtains the hierarchical perception attention weight matrix through fixed guidance and preset medical type hierarchical perception. The output of the current attention layer is obtained through the hierarchical perception attention weight matrix. Optionally, the fixed guidance is to query the entity of the word vector in the knowledge graph through the attention score matrix, determine the preset type of the entity, and obtain the attention weight matrix of the layered perception by performing layered perception calculation based on the preset type. Optionally, the hierarchical perception calculation is based on a preset type to calculate the corresponding type relevance bias value; otherwise, the bias value is set to zero to obtain a relevance bias value matrix. The relevance bias value matrix is ​​then fused with the attention score matrix to obtain the hierarchical perception attention weight matrix. The output of the current attention layer is obtained based on the hierarchical perception attention weight matrix.

[0013] Optionally, the loss calculation of the neural network involves: constructing an ideal attention prior matrix, and calculating the alignment loss by comparing the average attention distribution of each attention head in the coding layer with the ideal attention prior matrix. Optionally, the loss calculation also includes differential loss calculation, which calculates the difference loss in attention distribution between different groups, and obtains the attention consistency loss through the difference loss and the alignment loss.

[0014] The purpose of this invention is to provide a computer program product that includes a computer program or instructions, which are executed by a processor to implement the above-described method for constructing a large-scale pediatric medical model based on the PRI hybrid expert system.

[0015] The purpose of this invention is to provide a computer device comprising a memory, a processor, and a computer program or instructions stored in the memory, wherein the computer program or instructions are executed by the processor to implement the above-described method for constructing a large-scale pediatric medical model based on the PRI hybrid expert system.

[0016] The purpose of this invention is to provide a computer-readable storage medium storing a computer program or instructions thereon, which is executed by a processor to implement the above-described method for constructing a large-scale pediatric medical model based on the PRI hybrid expert system.

[0017] Advantages of this invention: 1. This invention proposes a PRI hybrid expert system, which includes multiple modules. Every two modules constitute a network layer. Both the first and second modules include a dynamic routing module and an expert module. Each expert module has K experts. The number of experts in the first and second modules differs, and the number of experts in the first module is less than that in the second module, forming a pyramid. Furthermore, in both the first and second modules, one expert is fixed as the fixed expert, and the remaining experts are dynamic experts. Each time a module is executed, the fixed and dynamic experts are processed in parallel to obtain fixed and dynamic outputs. The dynamic output is obtained by selecting a dynamic expert through dynamic routing to process the data. The fixed and dynamic outputs are merged and then sent to the next module (the second module). The multi-channel processing of the fixed and dynamic experts forms a residual. Additionally, the experts or expert modules in the first module also include parallel feature extraction modules. The feature extraction modules are executed in parallel with the experts or expert modules to obtain the outputs of the feature extraction modules and the experts or expert modules. These two outputs are merged and then sent to the next layer (the second module). Only one module in the first and second modules adds a feature extraction module, while the other module only has an expert module, forming an interval. By configuring PRI as described above, more than 60% of the full parameters can be achieved when only one-quarter of the model's parameters are activated. This can significantly reduce the computing power required by the model, making it suitable for deployment in primary hospitals and demonstrating great clinical application value.

[0018] 2. In order to reduce the total number of model parameters in the PRI system and the amount of computing memory and computing resources (GPU or CPU), the inter-layer similarity is used to determine whether different network layers are functionally similar. This is because there are many redundant network layers in the model with tens of thousands of parameters, and their functions can be ignored. Therefore, when they are calculated to be functionally similar, inter-layer parameter sharing, layer skip connections, or dynamic layer selection are used to reduce the repeated calculation of similar network layers.

[0019] 3. Construct a hierarchical perception dual-channel lexical embedding layer, which integrates general semantics and medical concept hierarchical information, and dynamically adjusts the fusion weights through adaptive gating, so that the model can distinguish and strengthen the representation of key medical entities; 4. Introduce a pediatric semantic relevance bias matrix and hierarchical attention head division of labor in Transformer to transform the entity relationship distance in the knowledge graph into attention prior, guide the model to focus on medical associations, and design a composite loss function to supervise attention consistency and logical chain reasoning. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the process for constructing a large-scale pediatric medical model based on the PRI hybrid expert system, as provided in an embodiment of the present invention. Figure 2 A schematic diagram of a hybrid expert system module provided in an embodiment of the present invention; Figure 3 A schematic diagram of a computer device provided in an embodiment of the present invention; Figure 4 Schematic diagrams of the standard MOE, P-MOE, R-MOE, and I-MOE provided in embodiments of the present invention; Figure 5 A comparison chart of training time / effect of various hybrid expert systems in the industry, provided for embodiments of the present invention; Figure 6 This is a schematic diagram of dual-channel embedding provided in an embodiment of the present invention. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0023] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations appearing in a specific order are included. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The operation numbers, such as S101, S102, etc., are merely used to distinguish different operations and do not represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0024] Figure 1 A schematic diagram of the method for constructing a large-scale pediatric medical model based on the PRI hybrid expert system provided in this embodiment of the invention is shown, specifically including: S1. Obtain a dataset of pediatric clinical samples; In one embodiment, the pediatric clinical sample dataset includes one or more of the following: basic information data, disease diagnosis data, laboratory test data, biochemical test data, treatment data, and question and answer data.

[0025] In one embodiment, the collection of pediatric clinical sample data primarily comprises two data sources: first, anonymized authentic medical and patient data records obtained from collaborating pediatric hospitals or online health consultation platforms; and second, relevant text data collected from publicly available medical databases. All of this text data undergoes rigorous privacy anonymization processing to remove personally identifiable information, ensuring the compliance and security of data use.

[0026] The collected raw text undergoes refined annotation, which mainly includes two aspects: First, medical entity annotation is performed on the text. Professionals with medical backgrounds identify and annotate pediatric medical entities involved in the text according to predefined annotation specifications. The annotation categories must be closely integrated with subsequent steps and mainly include: Symptom description (e.g., "cough" or "abdominal pain"), disease name (e.g., "acute bronchitis" or "allergic rhinitis"), body part (e.g., "throat" or "abdomen"), examination items (e.g., "complete blood count" or "lung auscultation"), treatment method (e.g., "nebulizer therapy" or "oral rehydration salts"), etc. Second, we will label some high-quality question-answer pairs with their intent and logical relationships, such as the disease category to which the question belongs and the logical order of key suggestions in the answer. This labeled data will be used for subsequent loss function construction and model evaluation.

[0027] In one specific embodiment, an external knowledge base is constructed and collected to obtain or construct a structured pediatric medical knowledge graph. This graph should contain rich entities such as pediatric diseases, symptoms, drugs, and examinations, as well as semantic relationships between entities (e.g., "disease-accompanying symptoms" and "drug-treatment of disease"). This knowledge graph can be derived from the extraction and organization of pediatric-related sections in public medical knowledge bases (such as UMLS) or self-built based on authoritative medical textbooks and clinical guidelines. In addition, several key dictionaries and terminology databases are constructed or collected: first, a "pediatric fuzzy expression dictionary," specifically collecting colloquial and vague words commonly used by children and parents to describe discomfort (e.g., "uncomfortable," "stomach upset," "wheezing"); second, a "medical standard terminology dictionary," used to standardize the expression of medical concepts. This is obtained by fine-tuning a general biomedical BERT model (such as BioBERT) on annotated pediatric medical text. Its function is to accurately identify pediatric medical entities defined by the aforementioned annotation categories from the text.

[0028] S2. The clinical sample dataset is input into the neural network of the PRI hybrid expert system module for training to obtain a large model of the PRI hybrid expert system for children; the neural network of the PRI hybrid expert system module includes an embedding layer, an encoding layer, a hybrid expert module, a decoding layer, and an output layer, and the hybrid expert module is a module that includes an expert module and a dynamic routing module for semantic reasoning.

[0029] In one embodiment, the hybrid expert module comprises N identical network layers stacked together, where N is a natural number greater than 2, such as... Figure 2 As shown, each network layer includes a first module and a second module, which are connected in series. The first and second modules include an expert module and a dynamic routing module. The expert module includes K experts, where K is a natural number greater than 1. The number of expert modules in the first module is less than the number of expert modules in the second module. After receiving input data, the first module selects an expert module through the dynamic routing module to perform feature extraction and obtain the output of the first module. The output of the first module is then sent to the second module, where the dynamic routing module selects an expert module to perform semantic reasoning and obtain the output of the current network layer. The output of the current network layer is then sent to the next layer until the output of the expert system is obtained.

[0030] In one embodiment, the expert modules in the first and second modules further include fixed expert modules. One expert is fixed as a fixed expert module, and the remaining experts are dynamic expert modules. After receiving input data, the first module executes the fixed expert module and the dynamic expert module in parallel. The dynamic expert module selects an expert module through a dynamic routing module to perform feature extraction and obtain a first dynamic expert output result. The fixed expert module performs feature processing on the input data through a fixed expert to obtain a first fixed expert output result. The first dynamic expert output result and the first fixed expert output result are fused and then input to the second module. The second module executes the fixed expert module and the dynamic expert module in parallel to obtain a second dynamic expert output result and a second fixed expert output result. The second dynamic expert output result and the second fixed expert output result are fused to obtain the output of the current network layer.

[0031] In one embodiment, the first module further includes a feature extraction module, which is connected in parallel with the expert module. After receiving data, the first module sends the data to the feature extraction module and the expert module respectively. The feature extraction module extracts features from the received data to obtain feature output. The expert module selects an expert and performs feature extraction to obtain the expert module output. The feature output and the expert module output are then sent to the second module. The second module obtains the output of the current network layer through the dynamic routing module and the expert module.

[0032] In one embodiment, the expert module of the first module further includes a feature extraction module, which is parallel to the expert. After receiving data, the first module selects an expert through a dynamic routing module to perform feature extraction to obtain expert output, or obtains expert output through dynamic experts and fixed experts. The received data is then processed by the feature extraction module to extract features to obtain feature output. The feature output and expert output are then fused to obtain the output of the first expert module.

[0033] In one embodiment, the network layers determine whether to share parameters by calculating inter-layer similarity. The specific process is as follows: calculate the block influence degree of the network layer to obtain the block influence score; determine whether the network layers are similar based on the block influence score, calculate the difference between the block influence scores of the network layers, and when the difference is greater than a preset threshold, the network layers are determined to be similar network layers, and parameter sharing is performed on the similar network layers; otherwise, the network layer obtains the parameters of the current network layer by performing feature processing on the input data.

[0034] In one embodiment, the first module and the second module further include a self-attention layer and a normalization layer. After receiving data, the first module sequentially passes through the self-attention layer and the normalization layer, and then selects an expert module through the dynamic routing module to extract features to obtain feature data. The feature data is then processed by the normalization layer to obtain the output of the first module. The output of the first module is then input to the second module, which sequentially passes through the self-attention layer and the normalization layer, and then selects an expert module through the dynamic routing module to perform semantic reasoning to obtain reasoning features. The reasoning features are then processed by the normalization layer to obtain the output of the current network layer. The output of the current network layer is then input to the next layer until the output of the expert system is obtained.

[0035] In one embodiment, experts are selected through dynamic routing in the first and second modules. Data passes through the first and second modules sequentially to form the basic network layer of the P expert system. In the first and second modules, one expert is selected as a fixed expert module, and the remaining experts are dynamic expert modules. An expert is selected through dynamic routing. Data passes through the first and second modules sequentially to form the basic network layer of the PR expert system. The feature extraction module and the expert module in the first module run in parallel, while the second module remains unchanged. After the data passes through the first module, two types of outputs are obtained. The two types of outputs are fused and input to the second module to form the basic network layer of the PRI expert system. N identical PRI network layers are stacked to obtain the PRI hybrid expert system.

[0036] In one specific embodiment, the PRI of the present invention is built in a Transformer, which includes multiple stacked attention modules. Each attention module includes a multi-head attention layer, a normalization layer, a feedforward neural network layer, and a normalization layer. The multi-head attention layer and the feedforward neural network also include residual connections, wherein the PRI of the present invention is connected to the feedforward neural network.

[0037] In one specific embodiment, the large-scale model developed in this invention will be a trillion-parameter MOE structure, employing a self-developed hybrid structure and hybrid expert system, referred to as the PRI structure (Pyramid-Residual-Interval). Compared to the standard MOE architecture, the MOE architecture of this invention does not change the internal structure of the MLP and Attention layers, but only changes the configuration of the hybrid expert MLP layer. Specifically, the Baichuan-MOE structure is a combination of three configuration methods, such as... Figure 4 As shown: The number of MoE experts varies in each layer of Pyramid; Residual-fixed MoE experts are always active; Interval - Only some layers are MoE.

[0038] The time / effect comparison of training based on the above PRI experts and existing expert systems is as follows: Figure 5 As shown, with a parameter count reaching trillions, only about 25% of the total parameters are activated, but the effect of approximately 60% of the total parameters can be obtained.

[0039] In one specific embodiment, the MOE inter-layer similarity calculation of the present invention calculates the importance of each layer through Block Influence (BI), and determines whether it is a functionally similar layer or a redundant layer based on its importance. When it is determined to be a functionally similar layer, parameter sharing, layer skip connection, or dynamic layer selection are performed.

[0040] Parameter sharing: Identify functionally similar or redundant layers and adopt a parameter sharing strategy, that is, let these layers share the same set of parameters, which significantly reduces the total number of parameters in the model, thereby reducing memory usage and accelerating the training process.

[0041] Layer skip connections: By establishing direct skip connections between certain key layers, information can be passed directly across multiple similar or less contributing layers. This preserves the expressive power of important layers while reducing computational paths and improving efficiency. For example, allowing every two layers to share the same set of key-value caches can directly halve the amount of key-value cache stored in the model.

[0042] Dynamic layer selection: The layers participating in the computation are dynamically selected based on the specific content of the input sequence, avoiding full computation of all layers in every forward propagation. This reduces unnecessary computation and allows for targeted use of model resources. For example, it can dynamically select whether the current token enters a layer, or use an early stopping mechanism to exit early in a shallow layer and make predictions.

[0043] The specific content of the sequence includes the sequence sample type, sequence length, and sequence task type.

[0044] In one specific embodiment, for the Transformer that builds the hybrid expert system PRI, its attention layer includes data processing using grouped query attention in terms of width, data processing using multi-head attention and sliding window attention in terms of length, and in terms of depth, attention modules identify functionally similar or redundant layers through inter-layer similarity calculation. For functionally similar layers, parameter sharing, layer skipping connections, or dynamic layer selection are performed.

[0045] Dynamic selection involves selecting layers based on different data sample types and tasks during data processing. As a result, functionally similar layers are not pruned or deleted; instead, data processing is completed through parameter sharing or layer skip connections.

[0046] In one specific embodiment, the model configured for the PRI hybrid expert system (Transform) updates parameters via LoRA during training, introducing low-rank matrices as additional biases to some attention layer parameters to capture task-related changes.

[0047] In one specific embodiment, the model configured with the PRI hybrid expert system (Transform) uses MOFO for neural selection and updating. By selectively updating neurons, the model can focus on improving its performance on specific tasks and avoid catastrophic forgetting. Furthermore, L-regularization, by incorporating the L1 / L2 norm into the loss function, encourages parameter sparsity, thereby reducing model complexity. This approach can help identify the most important features for specific tasks in medical data processing and reduce unnecessary computation, thus improving the model's interpretability and robustness.

[0048] MOFO optimizes specific capabilities of the model by calculating the momentum of model neurons in the PRI hybrid expert system (Transform) and selecting the neuron with the largest momentum for updating.

[0049] In one embodiment, the embedding layer performs word embedding through parallel standard word embedding channels and medical concept channels to obtain embedding vectors, and the embedding vectors are input to the encoding layer for encoding; the word embedding of the medical concept channel is obtained through the hierarchical position information in the knowledge graph; Optionally, the word embedding generation process of the medical concept channel is as follows: obtain pediatric clinical sample data and convert it into a word sequence; query the entity corresponding to each word in the word sequence in the pediatric knowledge graph to obtain the hierarchical encoding vector of the entity; and map the hierarchical encoding vector to the embedding vector of the medical concept channel. Optionally, the hierarchical encoding vector is a medical classification hierarchical encoding vector.

[0050] In one embodiment, the query further includes entity determination. When a word is queried in the pediatric knowledge graph, it is determined whether the query result is an entity. If the query result is an entity, the hierarchical encoding vector of the entity is obtained. If the query result is not an entity, the hierarchical encoding vector is set to zero vector.

[0051] In one embodiment, the standard word embedding channel and the medical concept channel obtain standard word embeddings and medical concept embeddings, and the gating weights are calculated through a dynamic gating network. The standard word embeddings and medical concept embeddings are then fused through the gating weights to obtain an embedding vector. Optionally, the dynamic gating network calculates gating weights by weighting standard word embeddings, medical concept embeddings, and the contextual importance of lexical sequences.

[0052] In one embodiment, the neural network model is a Transformer, and the hybrid expert module is connected to the feedforward neural network in the Transformer. The multi-head attention in the Transformer obtains the output of the multi-head attention layer through hierarchical perception calculation, and the output of the multi-head attention layer is input to the feedforward neural network and the hybrid expert module. The hierarchical perception calculation involves grouping the multi-head attention, and each group of attention heads undergoes a preset medical type hierarchical perception through fixed guidance to obtain a hierarchical perception attention weight matrix. The output of the current attention layer is obtained through the hierarchical perception attention weight matrix. Optionally, the fixed guidance is to query the entity of the word vector in the knowledge graph through the attention score matrix, determine the preset type of the entity, and obtain the attention weight matrix of the layered perception by performing layered perception calculation based on the preset type. Optionally, the hierarchical perception calculation is based on a preset type to calculate the corresponding type relevance bias value; otherwise, the bias value is set to zero to obtain a relevance bias value matrix. The relevance bias value matrix is ​​then fused with the attention score matrix to obtain the hierarchical perception attention weight matrix. The output of the current attention layer is obtained based on the hierarchical perception attention weight matrix.

[0053] In one embodiment, attention calculation includes generating a pediatric semantic relevance bias matrix, obtaining any L positions in the attention score matrix and the corresponding word vectors, where L is a natural number greater than 1, querying the entities of the word vectors in the knowledge graph, calculating the relevance between the entities to obtain a second relevance bias value matrix, fusing the second relevance bias value matrix with the attention score matrix to obtain an attention weight matrix, and obtaining the output of the current attention layer based on the attention weight matrix.

[0054] Optionally, the word vector is a word vector or character vector from a pediatric clinical sample text.

[0055] In one embodiment, the query further includes entity determination, querying the mapping data of the word vector in the knowledge graph, determining whether the mapping data is an entity, calculating the relevance bias value between entities when the determination result is an entity, and assigning the relevance bias value to zero when the determination result is not an entity, thereby obtaining a relevance bias value matrix.

[0056] The preset types include any one or more of the following: symptom-diagnosis, description-signs, examination-symptoms, diagnosis-treatment methods.

[0057] In one embodiment, the loss calculation of the neural network is as follows: construct an ideal attention prior matrix, and calculate the alignment loss by comparing the average attention distribution of each group of attention heads in the coding layer with the ideal attention prior matrix; Optionally, the loss calculation also includes differential loss calculation, which calculates the difference loss in attention distribution between different groups, and obtains the attention consistency loss through the difference loss and the alignment loss.

[0058] In one specific embodiment, children’s descriptive vocabulary coexists with standard medical terminology, and the same medical concept has different importance in different contexts. The embedding layer of the conventional Transformer model uses a single word embedding matrix to map each word to a fixed vector, which cannot distinguish word types or utilize entity hierarchy information in the knowledge graph. This can easily lead to the model having difficulty quickly focusing on key medical entities in the early stages of fine-tuning.

[0059] This invention employs a hierarchical perception dual-channel embedding layer, introducing a medical concept channel on top of standard word embedding. The embedding of the concept channel is modulated by the hierarchical position information of entities in the knowledge graph, and the information from the two channels is dynamically fused through adaptive fusion gating, such as... Figure 6 As shown, the specific steps are as follows: 1) Standard word embedding channel: Lexical serialization and standard word embedding acquisition; Word segmentation of pediatric clinical sample text yields a word sequence, defining... Indicates the first Each sample word sequence is a subset of the dataset. The Middle The word sequence obtained after segmenting each sample; Indicates the first In the nth sample word sequence The word element at each position; This indicates the total length of the sequence.

[0060] Furthermore, by searching the pre-trained word embedding matrix, the standard word embedding vector for each word in the sequence is obtained, represented as:

[0061] In the formula, Indicates word elements The corresponding standard word embedding vector has a dimension of Encodes the general semantics and contextual information of words; The standard word embedding matrix is ​​one of the learnable parameter matrices of the large language model to be built. During model initialization, it typically loads the weights of a large-scale pre-trained language model; these are trainable parameters with dimensions of [missing information]. ; The vocabulary size is determined by the word segmenter and vocabulary of the selected pre-trained language model. For example, the vocabulary size of BERT-base is approximately 30,000. This represents the dimension of the hidden layers of the model, specifically the dimension of the hidden layers of the Transformer model to be built, such as the hidden layers of the pre-trained model BERT-base. It's a fixed 768; This represents a one-hot encoding function, used to convert a word into a one-hot vector corresponding to its position in the vocabulary. The function accepts a word and returns a vector of length [length missing]. The vector is set to 1 at the corresponding vocabulary index position of the accepted lexicon and 0 at the other positions. In the actual implementation, this is accomplished by creating an all-zero tensor and assigning values ​​to the corresponding indices.

[0062] 2) Medical Concept Channel: Medical concept embedding generation; For each word in the word sequence, query the pediatric medical knowledge graph. If the word matches an entity in the knowledge graph, obtain the hierarchical encoding vector of that entity; otherwise, the hierarchical encoding vector is a zero vector. Then, map the hierarchical encoding vector to a medical concept embedding vector using a learnable linear transformation, as follows:

[0063] In the formula, Indicates word elements The corresponding medical concept embedding vector has a dimension of It is derived from the hierarchical information of entities in the knowledge graph, representing the structured and hierarchical attributes of lexical units as medical concepts; Indicates word elements The corresponding entities in the knowledge graph The hierarchical encoding vector in the knowledge graph has each dimension corresponding to a medical classification level, and its value is the normalized position index of the entity at that level. Knowledge graphs can be obtained from [the source]. It is necessary to predefine the position index of each entity in multiple medical classification levels (e.g., system -> organ -> disease -> symptom) by querying terms. For the corresponding entity, its predefined hierarchical encoding vector can be obtained. For example, the normalized hierarchical encoding of the entity "acute bronchitis" may be [respiratory system=0.2, lower respiratory tract=0.5, inflammatory diseases=0.7, infectious diseases=0.9]; The weight matrix of the concept embedding layer is a trainable parameter with dimension O(n). ; The bias vector representing the concept embedding layer is a trainable parameter with dimension O(n). ; Represents the hierarchical encoding vector The dimensions and values ​​correspond to the multi-level classification system commonly used in medical knowledge, such as: [body system, anatomical location, disease category, disease subcategory, symptom type, signs, examination, treatment] These are the 8 levels.

[0064] 3) Adaptive fusion gating weight calculation; Based on the standard word embeddings, medical concept embeddings, and their local context of the current lexical units, a gating network is used to calculate dynamic gating weights, dynamically determining whether the final representation should rely more on general semantics or domain knowledge, as shown below:

[0065] In the formula, Indicates word elements The fusion weights are used to fuse the dynamic gating weights embedded in the two channels, with a value range of [value range missing]. ; The parameter vector of the adaptive fusion gating network is a trainable parameter with dimension . ; express Transpose of; The weight matrix of the adaptive fusion gating network is a trainable parameter with dimension . ; Indicates word elements The local context vector is used to capture local context information and has a dimension of . It is obtained by average pooling of the standard word embeddings of all lexical units within the window, and is represented as follows: ; This represents the radius of one side of the pooling window. It is a preset hyperparameter that defines the range of the local context. Examples of possible values ​​are provided. ; This indicates the position index of a word, specifically referring to the position index of the current word in the sequence; Indicates word elements The corresponding standard word embedding vector, Indicates the first The word at position j in a sequence of sample words; This represents the dimension of the intermediate layer of the gated network, with examples of possible values. .

[0066] 4) Dual-channel embedding fusion and positional encoding addition; The standard word embeddings and medical concept embeddings are weighted and summed based on the fusion weights to obtain a fused word embedding vector. Then, the fused word embedding vector is added to the standard positional encoding vector to form the input representation sequence of the first layer of the Transformer model, as follows:

[0067] In the formula, Indicates the first The first sample The fused word embedding vector of each word is the gated fusion embedding vector with dimension 1. The calculation method is expressed as ; Indicates the first The first sample The fused word embedding vector of each word; Indicates the first The representation sequence of *n* samples is the input representation sequence fed into the first layer of the Transformer encoder, with dimensions of *n*. ; Indicates the first The standard positional encoding vector for each position is used to inject the sequence order information into the model and is a trainable parameter; Indicates the first The standard location encoding vector for each position.

[0068] In one specific embodiment, the self-attention mechanism of the standard Transformer treats all word pairs equally when calculating the association between word pairs, ignoring specific semantic constraints in pediatric consultation, such as the stronger association between symptom description and disease diagnosis word pairs, and the strong prior correlation between entities with direct relationships in the knowledge graph during attention calculation.

[0069] Conventional multi-head attention lacks the utilization of such prior domain knowledge, making it difficult to quickly learn these important connections in fine-tuning scenarios with limited data.

[0070] This invention improves upon standard scaled dot product attention by employing a learnable pediatric semantic relevance bias matrix. This matrix is ​​dynamically generated from the relational distances between medical entities corresponding to lexical terms in the knowledge graph, thereby guiding the model to focus on more medically relevant lexical pairs. Simultaneously, it adopts a hierarchical attention head division of labor, allowing different attention heads to focus on different levels of medical semantic associations. The specific steps are as follows: 1) Calculation of basic attention score; For the input sequence representation of each Transformer layer, the query matrix, key matrix, and value matrix are first obtained through linear transformation, and then the scaling dot product attention score is calculated, expressed as:

[0071] In the formula, This represents the basic attention score matrix, with dimensions of . The first of the basic attention score matrix Line number Column elements Indicates position Position Basic attention weights; Indicates the first The query matrix of the layer is calculated as follows: ; Indicates the first The key matrix of the layer is calculated as follows: ; express Transpose of; This represents the dimension of the key vector, used to scale the dot product result and prevent excessively large values ​​from causing gradient instability. A specific value is preferred. ; Indicates the number of heads of attention; Indicates the first Each sample was processed before The output of the layer Transformer is the first layer. Each sample was processed before The output sequence after processing by the layer Transformer encoder is used as the first... The input to the layer attention mechanism has a dimension of ; This represents the layer index of the Transformer encoder; Indicates the first The weight matrix used to generate the query matrix in the layer is a trainable parameter with dimension 1. ; Indicates the first The weight matrix used to generate the key matrix in the layer is a trainable parameter with dimension 1. .

[0072] At the same time, define Indicates the first The value matrix of the layer is calculated as follows: ; Indicates the first The weight matrix used to generate the value matrix in the layer is a trainable parameter with dimension 1. ; The dimension of the value vector is preferably set to... .

[0073] 2) Dynamic generation of the pediatric semantic relevance bias matrix; For any two positions in the attention score matrix, query the entity in the knowledge graph corresponding to their respective words. If both words correspond to entities, calculate the semantic relevance bias based on their relational distance in the knowledge graph; otherwise, the semantic relevance bias is zero, as shown below:

[0074] In the formula, Indicates the first The layer attention mechanism applies to the location Position Pediatric semantic relevance bias value; Represents the position in the sequence lexical units in knowledge graphs The corresponding entity in the knowledge graph is determined based on the input lexical sequence currently being processed by the l-th layer Transformer. The corresponding entity in the middle; Represents the position in the sequence lexical units in knowledge graphs The corresponding entity in the middle; Representing entities and In knowledge graph The relational distance in the equation is defined as the number of edges traversed by the shortest path between two entities. Indicates the first The layer dependency scalar parameter is a trainable parameter used to control the overall dependency of the layer on the semantic bias. This represents the temperature hyperparameter, used to adjust the decay rate of the influence of the relational distance on the bias value. A larger value results in faster decay. Example values ​​are provided. ; This represents the natural exponential function.

[0075] It should be noted that during the construction of the "medical concept channel," a mapping relationship between lexical units and knowledge graph entities has been established. For each lexical unit in the input lexical unit sequence currently being processed by the l-th layer Transformer, the knowledge graph will be queried. If the word element Able to match knowledge graphs If a standard medical entity in the word "fever" matches the entity "fever", then... That is, the entity, if the word element If no entity can be matched, such as punctuation marks, the common verb "is", or non-medical terms not included in the knowledge graph, then... Considered empty or not part of the entity set That is, the semantic relevance bias value for pediatric cases corresponding to "other situations". The value is 0. Based on this, and During the preprocessing and embedding layer construction stages, external knowledge graphs are queried. , which are the words in the input sequence and word elements When calculating attention bias, the location is checked in relation to the corresponding medical entity. and location Have all the lexical terms been successfully associated with entities? If so, then based on the entity... and entity Calculate the bias value for relational distances in a knowledge graph.

[0076] Furthermore, define Indicates the first The pediatric semantic relevance bias matrix of the layer has dimensions of Pediatric semantic relevance bias matrix The Line number Column elements are .

[0077] 3) Bias fusion and attention output calculation; The dynamically generated pediatric semantic relevance bias matrix is ​​element-wise added to the basic attention score matrix, and then normalized using the Softmax function to obtain the attention weight matrix. This weight matrix is ​​then used to perform a weighted summation of the value matrix to obtain the output of this layer's attention mechanism, represented as:

[0078] In the formula, Indicates the first The output of the layer with a pediatric semantic bias attention mechanism incorporates a medical knowledge-driven prior relevance bias into the standard attention mechanism, representing the aggregation result of contextual information guided by domain knowledge. When guiding the model to calculate attention, it tends to focus on word pairs that are strongly related in medical knowledge, thereby accelerating the model's learning in data-scarce scenarios and improving the medical logic of its reasoning. Indicates the first The attention weight matrix of the layer, the final attention weight matrix after fusing pediatric semantic bias, is calculated as follows: Attention weight matrix The element in the t-th row and s-th column is Characterizing position Position Normalized attention weights; This represents the function that performs Softmax normalization row by row, ensuring that the sum of the attention weights of each position to all positions is equal. .

[0079] 4) Implementation of attention head division of labor in hierarchical perception; In multi-head attention mechanisms, different medical semantic constraints are applied to different attention heads to achieve functional division of labor. Specifically, focus on the number of heads The attention heads are divided into several groups, each responsible for focusing on a specific type of association between medical entities, such as "symptom-diagnosis" associations or "description-sign" associations. For attention heads focusing on "symptom-diagnosis" associations, attention is only given when the position... and location Corresponding entity and The pediatric semantic relevance bias value is calculated only when the symptoms belong to the "symptom" category and the disease category, respectively. Otherwise, the bias is zero. For attention heads that focus on the "description-sign" association, the bias is zero only if the position... and location Corresponding entity and The pediatric semantic relevance bias value is calculated only when the terms belong to the categories of "patient descriptive words" and "clinical signs," respectively. The bias calculation rules for other attention heads remain unchanged; In practice, entity category information is derived from a knowledge graph. The predefined category labels of the entities in the model guide the different attention heads to focus on different levels of medical logical relationships through this division of labor. Finally, the outputs of all attention heads are processed by Transformer layer components such as concatenation and feedforward networks, and hybrid expert modules to obtain the final output of this layer. Specifically, firstly, the output calculated for each attention head... The outputs are concatenated along the last dimension. The concatenated result is then passed through a learnable linear projection layer to unify the dimensions. Residual connections and layer normalization are then applied, and the normalized result is input into a standard feedforward neural network. Finally, residual connections and layer normalization are applied again, resulting in the final representation. Layer output ; in, Indicates the first The sample was processed by the first... The output representation sequence is processed by a layer-by-layer perceptual Transformer encoder, which includes a multi-head attention and feedforward network with pediatric semantic bias.

[0080] In one specific embodiment, during the model training phase, conventional fine-tuning methods typically only use the cross-entropy loss function, whose core objective is to maximize the probability of predicting the next token. However, in the pediatric consultation scenario, this method has two limitations: First, it fails to effectively supervise whether the model's internal representations, especially the knowledge-enhanced attention mechanism, truly learn and follow the logical connections of pediatric medicine; second, it fails to fully utilize the structured knowledge injected during the preprocessing phase, such as entity alignment results and knowledge statement sets, to construct more refined supervision signals.

[0081] This invention employs a composite supervised loss function, which not only includes the standard sequence generation loss but also two domain-knowledge-based regularization losses. It aims to guide model learning from two dimensions: internal attention consistency and external logical chain reasoning, ensuring that the model output conforms to both linguistic rules and strictly follows the inherent logic of pediatric medical knowledge. The specific steps are as follows: In the hierarchical perception multi-head attention mechanism, different attention heads are designed to focus on specific types of medical entity associations. An ideal model should exhibit intra-group consistency and inter-group specificity in attention patterns; that is, the attention distribution of attention heads within the same group towards strongly associated medical entities should be similar, while the attention focus of attention heads in different groups should differ. To achieve this goal: This invention first utilizes the entity alignment results from the preprocessing stage and the entity category information from the knowledge graph to construct an ideal attention prior matrix for each sample in a batch. This matrix identifies the word pairs that the model should give high attention to. Then, it calculates the average attention distribution of each group of attention heads in a specific layer and encourages them to move closer to the ideal prior through alignment loss, while simultaneously encouraging the separation of attention distributions between different groups through differentiation loss. Finally, it calculates the attention consistency loss, expressed as:

[0082] In the formula, It represents the loss of attentional consistency, used to measure and guide the degree to which the model's internal attention mechanism learns and follows prior knowledge in pediatric medicine; This indicates the training batch size, which is the number of samples contained in the training batch. This indicates the total number of attention head groups, for example, if grouped by "symptom-diagnosis", "description-signs", and "other". ; Indicates the first The weight coefficients of each attention head group are trainable parameters used to balance the importance of different groups in the total loss function. This represents the Kullback-Leibler divergence, used to measure the difference between two probability distributions; Indicates temperature coefficient The row-wise Softmax normalization function, temperature coefficient Used to control the smoothness of the output distribution; a value less than 1 makes the distribution sharper, and a value greater than 1 makes the distribution smoother. Indicates the first The sample at the th The average attention matrix within a group of attention heads is obtained by averaging the attention weight matrices of all attention heads within that group. Indicates the first The sample at the th The average attention matrix within each attention head group; Indicates the first The ideal attention prior matrix for each sample has dimensions of . It is constructed from the fuzzy word alignment results in the preprocessing stage and the entity category information of the knowledge graph, and is used to identify the word pairs that the model should pay high attention to; Indicates the first Attention mask matrices for attention heads, with dimensions of . Based on the predefined medical entity association types for this grouping, irrelevant associations are filtered out when calculating the alignment loss, for two positions in the sequence. and location Query its corresponding entity and entity In a knowledge graph, if the category labels are... The category combination conforms to the first The type of group focus (e.g., "symptom-diagnosis"), then The corresponding Set it to 1 otherwise to 0, and then when calculating the Kullback-Leibler divergence, it is compared with the ideal attention prior matrix. Element-wise multiplication is used to filter out irrelevant associations; This represents the element-wise multiplication operator; The weight hyperparameter represents the differential loss, which controls the contribution ratio of the differential loss term to the total loss function. An example value is 0.1. Index representing the attention head group; Indicates difference from The index of the attention head group; Let represent the Jensen-Shannon divergence, a symmetric smoothed version of the KL divergence, used to measure the similarity between two distributions, with a range of . ; Indicates the first The cross-batch average attention matrix of the attention head group, within a training batch, for the _th ... Calculate the number of samples. The average attention matrix of the group Then, the average attention matrix of all samples in the batch is averaged over the batch dimension to obtain the first... The average attention matrix across batches for each attention head group ; This represents the scaling factor, used to adjust the sensitivity of the differential loss to the Jensen-Shannon divergence value; an example value is 1.0.

[0083] In the specific implementation, for any two positions in the sequence and location If their lexical units and word elements All correspond to entities in the knowledge graph, and (a) in fuzzy word alignment, Alignment from fuzzy words to (or vice versa); or (b) in knowledge graphs In the middle, their relational distance If the value is less than a preset threshold (e.g., 1 or 2), then within the ideal attention prior matrix... The corresponding Set it to 1, otherwise set it to 0, thus identifying word pairs that "should be given high attention".

[0084] The PRI hybrid expert system constructed in this invention has a large model of children that can serve as a base model. For data from newly added pediatric fields, the base model can be fine-tuned to complete various types of tasks.

[0085] The present invention also discloses a computer program product or system, including a computer program that, when executed by a processor, implements the steps of the above-described PRI hybrid expert system construction method or the above-described pediatric medical large model construction method based on the PRI hybrid expert system.

[0086] The pediatric medical large-scale model construction system based on the PRI hybrid expert system provided in this embodiment of the invention specifically includes: Acquisition Unit: Acquiring a dataset of pediatric clinical samples; Training Unit: The clinical sample dataset is input into the neural network of the PRI hybrid expert system module for training to obtain the large pediatric model of the PRI hybrid expert system; The neural network of the PRI hybrid expert system module includes an embedding layer, an encoding layer, a hybrid expert module, a decoding layer, and an output layer. The hybrid expert module is a module that includes an expert module and a dynamic routing module for semantic reasoning.

[0087] Figure 3 An embodiment of the present invention provides a schematic diagram of a computer device, specifically including: The system includes a memory and a processor; the memory is used to store program instructions; the processor is used to invoke the program instructions, which are executed when any of the above-described PRI hybrid expert system construction methods or the above-described PRI-based pediatric medical large-scale model construction methods are executed.

[0088] The present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, is either the above-described PRI hybrid expert system construction method or the above-described pediatric medical large model construction method based on the PRI hybrid expert system.

[0089] The verification results of this verification embodiment show that assigning inherent weights to indications can improve the performance of this method compared to the default settings. Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of devices or units, and may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separated; the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of this embodiment. Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, which may include: read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0090] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0091] The computer device provided by the present invention has been described in detail above. For those skilled in the art, there will be changes in the specific implementation and application scope based on the ideas of the embodiments of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for constructing a large-scale pediatric medical model based on the PRI hybrid expert system, characterized in that, include: Obtain a dataset of pediatric clinical samples; The clinical sample dataset is fed into the neural network of the PRI hybrid expert system module for training to obtain a large pediatric model of the PRI hybrid expert system; The neural network of the PRI hybrid expert system module includes an embedding layer, an encoding layer, a hybrid expert module, a decoding layer, and an output layer. The hybrid expert module is a module that includes an expert module and a dynamic routing module for semantic reasoning.

2. The method for constructing a large-scale pediatric medical model based on the PRI hybrid expert system according to claim 1, characterized in that, The hybrid expert module comprises N identical network layers stacked together, where N is a natural number greater than 2. Each network layer includes a first module and a second module, which are connected in series. The first and second modules each include an expert module and a dynamic routing module. Each expert module contains K experts, where K is a natural number greater than 1. The number of expert modules in the first module is less than the number of expert modules in the second module. After receiving input data, the first module selects an expert module through the dynamic routing module to perform feature extraction and obtain the output of the first module. The output of the first module is then sent to the second module, where the dynamic routing module selects an expert module to perform semantic reasoning and obtain the output of the current network layer. The output of the current network layer is then sent to the next layer until the output of the expert system is obtained.

3. The method for constructing a large-scale pediatric medical model based on the PRI hybrid expert system according to claim 2, characterized in that, The expert modules in the first and second modules also include fixed expert modules. One expert is fixed as the fixed expert module, and the remaining experts are dynamic expert modules. After receiving input data, the first module executes the fixed expert module and the dynamic expert module in parallel. The dynamic expert module selects an expert module through the dynamic routing module to perform feature extraction and obtain the first dynamic expert output result. The fixed expert module performs feature processing on the input data through the fixed expert to obtain the first fixed expert output result. The first dynamic expert output result and the first fixed expert output result are fused and then input to the second module. The second module executes the fixed expert module and the dynamic expert module in parallel to obtain the second dynamic expert output result and the second fixed expert output result. The second dynamic expert output result and the second fixed expert output result are fused to obtain the output of the current network layer.

4. The method for constructing a large-scale pediatric medical model based on the PRI hybrid expert system according to claim 2 or 3, characterized in that, The first module also includes a feature extraction module, which is connected in parallel with the expert module. After receiving data, the first module sends the data to the feature extraction module and the expert module respectively. The feature extraction module extracts features from the received data to obtain feature output. The expert module selects an expert and performs feature extraction to obtain the expert module output. The feature output and the expert module output are sent to the second module. The second module obtains the output of the current network layer through the dynamic routing module and the expert module. Optionally, the expert module of the first module further includes a feature extraction module. The feature extraction module is parallel to the expert. After receiving data, the first module selects an expert through the dynamic routing module to perform feature extraction to obtain the expert output, or obtains the expert output through dynamic experts and fixed experts. The received data is then processed by the feature extraction module to obtain feature output. The feature output and the expert output are then fused to obtain the output of the first expert module. Optionally, the first module and the second module further include a self-attention layer and a normalization layer. After receiving data, the first module passes through the self-attention layer and the normalization layer in sequence, and then selects an expert module through the dynamic routing module to extract features to obtain feature data. The feature data is then processed by the normalization layer to obtain the output of the first module. The output of the first module is then input to the second module, which first passes through the self-attention layer and the normalization layer in sequence, and then selects an expert module through the dynamic routing module to perform semantic reasoning to obtain reasoning features. The reasoning features are then processed by the normalization layer to obtain the output of the current network layer. The output of the current network layer is then input to the next layer until the output of the expert system is obtained. Optionally, the neural network of the hybrid expert module can be configured with dynamic GPU and / or CPU scheduling and model parameter storage paths to complete the GPU and / or CPU usage and model parameter memory usage for dynamic inference of the neural network.

5. The method for constructing a large-scale pediatric medical model based on the PRI hybrid expert system according to claim 1, characterized in that, The embedding layer performs word embedding through parallel standard word embedding channels and medical concept channels to obtain embedding vectors, which are then input to the encoding layer for encoding. The word embedding of the medical concept channel is obtained through the hierarchical position information in the knowledge graph. Optionally, the word embedding generation process of the medical concept channel is as follows: obtain pediatric clinical sample data and convert it into a word sequence; query the entity corresponding to each word in the word sequence in the pediatric knowledge graph to obtain the hierarchical encoding vector of the entity; and map the hierarchical encoding vector to the embedding vector of the medical concept channel. Optionally, the hierarchical encoding vector is a medical classification hierarchical encoding vector; Optionally, the query also includes entity determination. When a word is queried in the pediatric knowledge graph, it is determined whether the query result is an entity. If the query result is an entity, the hierarchical encoding vector of the entity is obtained. If the query result is not an entity, the hierarchical encoding vector is set to zero vector. Optionally, the standard word embedding channel and the medical concept channel are used to obtain standard word embeddings and medical concept embeddings. The gating weights are calculated through a dynamic gating network, and the standard word embeddings and medical concept embeddings are fused through the gating weights to obtain an embedding vector. Optionally, the dynamic gating network calculates gating weights by weighting standard word embeddings, medical concept embeddings, and the contextual importance of lexical sequences.

6. The method for constructing a large-scale pediatric medical model based on the PRI hybrid expert system according to any one of claims 1-5, characterized in that, The neural network model is a Transformer. The hybrid expert module is connected to the feedforward neural network in the Transformer. The multi-head attention in the Transformer obtains the output of the multi-head attention layer through hierarchical perception calculation. The output of the multi-head attention layer is then fed to the feedforward neural network and the hybrid expert module. The hierarchical perception calculation involves grouping multi-head attention, with each group of attention heads undergoing preset medical type hierarchical perception through fixed guidance to obtain a hierarchical perception attention weight matrix, and then obtaining the output of the current attention layer through the hierarchical perception attention weight matrix. Optionally, the fixed guidance is to query the entity of the word vector in the knowledge graph through the attention score matrix, determine the preset type of the entity, and obtain the attention weight matrix of the layered perception by performing layered perception calculation based on the preset type. Optionally, the hierarchical perception calculation is based on a preset type to calculate the corresponding type relevance bias value; otherwise, the bias value is set to zero to obtain a relevance bias value matrix. The relevance bias value matrix is ​​then fused with the attention score matrix to obtain the hierarchical perception attention weight matrix. The output of the current attention layer is obtained based on the hierarchical perception attention weight matrix.

7. The method for constructing a large-scale pediatric medical model based on the PRI hybrid expert system according to claim 6, characterized in that, The loss calculation of the neural network is as follows: Construct an ideal attention prior matrix, and calculate the alignment loss by comparing the average attention distribution of each attention head in the coding layer with the ideal attention prior matrix; Optionally, the loss calculation also includes differential loss calculation, which calculates the difference loss in attention distribution between different groups, and obtains the attention consistency loss through the difference loss and the alignment loss.

8. A computer program product comprising a computer program or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the method for constructing a large-scale pediatric medical model based on the PRI hybrid expert system as described in any one of claims 1-7.

9. A computer device comprising a memory, a processor, and a computer program or instructions stored in the memory, characterized in that, The computer program or instructions are executed by the processor to implement the method for constructing a large-scale pediatric medical model based on the PRI hybrid expert system as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, The computer program or instructions are executed by the processor to implement the method for constructing a large-scale pediatric medical model based on the PRI hybrid expert system as described in any one of claims 1-7.

Images