Children medical big data processing method and device based on distributed mode training architecture and program product

Abstract

The invention relates to the field of intelligent medical treatment, in particular to a child medical big data processing method and device based on a distributed mode training architecture and a program product. Comprising the steps of obtaining a pre-training language model with frozen bottom layer parameters, and newly adding children inquiry training data; and outputting the newly added children inquiry training data to the pre-training language model for training, performing loss monitoring through a verification set in the newly added training data, and dynamically controlling the pre-training model to unfreeze bottom layer parameters of a network layer to obtain a trained children inquiry large model. The method has good application value.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent medical treatment, in particular to a child medical big data processing method, device, program product and computer readable storage medium of a distributed mode training architecture. BACKGROUND

[0002] With the in-depth application of artificial intelligence technology in the fields of education, medical treatment and the like, the value mining of child big data has become a research hotspot. The distributed training framework is widely used in child behavior analysis, personalized education recommendation and the like due to its ability to efficiently process massive data. The current research has realized the joint training of cross-modal data (such as voice, image, learning record) and explored privacy protection technologies such as federated learning. However, the existing framework has significant drawbacks: firstly, child data has strong time sequence and sparsity, and the traditional distributed framework (such as TensorFlow, PyTorch) is not suitable for non-uniform data distribution; secondly, the communication overhead and synchronization mechanism of the existing framework cannot meet the real-time requirement of child interaction scenes, and the balance between privacy protection and model performance still needs to be optimized. SUMMARY

[0003] In view of the above problems, the present application provides a distributed child medical inquiry large model construction method, which specifically comprises: obtaining a pre-trained language model with frozen bottom layer parameters and new child inquiry training data; feeding the new child inquiry training data into the pre-trained language model for training, monitoring the loss through the validation set in the new training data, dynamically controlling the bottom layer parameters of the unfreezing network layer of the pre-trained model, and obtaining a trained child inquiry large model.

[0004] Optionally, the dynamic control is performed by comparing the loss reduction speed of the validation set with a preset threshold, and when the loss reduction speed of the validation set is less than the preset threshold for consecutive N rounds, the bottom layer parameters of the network layer are unfrozen, until the loss is unchanged to obtain the trained child inquiry large model, and N is a natural number greater than 1. Optionally, the unfreezing is layer-by-layer unfreezing, and when the loss reduction speed of the validation set is less than the preset threshold for consecutive N rounds, the bottom layer parameters of the next network layer are unfrozen, and the step is repeated until the bottom layer parameters of all network layers of the pre-trained language model are unfrozen and the loss is unchanged, to obtain the trained child inquiry large model.

[0005] Optionally, the unfreezing controls the parameter unfreezing of each network layer through an unfreezing gate function, and the calculation of the unfreezing gate function is represented as:

[0006] wherein, represents the training step number of the i-th layer layer. the degree of thawing at the current time, Sig represents a Sigmoid function, represents the the training step number at which the layer begins to thaw, represents a smoothing parameter for controlling the thawing speed.

[0007] Optionally, the training further comprises a second stage training, freezing all parameters of the trained pediatric examination large model, adding a low-rank pediatric adaptation module to each layer of the trained pediatric examination large model to obtain a second pediatric examination large model, and inputting the newly added pediatric examination training data into the second pediatric examination large model for training to obtain a final pediatric examination large model; the low-rank pediatric adaptation module dynamically activates by calculating the feature activity of the current layer in the pediatric task; Optionally, the dynamic activation is to determine whether to perform low-rank decomposition calculation through the feature activity; when the feature activity result of the low-rank pediatric adaptation module is greater than a first preset threshold, the input of the feedforward network of the current layer is calculated by low-rank decomposition to obtain a low-rank output, and the low-rank output and the output of the feedforward network are fused to obtain the feedforward network output of the current layer; when the feature activity result of the low-rank pediatric adaptation module is less than or equal to the first preset threshold, the output of the feedforward network is taken as the feedforward network output of the current layer.

[0008] Optionally, the feature activity is obtained by calculating the average value of the position vector change rate of the token recognized as a medical entity in the output of the current layer, and the output of the current layer is the output of the feedforward network. Optionally, the low-rank output is obtained by controlling the contribution intensity of the output features through a scalar gating parameter to obtain a second low-rank output, and the second low-rank output and the output of the feedforward network are fused to obtain the feedforward network output of the current layer.

[0009] The output calculation method of the current layer feedforward network is represented as:

[0010] represents the the final output of the layer feedforward network, represents the output of the original feedforward network, is an indicator function, represents the the pediatric feature activity of the layer, represents the activation threshold of the pediatric adaptation module, represents the scalar gating parameter of the layer, represents the output of the low-rank pediatric adaptation module.

[0011] ​The present application aims to provide a pediatric medical big data processing method based on a distributed mode training architecture, comprising: Obtaining pediatric inquiry data; Inputting the pediatric inquiry data into the distributed pediatric inquiry big model constructed by the distributed pediatric inquiry big model construction method to obtain an inquiry result.

[0012] The present application aims to provide a computer program product comprising a computer program or instructions thereon, which are executed by a processor to implement the distributed pediatric inquiry big model construction method or the pediatric medical big data processing method based on the distributed mode training architecture.

[0013] The present application aims to provide a computer device comprising a memory, a processor and a computer program or instructions stored on the memory, which are executed by the processor to implement the distributed pediatric inquiry big model construction method or the pediatric medical big data processing method based on the distributed mode training architecture.

[0014] The present application aims to provide a computer readable storage medium having a computer program or instructions stored thereon, which are executed by a processor to implement the distributed pediatric inquiry big model construction method or the pediatric medical big data processing method based on the distributed mode training architecture.

[0015] Advantages of the present application: 1. To solve the problem that the current training framework is not suitable for pediatric interactive scenarios and pediatric disease data, the present application proposes a progressive parameter fine-tuning and hierarchical selective reparameterization strategy, which efficiently adapts pediatric knowledge while maintaining general ability through dynamic unfreezing and low-rank adaptation modules under limited data, avoiding catastrophic forgetting.

[0016] 2. The present application proposes a multi-granularity semantic enhancement and dynamic gating fusion mechanism, which accurately converts children's spoken language descriptions into enhanced text inputs rich in structured medical knowledge through fuzzy word alignment, knowledge graph path transformation and context perception fusion.

[0017] 3. The present application constructs a hierarchical perception dual-channel word embedding layer, which fuses general semantic and medical concept hierarchical information, and dynamically adjusts the fusion weight through adaptive gating, so that the model can distinguish and strengthen the representation of key medical entities. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to make the technical solutions in the embodiments of the present application clearer, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained according to these drawings without creative labor for those skilled in the art.

[0019] Figure 1 A distributed pediatric medical consultation large model construction method flowchart is provided for the embodiments of the present application. Figure 2 A distributed pediatric medical consultation large model construction system schematic diagram is provided for the embodiments of the present application. Figure 3 A computer device schematic diagram is provided for the embodiments of the present application. Figure 4 A two-stage progressive parameter fine-tuning strategy schematic diagram is provided for the embodiments of the present application. Figure 5 A diagnostic accuracy rate comparison result of different symptom types under different models is provided for the embodiments of the present application. Figure 6 A medical logic consistency score result under different enhancement methods is provided for the embodiments of the present application. Figure 7 A probability density under different methods is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0020] In order to make the technical solutions in the embodiments of the present application clearer, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained according to these drawings without creative labor for those skilled in the art.

[0021] In some processes described in the specification and claims of the present application and the above-mentioned drawings, a plurality of operations appearing in a specific order are included, but it should be clearly understood that these operations can be executed or in parallel without the order in which they appear in this text. The serial numbers of the operations, such as S101, S102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and these operations can be executed in sequence or in parallel. It should be noted that the "first", "second", etc. in this text are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence. "First" and "second" are different types.

[0022] Figure 1 The distributed pediatric medical consultation large model construction method schematic diagram provided by the embodiments of the present application specifically includes: S1: Obtain a pre-training language model with frozen bottom layer parameters and new pediatric consultation training data; In an embodiment, the pre-trained language model comprises any one or several of the following: Transformer, Mamba-2, VL-Mamba, RWKV-7, Nemotron-H, GPT, T5; Preferably, the pre-trained model is a Transformer.

[0023] In an embodiment, the new pediatric inquiry training data is an enhanced answer text, which is a text sequence of structured semantic information generated by semantic enhancement and knowledge injection of pediatric inquiry data; Optionally, the process of enhancing the answer text is: obtaining a pediatric inquiry data set; The pediatric inquiry data is subjected to named entity recognition and fuzzy expression extraction to obtain medical term entities and fuzzy words; The semantic similarity and knowledge graph correlation of the fuzzy words and the medical term entities are calculated, and each fuzzy word is matched with the medical term entities to obtain a matching result; In the matching result, the medical term entities are searched in the external pediatric knowledge graph to retrieve all related entities and relationships within the jump neighborhood and construct a local subgraph, a connection path is randomly generated in the local subgraph, and the path is converted into a knowledge statement set; The knowledge statement set is fused with the pediatric inquiry data to obtain an enhanced answer text. The jump neighborhood is obtained by different jumps in the neighborhood of the medical term entities.

[0024] Optionally, the fusion of the knowledge statement set and the pediatric inquiry data is performed by a dynamic gating mechanism, which calculates the gating weight of each knowledge statement related to the context of the pediatric inquiry data, only fuses the related knowledge with high weight, and obtains the enhanced answer text.

[0025] In a specific embodiment, the collection of pediatric inquiry text data mainly includes two parts: one is to obtain desensitized real doctor-patient dialogue records from cooperative pediatric hospitals or online health consultation platforms, covering complete question and answer pairs from the description of children's symptoms by parents to the diagnosis and advice given by doctors; the other is to collect related question and answer texts from public medical question and answer communities and pediatric health popular science articles. All these text data need to be strictly subjected to privacy desensitization processing to remove personal information, ensuring the compliance and safety of data use.

[0026] The collected original text constitutes an initial pediatric interview dataset in the form of a series of "question-answer" pairs, which are then finely annotated. The annotation work mainly includes two aspects: first, medical entity annotation of the interview question text is performed by professional personnel with medical background according to pre-defined annotation specifications to identify and annotate pediatric medical entities involved in the text. The annotation categories need to be closely combined with subsequent steps and mainly include: symptom description (such as "cough" and "abdominal pain"), disease name (such as "acute bronchitis" and "allergic rhinitis"), body part (such as "throat" and "abdomen"), examination item (such as "blood routine" and "lung auscultation"), and treatment method (such as "atomization therapy" and "oral rehydration salt").

[0027] Second, the intent and logical relationship of part of the high-quality question and answer pairs are annotated, such as the disease category to which the question belongs and the logical order of the key suggestions in the answer. This part of the annotated data will be used for subsequent loss function construction and model evaluation.

[0028] A pre-trained base model is obtained, a large parameter scale, and a mature large language model pre-trained on general text corpus is selected as the base architecture, such as GPT series model based on Transformer decoder architecture or T5 model based on encoder-decoder architecture. The pre-training weight file of the model is downloaded from the public model repository. This model will serve as the skeleton of the pediatric interview large model we build, and its powerful general language understanding and generation capability is the starting point for domain adaptation.

[0029] In addition, a general pre-trained word vector model such as Word2Vec or GloVe is obtained to support semantic calculation and embedding, which is used to provide general semantic representation of words.

[0030] In a specific embodiment, pediatric interview text data faces three significant challenges: the language used by children to describe symptoms is often imprecise and colloquial, containing a large number of ambiguous words and emotional expressions, which can introduce noise and ambiguity when directly training a model using the original text; the data size of high-quality pediatric medical question and answer pairs is limited, and directly fine-tuning a large-scale pre-trained model can easily overfit; pediatric medical knowledge has strict hierarchical and logical correlation, and conventional text segmentation and embedding methods cannot explicitly model this structured knowledge.

[0031] Conventional methods directly segment and sequence the question and answer pairs, or only perform simple synonym replacement data augmentation, which cannot overcome the three difficulties of imprecise child language, data scarcity, and knowledge structuring.

[0032] The application adopts a data preprocessing method combining an external pediatric medical knowledge graph and context-aware semantic diffusion, generates an enhanced text sequence rich in structured semantic information through semantic enhancement and knowledge injection. 1) Context-aware medical term extraction and fuzzy word alignment; Given a pediatric inquiry data set, first, a lightweight pediatric field named entity recognition model is used to identify standard medical term entities from the question text, and at the same time, a fuzzy word dictionary is used to match the fuzzy expressions in the question, and by calculating the semantic similarity and knowledge graph correlation, each fuzzy word is aligned to the most relevant standard medical term entity, represented as:

[0033] In the formula, represents the standard medical term aligned with the th fuzzy word in the th question, is the standard medical term entity determined after alignment, and the fuzzy word described by the child is most relevant in terms of semantics and medical knowledge, representing the professional and accurate conversion result of the original fuzzy and inaccurate description; represents the sample index, used to identify the th question and answer pair; represents the fuzzy word index; represents the fuzzy word instance identified from the th question text , the fuzzy word instance is specifically determined by a pre-constructed "pediatric fuzzy expression dictionary", which collects non-professional and colloquial words used by children when describing symptoms, such as "uncomfortable", "restless", "stomach gurgling", "a little uncomfortable", and uses them as a pattern library; represents a set of all entities contained in the external pediatric medical knowledge graph ; represents the external pediatric medical knowledge graph; represents the standard medical term entity, which is a candidate element in the set , such as "fever", "acute upper respiratory tract infection", "abdominal pain", etc.; represents the operation of finding the entity in the set that maximizes the objective function value; represents a pre-trained word vector model function, which serves to map the input word or entity into a fixed-dimension dense semantic vector, for example, the pre-trained word vector model can be Word2Vec, GloVe or the word embedding layer of the Transformer model pre-trained on a large general corpus, which is used to provide a general semantic vector representation of the vocabulary; represents a cosine similarity calculation function, which is used to measure the directional similarity of two word vectors in the semantic space, with a value range of ; represents the fuzzy lexical instance and the entity in the knowledge graph The correlation strength in the knowledge graph is measured by the reciprocal of the number of edges passed by the shortest path between the two entities, and the closer the distance, the greater the correlation strength. represents a hyperparameter that balances the semantic similarity and the graph correlation degree weight, which is used to control the contribution proportion of the two to the final decision, with a value range of , and the value example is .

[0034] In specific implementation, a lightweight pediatric named entity recognition model can be obtained by fine-tuning a general medical named entity recognition model (such as a BERT-based model) on public pediatric medical text (such as pediatric textbook abstracts, clinical guidelines, and de-sensitized electronic medical record data), or directly using a model pre-trained on biomedical corpus and open for download (such as BioBERT, ClinicalBERT) for fine-tuning.

[0035] In specific implementation, the external pediatric medical knowledge graph G can be derived from a publicly constructed pediatric disease knowledge graph (such as extracting the pediatric related part from the medical knowledge base UMLS to construct), or self-built by using medical ontology (such as SNOMEDCT, ICD-10) and combining pediatric clinical practice knowledge.

[0036] 2) Subgraph retrieval and semantic diffusion based on knowledge graph; For the entity set corresponding to each question, the All related entities and relationships within the jump neighborhood form a local subgraph, and then a random walk algorithm is used on the local subgraph to generate multiple paths, and these paths are converted into natural language descriptions to obtain a set of knowledge statements related to the original question, denoted as:

[0037] In the formula, The set of knowledge statements for the i-th question contains p natural language descriptions and is a set of natural language sentences. It converts the structured "entity-relationship-entity" relationships in the knowledge graph into a text form that the model can understand, representing the structured and semi-structured background medical knowledge related to the current question. Indicates the first The first question related to the Each knowledge statement text is generated by knowledge graph path transformation and encodes structured semantic relationships extracted from the knowledge graph; The number of knowledge statements is controlled by the random walk parameter, with examples of possible values. ; This represents the number of hops in a knowledge graph retrieval, defining the scope of the neighborhood retrieval. Examples of possible values ​​are provided. .

[0038] In one embodiment, the original problem is assumed to be: "A child has a cough with phlegm and is a little wheezing." Then, a named entity recognition model identifies "cough," "phlegm," and "wheezing," and uses these entities as a starting point to refine the knowledge graph. Search for it in the middle All relevant entities within the jump (such as "bronchitis", "asthma", "lung auscultation") and their relationships (such as "possible accompaniment", "is a symptom", "needs examination") are analyzed, and then a random walk is performed on the subgraph to generate paths such as "cough -> possible accompaniment -> expectoration -> is a symptom -> bronchitis". Finally, the paths are converted into natural language descriptions. For example, "A cough may be accompanied by phlegm, which is one of the symptoms of bronchitis."

[0039] 3) Dynamic gating fusion generates enhanced problem text; To avoid introducing irrelevant information through simple concatenation, a dynamic gating mechanism is employed to fuse the original question with a set of relevant knowledge statements. This mechanism calculates a gating weight for each knowledge statement related to the context of the original question, fusing only the knowledge statements with higher weights to generate the semantically enhanced question text, represented as follows:

[0040] In the formula, Indicates the first The semantically enhanced question text is a question text that has been semantically enhanced. It retains the context and intent of the original pediatric consultation question text while integrating external medical knowledge, making it more precise and richer in information. Indicates the first The original text of a child's medical consultation usually contains a lot of colloquial, vague, emotional, and non-professional expressions; represents a text concatenation operator; represents all knowledge statements indexed from to ; perform a weighted concatenation operation; represents a knowledge statement index, taking a value range from to ; represents the th knowledge statement text related to the th question; represents the th knowledge statement text of the th question , whose value range is , representing the semantic relevance of the knowledge statement to the current question, and the calculation method is represented as ; represents a Sigmoid activation function, which maps real numbers to interval; represents a dynamic gating network parameter vector, which is a trainable parameter, and the dimension is ; represents the transpose of ; represents a dynamic gating network weight matrix, which is a trainable parameter, and the dimension is ; represents a function for obtaining the input text [CLS] token vector representation using a pre-trained language model, which is used to represent the overall semantics of the input text. The pre-trained language model can be a standard pre-trained Transformer encoder model such as BERT, RoBERTa, ALBERT, etc. represents a vector concatenation operation; represents a hyperbolic tangent activation function that maps input values to interval; represents the dimension of the pre-trained language model hidden layer, with a value example of .

[0041] It should be noted that "related knowledge:" is a fixed text prefix used to identify the subsequent content in the concatenated enhanced text as a related knowledge statement introduced from an external knowledge base.

[0042] Further, only the doctor's answer text is subjected to medical terminology standardization processing to obtain an enhanced answer text; In an implementation manner, the medical term standardization processing replaces non-standard expressions, abbreviations or synonyms possibly existing in the doctor's answer text into unified and standard medical terms by querying a medical standard term dictionary, for example, replacing "flowing clear nasal discharge" into "nasal discharge" and replacing "high fever" into "high fever"; Specifically, the definition represents the semantic enhanced answer text of the first , is the enhanced answer text after the medical term standardization processing, and the final output enhanced child consultation data set , wherein is the total number of samples.

[0043] In a specific embodiment, the child description vocabulary coexists with the standard medical term, and the importance of the same medical concept is different in different contexts. The embedding layer of the conventional Transformer model uses a single word embedding matrix to map each word element to a fixed vector, which cannot distinguish word element types or utilize entity hierarchical information in the knowledge graph, and is prone to cause the model to have difficulty in quickly focusing on key medical entities in the early stage of fine-tuning.

[0044] The application adopts a hierarchical perception double-channel embedding layer, introduces a medical concept channel on the basis of standard word embedding, the embedding of the concept channel is modulated by the hierarchical position information of the entity in the knowledge graph, and the information of the two channels is dynamically fused through an adaptive fusion gate, and the specific steps are as follows: 1) standard word embedding channel: word element serialization and standard word embedding acquisition; The semantic enhanced question text or answer text is segmented to obtain a word element sequence, and the definition represents the first sample word element sequence, which is obtained by segmenting the enhanced question text or enhanced answer text of the first sample in the child consultation data set ; represents the word element at the first position in the first sample word element sequence; represents the total length of the sequence.

[0045] Further, by searching a pre-trained word embedding matrix, the standard word embedding vector of each word element in the sequence is obtained, and is represented as:

[0046] In the formula, represents the standard word embedding vector corresponding to the word element , and the dimension is , which encodes the general semantics and context information of the word element. represents a standard word embedding matrix, which is one of the learnable parameter matrices of the large language model to be constructed. In model initialization, the weight of a large-scale pre-trained language model is usually loaded. It is a trainable parameter with a dimension of ; represents the size of the word table, which is determined by the tokenizer and word table of the pre-trained language model selected. For example, the word table size of BERT-base is about 30,000; represents the dimension of the model hidden layer, that is, the dimension of the hidden layer of the Transformer model to be constructed. For example, the dimension of the pre-trained model BERT-base is fixed at 768; represents a one-hot encoding function, which is a function for converting a token into a one-hot vector corresponding to its position in the word table. The function accepts a token and returns a vector with a length of , which is 1 at the position corresponding to the word table index of the accepted token and 0 at the remaining positions. In specific implementation, it is completed by creating a zero tensor and assigning values to the corresponding index.

[0047] 2) Medical concept channel: medical concept embedding generation; For each token in the token sequence, query the pediatric medical knowledge graph. If the token matches an entity in the knowledge graph, obtain the hierarchical encoding vector of the entity. Otherwise, the hierarchical encoding vector is a zero vector. Then, map the hierarchical encoding vector to a medical concept embedding vector using a learnable linear transformation, denoted as:

[0048] In the formula, represents the token corresponding medical concept embedding vector, with a dimension of , which is encoded by the hierarchical information of the entity in the knowledge graph, representing the structured and hierarchical attributes of the token as a medical concept; represents the token corresponding entity in the knowledge graph hierarchical encoding vector, each dimension of which corresponds to a medical classification hierarchy, and the value is the normalized position index of the entity at that level, which is obtained from the knowledge graph . The knowledge graph needs to predefine the position index of each entity in multiple medical classification hierarchies (such as system -> organ -> disease -> symptom) by querying the token The corresponding entity, i.e. the pre-defined hierarchical encoding vector of the entity can be obtained, for example, the normalized hierarchical encoding of the entity "acute bronchitis" can be [respiratory system = 0.2, lower respiratory tract = 0.5, inflammatory disease = 0.7, infectious disease = 0.9]; denotes the weight matrix of the concept embedding layer, is a trainable parameter, and has a dimension of ; denotes the bias vector of the concept embedding layer, is a trainable parameter, and has a dimension of ; denotes the dimension of the hierarchical encoding vector , and the value corresponds to a common multi-level classification system in medical knowledge, for example: [body system, anatomical site, disease category, disease subclass, symptom type, sign, examination, treatment] 8 levels.

[0049] 3) Adaptive fusion gate weight calculation; According to the standard word embedding, medical concept embedding and local context of the current token, the dynamic gate weight is calculated by using the gating network, and it is dynamically decided whether the general semantic or the domain knowledge should be more relied on in the final representation, which is represented as:

[0050] In the formula, denotes the fusion weight of the token , which is used for dynamically fusing the gate weight of the two channel embeddings, and the value range is ; denotes the parameter vector of the adaptive fusion gating network, is a trainable parameter, and has a dimension of ; denotes the transpose of ; denotes the weight matrix of the adaptive fusion gating network, is a trainable parameter, and has a dimension of ; denotes the local context vector of the token , which is used to capture local context information, and has a dimension of , which is obtained by averaging the pooling of the standard word embedding of all tokens in the window, and is represented as ; denotes the single-sided radius size of the pooling window, which is a preset hyperparameter, and defines the range of local context, and the value example is ; This indicates the position index of a word, specifically referring to the position index of the current word in the sequence; Indicates word elements The corresponding standard word embedding vector, Indicates the first The word at position j in a sequence of sample words; This represents the dimension of the intermediate layer of the gated network, with examples of possible values. .

[0051] 4) Dual-channel embedding fusion and positional encoding addition; The standard word embeddings and medical concept embeddings are weighted and summed based on the fusion weights to obtain a fused word embedding vector. Then, the fused word embedding vector is added to the standard positional encoding vector to form the input representation sequence of the first layer of the Transformer model, as follows:

[0052] In the formula, Indicates the first The first sample The fused word embedding vector of each word is the gated fusion embedding vector with dimension 1. The calculation method is expressed as ; Indicates the first The first sample The fused word embedding vector of each word; Indicates the first The representation sequence of *n* samples is the input representation sequence fed into the first layer of the Transformer encoder, with dimensions of *n*. ; Indicates the first The standard positional encoding vector for each position is used to inject the sequence order information into the model and is a trainable parameter; Indicates the first The standard position encoding vector for each position.

[0053] S2: Input the newly added pediatric consultation training data into the pre-trained language model for training, monitor the loss through the validation set in the newly added training data, dynamically control the unfreezing of the low-level parameters of the network layer in the pre-trained model, and obtain the trained pediatric consultation model.

[0054] In one embodiment, the dynamic control is performed by comparing the loss reduction speed of the validation set with a preset threshold, and when the loss reduction speed of the validation set is less than the preset threshold for consecutive N rounds, the bottom layer parameters of the network layer are unfrozen until the loss is unchanged to obtain the trained pediatric inquiry large model, and N is a natural number greater than 1.

[0055] In one embodiment, the unfreezing is layer-by-layer unfreezing, when the loss reduction speed of the validation set is less than the preset threshold for consecutive N rounds, the bottom layer parameters of the next network layer are unfrozen, and the step is repeated until the network layer parameters of the pre-trained language model are all unfrozen and the loss is unchanged, and the trained pediatric inquiry large model is obtained.

[0056] In one embodiment, the unfreezing is performed by an unfreezing gate function to control the parameter unfreezing of each network layer, and the calculation of the unfreezing gate function is represented as:

[0057] wherein, represents the unfreezing degree of the i-th layer at the training step t, Sig represents the Sigmoid function, represents the training step at which the i-th layer parameter plan starts to be unfrozen, represents the smoothing degree parameter for controlling the unfreezing speed.

[0058] In one embodiment, the training further includes a second stage training, freezing all parameters of the trained pediatric inquiry large model, adding a low-rank pediatric adaptation module to each layer of the trained pediatric inquiry large model to obtain a second pediatric inquiry large model, and inputting the new pediatric inquiry training data into the second pediatric inquiry large model for training to obtain a final pediatric inquiry large model; the low-rank pediatric adaptation module dynamically activates by calculating the feature activity of the current layer in the pediatric task. In one embodiment, the dynamic activation is performed by judging whether to perform low-rank decomposition calculation according to the feature activity; when the feature activity result of the low-rank pediatric adaptation module is greater than a first preset threshold, low-rank decomposition calculation is performed on the input of the feedforward network of the current layer to obtain a low-rank output, and the low-rank output and the output of the feedforward network are fused to obtain the output of the feedforward network of the current layer; when the feature activity result of the low-rank pediatric adaptation module is less than or equal to the first preset threshold, the output of the feedforward network is used as the output of the feedforward network of the current layer.

[0059] In one embodiment, the feature activity is obtained by calculating the average value of the position vector change rate of the word units recognized as medical entities in the output of the current layer, and the output of the current layer is the output of the feedforward network. ​​​Optionally, the low-rank output is used to control the contribution strength of the output features through scalar gating parameters to obtain a second low-rank output, and the second low-rank output is fused with the output of the feedforward network to obtain the feedforward network output of the current layer.

[0060] In one embodiment, the output calculation method of the current layer feedforward network is expressed as follows:

[0061] Indicates the first The final output of the layer feedforward network, This represents the output of the original feedforward network. It is an indicator function. Indicates the first The activity level of pediatric characteristics in the layer, This indicates the activation threshold of the pediatric adaptation module. Indicates the first Scalar gating parameters of the layer This represents the output of the low-rank pediatric adapter module.

[0062] In one specific embodiment, directly fine-tuning the model containing a complex domain adaptation structure can easily lead to training instability and overfitting in the pediatric field where data is limited. Conventional fine-tuning strategies cannot adapt to the differences in the degree of dependence of different components in the model on domain knowledge. For example, the medical concept channel needs to learn domain knowledge quickly, while the underlying general language comprehension ability should be relatively stable.

[0063] This invention proposes a two-stage progressive parameter fine-tuning strategy. Based on the domain specificity of model components, parameters are dynamically and progressively unfrozen and optimized in stages. A selective reparameterization mechanism is employed to efficiently expand the model's pediatric knowledge capacity while avoiding catastrophic amnesia, resulting in a fully trained large-scale pediatric consultation model. Figure 4 As shown, the specific steps are as follows: 1) Gradual unfreezing of dynamic loss weight scheduling; In the first stage, most of the low-level parameters of the pre-trained language model are frozen, and only the newly added domain adaptation parameters and the parameters of the top few Transformer layers are trained. As training progresses, the decision on when to unfreeze more layers is dynamically made based on changes in the knowledge-related loss on the validation set, specifically according to the attention consistency loss on the validation set. Or compare learning loss The changes can be used to determine the outcome. For example, a monitoring window and threshold can be set. When the rate of decrease of these losses is lower than the preset threshold in several consecutive rounds, it is considered that the learning of the current unfrozen layer has approached saturation, and the parameters of the next batch of layers are unfrozen. By unfreezing the gating function and the learning rate scaling factor, the unlocking of parameters and the learning rate of each layer are smoothly controlled, achieving a transition from rapid adaptation of domain knowledge to gradual adjustment of general capabilities. The gradient update amount is calculated as follows:

[0064] In the formula, Indicates the first Layer Transformer parameters The gradient update amount determines the actual amount of parameters that are updated in this optimization step. Indicates the first Layer in training steps The degree of thawing at that time, with a value range of Calculated using the Sigmoid function, a value close to 0 indicates that the parameters are essentially frozen, while a value close to 1 indicates that the parameters are completely thawed, represented as... ,when hour, Characterization freeze; when hour, Indicates complete thawing; Indicates the first The training steps at which the layered parameter plan begins to unfreeze; This parameter represents the smoothness of the defrosting process, with an example value of 330. Indicates the current training step count; Indicates the first Layer parameters in training steps The scaling factor of the learning rate used relative to the base learning rate, and the degree of unfreezing. Related, for example A smaller effective learning rate is used when the system is first thawed, and it increases as the degree of thawing increases. This represents the minimum scaling factor, with an example value of 0.01. This represents the maximum scaling factor, with an example value of 1.0. Represents the total loss function For the Layer Layer Transformer parameters The gradient.

[0065] 2) Hierarchical selective reparameterization; Once the model has passed the first stage of learning, the newly added pediatric adaptation parameters have completed initial training. The second stage aims to deepen the model's understanding of complex pediatric logic while preventing excessive alteration of the original model's general representation capabilities. To this end, a low-rank pediatric adaptation module is introduced into each layer of the Transformer's feedforward network. The activation of this module and its low-rank dimension are dynamically determined based on the layer's feature activity in the pediatric task. The adaptation module is only activated for layers with high activity. The final output of the feedforward network is calculated as follows:

[0066] In the formula, Indicates the first The final output of the layer feedforward network, which dynamically integrates the original pre-trained model and the ability to learn new features for the pediatric domain, represents the nonlinear transformation result of the layer on the input features after domain adaptation. This represents the output of the original feedforward network, i.e., the output of the original feedforward network of the pre-trained model; It is an indicator function; its value is 1 when the condition is true, and 0 otherwise. Indicates the first The pediatric feature activity of the layer is specifically obtained by calculating the norm average of the rate of change of the position vectors corresponding to the words that are identified as medical entities in the output of that layer; This indicates the activation threshold of the pediatric adaptation module, which only includes the activity level of pediatric features. The adaptation module will only be enabled for layers that exceed this threshold; Indicates the first The scalar gating parameters of the layer are trainable parameters used to control the contribution strength of the pediatric adaptation module to the final output; This represents the output of the low-rank pediatric adaptation module, which achieves feature transformations specific to the pediatric domain through low-rank decomposition. The calculation method is represented as follows: ; Indicates the first The input to the layer feedforward network is the feature representation after passing through the multi-head attention mechanism of that layer and then through residual connections and layer normalization; This represents the ReLU activation function, which is a non-linear activation function. Indicates the first The first trainable weight matrix of the low-rank adaptation module is the trainable parameters with dimension . ; Indicates the first The second trainable weight matrix of the layer low-rank adaptation module is a trainable parameter, and the dimension is ; represents the pediatric feature activity of the i-th layer The dynamic low-rank dimension of the adaptation module of the i-th layer is calculated dynamically according to the pediatric feature activity of the i-th layer , and is constrained in the range of , and is represented as ; represents the maximum value of the pediatric feature activity in all layers; represents the floor function; represents the minimum low-rank dimension allowed, which is preferably set to 4; represents the maximum low-rank dimension allowed, which is preferably set to 32; represents the scaling coefficient of the pediatric feature activity , and an example of the value is 1.0.

[0067] In a specific implementation, after the basic pediatric knowledge has been learned through the first stage, the goal of the second stage is to further deepen the understanding of the field without destroying the general language ability obtained by the model in the pre-training stage. Therefore, in the second stage fine-tuning process, all original parameters of the pre-trained language model are set to a non-trainable state, i.e., a frozen state, and only the newly introduced low-rank adaptation module parameters and the learnable parameters in the loss function are trained.

[0068] It should be noted that the core idea of the low-rank pediatric adaptation module is that a certain weight matrix in the pre-trained model is not directly updated, but is indirectly adapted to the new task by using the incremental matrix of low-rank decomposition.

[0069] After completing the two-stage training, all parameters (including the pre-trained backbone, the first stage adaptation parameters, and the second stage low-rank adaptation parameters) are integrated, i.e., the final pediatric examination large model is obtained. During inference, the model can understand the imprecise description of children, refer to medical knowledge, and generate professional and logically rigorous answers.

[0070] In one embodiment, focusing on the diagnostic ability of the model for different types of children's symptoms, whether the present technology has stable and superior generalization ability on various pediatric symptoms or only performs well on specific categories is tested. Six common types of children's symptoms are selected for the experiment, including respiratory system, digestive system, nervous system, skin, systemic and behavioral abnormalities. The comparison technology includes the present technology, two different pre-training model fine-tuning methods and a conventional system, the two different pre-training model fine-tuning methods are pre-training transformer fine-tuning based on bidirectional encoder representation fine-tuning and optimized pre-training transformer fine-tuning, and the conventional system is a medical question and answer system based on rules and templates. The experimental results are grouped column charts as shown in Figure 5 The experimental results show that in all six symptom categories, the red column representing the present technology is the highest, indicating that the model has excellent recognition effect on symptoms with typical sign descriptions. The hierarchical perception of dual-channel word embedding and attention mechanism enables the model to not only understand standard terms such as "fever" and "cough", but also accurately grasp the medical connotation behind non-professional descriptions such as "restlessness" and "spirit not good" through the semantic enhancement module, thereby achieving robustness improvement across symptom types.

[0071] In one embodiment, the medical logic consistency of the answers generated by the model is analyzed to ensure that the output conforms to the internal logic of pediatric medical knowledge. The experiment compares direct fine-tuning, data augmentation, knowledge graph enhancement and the present method, as shown in Figure 6 、 Figure 7 The experimental configuration uses medical logic consistency score as a measure, which is calculated by expert evaluation or knowledge graph-based automated inspection. The higher the score, the better the logic consistency. The score distribution of multiple experiments is collected to show the median, quartile and outliers in the box plot, and the probability density distribution of the score is shown in the kernel density estimation plot. In the box plot, the horizontal axis represents different methods, and the vertical axis represents the consistency score. The box of the present method is the highest, and the box is narrow, with few outliers, indicating that the score is concentrated and stable at a high level. The box of other methods is lower, and the box is wider or has more outliers, indicating that the score is scattered and fluctuates greatly. In the kernel density estimation plot, the horizontal axis represents the consistency score, and the vertical axis represents the probability density. The curve of the present method has the highest peak and is located in the high score area, and the curve shape is steep, indicating that the score is highly concentrated. The curve of other methods has a lower peak and is distributed diffusely. The experimental results show that the present method guides the model to learn the logical correlation of pediatric medicine through the attention consistency loss and the contrastive learning loss in the composite supervision loss function, while the traditional method only relies on the cross-entropy loss, which cannot guarantee the logical rigor, so the score is lower and unstable.

[0072] The second aspect of the embodiment of the present application provides a children's medical big data processing method based on a distributed pattern training architecture, comprising: acquiring pediatric interview data; feeding the pediatric interview data into the distributed pediatric interview large model constructed by the method described above to obtain an interview result.

[0073] In one specific embodiment, after the training of the pediatric interview large model is completed through the foregoing steps, it can be deployed and applied to an actual pediatric health consultation scenario; The reasoning application process starts with user input. When a parent or guardian inputs a question description about a child's health through an application, a web page, or a smart device, the system first starts the multi-granularity semantic enhancement and knowledge injection preprocessing process consistent with the training stage. Specifically, the system automatically extracts medical terms and aligns fuzzy words in the original question text input by the user with context awareness, converts the colloquial description into standard medical terms using the pediatric fuzzy expression dictionary and the lightweight named entity recognition model that has been constructed; at the same time, based on the same pediatric medical knowledge graph, the knowledge statements related to the identified entities are retrieved and recognized, and an enhanced question text with enhanced semantics and more accurate information is generated through a dynamic gating fusion mechanism. Then, the enhanced question text is sent to the trained pediatric interview large model. Inside the model, the hierarchical perception dual-channel word embedding layer first works to generate an embedding representation that fuses general semantics and medical concept level information for each word in the input text. Finally, the trained pediatric interview large model outputs the final answer, and the generated answer is presented to the user. The content usually includes medical explanation of symptoms, possible cause analysis, home care suggestions, dangerous signs to be vigilant about, and clear guidance on when to seek medical treatment.

[0074] The entire reasoning application system can be deployed on a cloud or an edge server, providing intelligent interview core capabilities for various pediatric health service platforms through an API interface, realizing 7x24 hour preliminary health consultation, assisting parents in decision-making, and relieving the pressure on medical resources.

[0075] At the same time, the system should be designed to have a feedback mechanism to allow professional doctors to evaluate and correct the answers of the model. These high-quality feedback data can be fed back for continuous incremental training and optimization of the model, forming a closed loop of application and model performance improvement.

[0076] The disclosure also provides a computer program product or system, including a computer program that, when executed by a processor, implements the distributed pediatric medical interview large model construction method described above, or performs the steps of the pediatric medical big data processing method based on the distributed mode training architecture described above.

[0077] Figure 2The distributed child medical inquiry large model construction system provided by the embodiment of the application specifically comprises: The acquisition unit acquires the pre-training language model of the frozen bottom layer parameter and the new child inquiry training data. The training unit inputs the new child inquiry training data into the pre-training language model for training, monitors the loss through the verification set in the new training data, dynamically controls the bottom layer parameter of the unfreezing network layer of the pre-training model, and obtains the trained child inquiry large model.

[0078] Figure 3 The computer device provided by the embodiment of the application specifically comprises: The memory is used for storing program instructions, and the processor is used for calling the program instructions to execute any one of the distributed child medical inquiry large model construction methods or the child medical big data processing methods based on the distributed mode training architecture.

[0079] The embodiment of the application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to execute any one of the distributed child medical inquiry large model construction methods or the child medical big data processing methods based on the distributed mode training architecture.

[0080] The verification result of the verification embodiment shows that assigning inherent weights to the indications can improve the performance of the method compared with the default setting. It can be clearly understood by those skilled in the art that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described here. In the several embodiments provided by the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms. The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme. In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can be a physically independent unit, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of software functional units. Those skilled in the art can understand that all or part of the steps of the various methods in the above embodiments can be completed by programs instructing relevant hardware, and the programs can be stored in a computer readable storage medium, which can include read only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0081] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiment methods can be completed by programs instructing relevant hardware, and the programs can be stored in a computer readable storage medium, and the above-mentioned medium storage can be read only memory, magnetic disk or optical disk, etc.

[0082] The computer device provided by the present application has been described in detail above. For those skilled in the art, according to the idea of the embodiment of the present application, there will be changes in specific implementation and application range. In view of the above, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for constructing a distributed large-scale model for pediatric medical consultation, characterized in that, include: Obtain the pre-trained language model with frozen underlying parameters and add training data for pediatric medical consultations; The newly added pediatric consultation training data is input into the pre-trained language model for training. Loss is monitored using the validation set in the newly added training data, and the underlying parameters of the network layer of the pre-trained model are dynamically controlled to unfreeze, resulting in a well-trained large-scale pediatric consultation model.

2. The method for constructing a distributed large-scale pediatric medical consultation model according to claim 1, characterized in that, The dynamic control compares the rate of loss reduction of the validation set with a preset threshold. When the rate of loss reduction of the validation set is less than the preset threshold for N consecutive rounds, the underlying parameters of the network layer are unfrozen until the loss remains unchanged, thus obtaining a well-trained large model for children's medical consultation. N is a natural number greater than 1. Optionally, the unfreezing is performed layer by layer. When the loss of the validation set decreases at a rate less than a preset threshold for N consecutive rounds, the bottom layer parameters of the next network layer are unfrozen. This step is repeated until all the bottom layer parameters of the network layers of the pre-trained language model are unfrozen and the loss remains unchanged, thus obtaining a well-trained large model for children's medical consultation.

3. The method for constructing a distributed large-scale pediatric medical consultation model according to claim 1, characterized in that, The unfreezing process controls the parameter unfreezing of each network layer through a unfreezing gate function. The calculation of the unfreezing gate function is expressed as follows: in, Indicates the first Layer in training steps The degree of thawing at that time, Sig represents the Sigmoid function. Indicates the first The training steps to begin unfreezing are planned based on the layered parameters. This refers to the smoothness parameter that controls the thawing speed.

4. The method for constructing a distributed large-scale pediatric medical consultation model according to claim 1, characterized in that, The training also includes a second stage of training, in which all parameters of the trained pediatric consultation model are frozen, and a low-rank pediatric adaptation module is added to each layer of the feedforward network in the trained pediatric consultation model to obtain a second pediatric consultation model. The newly added pediatric consultation training data is input into the second pediatric consultation model for training to obtain the final pediatric consultation model. The low-rank pediatric adaptation module dynamically activates the model by calculating the feature activity of the current layer in the pediatric task. Optionally, the dynamic activation determines whether to perform low-rank decomposition calculation based on feature activity. When the feature activity result of the low-rank pediatric adaptation module is greater than a first preset threshold, low-rank decomposition calculation is performed on the input of the feedforward network of the current layer to obtain a low-rank output. The low-rank output is fused with the output of the feedforward network to obtain the output of the feedforward network of the current layer. When the feature activity result of the low-rank pediatric adaptation module is less than or equal to the first preset threshold, the output of the feedforward network is used as the output of the feedforward network of the current layer.

5. The method for constructing a distributed large-scale pediatric medical consultation model according to claim 4, characterized in that, The feature activity is obtained by calculating the average rate of change of the position vectors corresponding to the words identified as medical entities in the current layer output, where the current layer output is the output of the feedforward network. Optionally, the low-rank output is used to control the contribution strength of the output features through scalar gating parameters to obtain a second low-rank output, and the second low-rank output is fused with the output of the feedforward network to obtain the feedforward network output of the current layer.

6. The method for constructing a distributed large-scale pediatric medical consultation model according to claim 4, characterized in that, The output calculation method of the current layer feedforward network is expressed as follows: Indicates the first The final output of the layer feedforward network, This represents the output of the original feedforward network. It is an indicator function. Indicates the first The activity level of pediatric characteristics in the layer, This indicates the activation threshold of the pediatric adaptation module. Indicates the first Scalar gating parameters of the layer This represents the output of the low-rank pediatric adapter module.

7. A method for processing pediatric medical big data based on a distributed pattern training architecture, characterized in that, include: Obtain children's medical consultation data; The pediatric consultation data is input into the distributed pediatric medical consultation model constructed by the distributed pediatric medical consultation model construction method according to any one of claims 1-6 to obtain the consultation results.

8. A computer program product comprising a computer program or instructions, characterized in that, The computer program or instructions are executed by the processor to implement the distributed pediatric medical consultation big data model construction method according to any one of claims 1-6, or to implement the pediatric medical big data processing method based on the distributed pattern training architecture according to claim 7.

9. A computer device comprising a memory, a processor, and a computer program or instructions stored in the memory, characterized in that, The computer program or instructions are executed by the processor to implement the distributed pediatric medical consultation big data model construction method according to any one of claims 1-6, or to implement the pediatric medical big data processing method based on the distributed pattern training architecture according to claim 7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, The computer program or instructions are executed by the processor to implement the distributed pediatric medical consultation big data model construction method according to any one of claims 1-6, or to implement the pediatric medical big data processing method based on the distributed pattern training architecture according to claim 7.

Images