Pediatric nursing knowledge question-answering method and system based on multi-modal large model enhancement

Abstract

The invention discloses a pediatric nursing knowledge question answering method and system based on multi-modal large model enhancement, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining multi-modal pediatric nursing related data, extracting knowledge entities, obtaining description text data of each knowledge entity, and forming text representation of the entities; establishing triple information according to the knowledge entities to generate a pediatric nursing knowledge basic database, and extracting triads to construct a knowledge graph; extracting multi-order graph data of each entity, and learning to generate knowledge graph representation of the entity; the text representation and the knowledge graph representation are fused, comprehensive knowledge information representation of the entity is obtained, the comprehensive knowledge information representation is used for finely adjusting a preset multi-modal large model, and pediatric nursing knowledge questions and answers are conducted through the finely-adjusted multi-modal large model. According to the method, more accurate, more reliable and more intelligent knowledge questions and answers are realized in a pediatric nursing scene through deep fusion of the structured constraint of the knowledge graph and the generalization ability of the multi-modal large model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and more specifically, to a pediatric nursing knowledge question-answering method and system based on multimodal large model enhancement. Background Art

[0002] Through preventive measures like regular health checkups, growth and development monitoring, and vaccinations, pediatric nursing can promptly identify and intervene in potential health issues that could impact children, such as jaundice, infection, and malnutrition, laying a solid foundation for their healthy growth. Home care complements hospital treatment, not only ensuring a child's speedy recovery but also identifying abnormalities that may be missed by the hospital and preventing them from developing. However, pediatric nursing requires a high degree of professionalism and meticulousness, and parents often experience anxiety and knowledge gaps regarding their children's health during home care. With the advancement of artificial intelligence technology, particularly the knowledge integration and reasoning capabilities of large language models, for example, the application of GPT-4 in the medical field has enabled it to achieve a 91.2% accuracy rate in answering medical questions (Nature Digital Medicine, 2023). In particular, its performance in diagnosing common pediatric diseases has surpassed that of resident physicians. Therefore, a pediatric nursing knowledge question-answering system based on a large language model provides accessible health information, helping parents address common illnesses and nursing issues, alleviating the burden on medical staff and improving the quality of home care. With the help of intelligent systems, parents can obtain basic information outside of medical consultation hours, reducing anxiety and misunderstandings caused by information asymmetry.

[0003] However, traditional large language models mainly focus on understanding the semantics of text. In complex knowledge systems, the structural logic between semantics needs to be enhanced through graph technologies such as knowledge graphs. Therefore, there is an urgent need for a pediatric nursing knowledge question-answering method and system based on a knowledge graph-driven fine-tuned multimodal large model enhancement. Summary of the Invention

[0004] In order to solve the above technical problems, the present invention proposes a pediatric nursing knowledge question and answer method and system based on multimodal large model enhancement, which effectively uses the authoritative knowledge base to control the quality of the massive information knowledge of the large model, thereby improving the content quality of pediatric nursing knowledge questions and answers.

[0005] The first aspect of the present invention provides a pediatric nursing knowledge question-answering method based on multimodal large model enhancement, comprising the following steps:

[0006] Acquire multimodal pediatric nursing-related data, extract knowledge entities, and obtain the descriptive text data of each knowledge entity to form a text representation of the entity;

[0007] Generating a pediatric nursing knowledge base database based on triple information formed from the knowledge entities, and constructing the triples in the pediatric nursing knowledge base database into a knowledge graph;

[0008] Extracting first-order neighbors, second-order neighbors, and third-order neighbors of each entity in the knowledge graph to construct multi-order graph data, and learning to generate knowledge graph representations of entities;

[0009] The text representation and the knowledge graph representation are fused into a vector representation to obtain a comprehensive knowledge information representation of the entity. The comprehensive knowledge information representation is used to fine-tune a preset multimodal large model, and the fine-tuned multimodal large model is used to perform pediatric nursing knowledge questions and answers.

[0010] In this solution, we acquire multimodal pediatric nursing data, extract knowledge entities, and obtain the descriptive text data of each knowledge entity to form a text representation of the entity. Specifically:

[0011] Extracting multimodal pediatric data from text, visual, and speech data sources, preprocessing the multimodal pediatric data in a preset manner, performing PDF parsing on text-based pediatric data, cropping key image regions on visual pediatric data, and converting speech-based pediatric data into text;

[0012] Use the BERT pre-trained model to encode the preprocessed text data, obtain semantic representation information of sentences in the text data, divide the semantic representation information according to random initial span values, and generate a span sequence;

[0013] Extracting feature information from the span sequence through a convolutional neural network, comparing the feature information with predefined entity type feature information, and obtaining the entity type in the span sequence based on the comparison result;

[0014] Integrating the entity type into a corresponding span sequence, filtering the span sequence, fusing the filtered span sequence with the text data, and re-dividing the fused text data using a preset span to obtain an updated span sequence;

[0015] Use the BiLSTM network to learn the updated span sequence, generate the corresponding hidden state through forward propagation and backpropagation, and concatenate the hidden state of the forward propagation with the hidden state of the backpropagation to obtain the hidden state information output by the BiLSTM network;

[0016] The hidden layer state information is fused with the entity type, and the resultant information is input into a linear layer to obtain the knowledge entity category. The descriptive text data of each knowledge entity is obtained as the text encoder input of the pre-trained CLIP model to generate a corresponding text feature vector. The pre-processed visual data is input into the image encoder of the pre-trained CLIP model to generate an image feature vector.

[0017] Calculate the similarity between the text feature vector and the image feature vector, associate the text entity with the visual entity based on the similarity, encode the corresponding entity description text data using the RoBERTa model, use the image features and text representation to splice and align them after dimensionality reduction to generate a supplemented text representation.

[0018] In this solution, a pediatric nursing knowledge base database is generated based on triple information formed according to the knowledge entities, specifically:

[0019] Insert a relation classification layer after the BiLSTM network to obtain text data with knowledge entity category embeddings. Then, fuse the predefined relation types with the text data with knowledge entity category embeddings to generate a relation classification sequence.

[0020] An attention mechanism is introduced to perform attention weighting on the relationship classification sequence, and the weighted relationship classification sequence is input into the Softmax layer for classification to obtain the corresponding relationship category, and a knowledge entity relationship triple is formed according to the knowledge entity category and the relationship category;

[0021] For incomplete sentences, a pre-trained large model is used to complete the relationship. A structured prompt is designed to guide the large model to generate triples. A preset number of candidate completion results are selected, and rule filters are used to retain the candidate completion results that meet the requirements to generate completion triples.

[0022] All triples were used to generate a pediatric nursing knowledge base database.

[0023] In this solution, the triples in the pediatric nursing knowledge base database are constructed into a knowledge graph, specifically:

[0024] Standardize the triples extracted from the pediatric nursing knowledge base database, remove duplicate and contradictory triples, and map non-standard expressions to the medical terminology database for term alignment;

[0025] Create independent nodes based on each knowledge entity, create edge structures based on relationship categories, generate knowledge graphs for storage, perform structured representation of the stored knowledge graphs, and generate initial embedding vectors for entities and relationships in the knowledge graphs;

[0026] Importing the initial embedding vector into the SACN model, encoding it using a weighted graph convolutional layer, performing multi-hop neighbor aggregation on the head entity, outputting an enhanced entity embedding, concatenating the enhanced entity embedding with the relation embedding, and importing it into the convolutional attention layer to extract joint features using one-dimensional convolution;

[0027] The importance scores of the feature dimensions in the joint features are calculated through the attention mechanism to generate the final predicted embedding. According to the final predicted embedding, the candidate relations with scores higher than the threshold are retained, the candidate relations are verified, and the candidate relations that pass the verification are formed into triples and inserted into the knowledge graph.

[0028] In this solution, the first-order neighbors, second-order neighbors, and third-order neighbors of each entity are extracted from the knowledge graph to construct multi-order graph data, and the knowledge graph representation of the entity is learned and generated. Specifically:

[0029] In the knowledge graph, all entities and relationships directly connected to the entity via a single edge are obtained as first-order neighbors, entities connected to the entity via two edges are obtained as second-order neighbors, and entities connected to the entity via three edges are obtained as third-order neighbors;

[0030] Construct multi-order graph data based on the first-order, second-order, and third-order neighbors of an entity, encode the multi-order graph data using a graph attention network, initialize the vector representation of each entity using a pre-trained embedding, update the entity itself using the features of the first-order neighbors, and gradually integrate the features of the second-order and third-order neighbors through iterative aggregation to update the entity's own vector representation;

[0031] The attention mechanism is used to assign attention scores to assign weights to neighbors, and the weighted neighbors are aggregated and updated during message passing to generate entity embeddings that integrate multi-order contexts as the knowledge graph representation of the entity.

[0032] In this solution, the text representation and the knowledge graph representation are fused into a vector representation to obtain a comprehensive knowledge information representation of the entity. The comprehensive knowledge information representation is used to fine-tune the preset multimodal large model, specifically:

[0033] Obtain the text representation and knowledge graph representation of the entity, build a multi-layer perceptron to align and project the text representation and knowledge graph representation, import the projected vector into the gated attention layer for attention weighting, add the weighted vectors, and obtain the comprehensive knowledge information representation of the entity;

[0034] Obtain a large model that supports multimodal input as the base model, build a LoRA adapter, select the cross-attention layer and feedforward network of the base model as the target layer for knowledge injection, add a low-rank decomposition matrix to the selected weight matrix, and use the comprehensive knowledge information representation as an additional input of the LoRA adapter, mapping it to the parameter space of the weight matrix through a linear layer;

[0035] During the fine-tuning stage of the preset multimodal large model, plain text or image questions are input to enable the large model to learn the basic answer pattern of pediatric nursing, perform general answer generation fine-tuning, splice the comprehensive knowledge information representation with the input, and adjust the activation value of the feedforward network through the weight matrix of the LoRA adapter to bias towards knowledge-related input and achieve knowledge enhancement fine-tuning.

[0036] In this solution, a fine-tuned multimodal large model is used to answer questions about pediatric nursing knowledge, specifically:

[0037] When a user asks a question, a fine-tuned pre-trained multimodal large model generates a general answer. The general answer is then represented by text. The keywords in the user's question and the keywords in the general answer are obtained from the knowledge graph. The representations of N keywords are spliced ​​and fused to form a comprehensive knowledge information representation of the question response content.

[0038] The comprehensive knowledge information representation of the question response content is compared with the text representation of the general answer. The quality of the pre-trained multimodal large model's response is judged by a threshold. If the response exceeds the threshold, it is considered correct and provided directly to the user.

[0039] If the threshold is not exceeded, the knowledge text corresponding to the above keywords is added to the user's question and re-input into the pre-trained multimodal large model as a prompt, requiring the large model to reply according to the knowledge text to obtain a new reply.

[0040] The second aspect of the present invention provides a pediatric nursing knowledge question-answering system based on multimodal large model enhancement, comprising: a multimodal data acquisition and preprocessing unit, a knowledge graph construction unit, a multimodal representation fusion unit, a large model fine-tuning unit, and a knowledge enhancement reasoning unit;

[0041] The multimodal data acquisition and preprocessing unit acquires multimodal pediatric nursing knowledge, extracts knowledge entities and obtains descriptive text data of each knowledge entity to form a text representation of the entity;

[0042] The knowledge graph construction unit generates a pediatric nursing knowledge base database based on triple information of knowledge entities, constructs the triples in the pediatric nursing knowledge base database into a knowledge graph, extracts the first-order neighbors, second-order neighbors, and third-order neighbors of each entity in the knowledge graph to construct multi-order graph data, and learns to generate knowledge graph representations of the entities;

[0043] The multimodal representation fusion unit aligns the dimensions of the text representation and the knowledge graph representation through a multi-layer perceptron, adds the elements together to fuse the vector representation, and obtains a comprehensive knowledge information representation of the entity;

[0044] The large model fine-tuning unit selects an adapted base model as a preset multimodal large model, and uses a comprehensive knowledge information representation to perform knowledge injection and fine-tune the preset multimodal large model;

[0045] The knowledge-enhanced reasoning unit uses a fine-tuned multimodal large model to perform pediatric nursing knowledge questions and answers, parses user input to generate general answers, enhances and optimizes answers through knowledge retrieval, and synthesizes and outputs the final answer.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] This invention achieves more accurate, reliable, and intelligent knowledge Q&A in pediatric care scenarios by deeply integrating the structured constraints of knowledge graphs with the generalization capabilities of multimodal large models. The clinical accuracy of knowledge answers is significantly improved compared to pure large models without graph integration, and the reasoning error rate is reduced in key scenarios such as drug contraindications and age-sensitive care recommendations. Furthermore, this invention implements age-sensitive personalized recommendations, improving the accuracy of age-related care recommendations based on explicitly modeling age group nodes in the graph and their relationships with drugs and care. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments or exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or exemplary descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained according to these drawings without paying any creative work.

[0049] Figure 1 A flowchart of a pediatric nursing knowledge question-answering method based on multimodal large model enhancement is shown;

[0050] Figure 2 A flowchart for constructing a knowledge graph in a pediatric nursing knowledge base database is shown;

[0051] Figure 3 A flowchart of using the fine-tuned multimodal macro model for pediatric nursing knowledge question answering is shown;

[0052] Figure 4 The block diagram of the pediatric nursing knowledge question-answering system based on multimodal large model enhancement is shown. DETAILED DESCRIPTION

[0053] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0054] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0055] Figure 1 A flowchart of the pediatric nursing knowledge question-answering method based on multimodal large model enhancement is shown.

[0056] like Figure 1 As shown, this embodiment provides a pediatric nursing knowledge question-answering method based on multimodal large model enhancement, including:

[0057] S102, acquiring multimodal pediatric nursing related data, extracting knowledge entities and obtaining descriptive text data of each knowledge entity to form a text representation of the entity;

[0058] S104, generating a pediatric nursing knowledge base database based on triple information of the knowledge entities, and constructing the triples in the pediatric nursing knowledge base database into a knowledge graph;

[0059] S106, extracting first-order neighbors, second-order neighbors, and third-order neighbors of each entity in the knowledge graph to construct multi-order graph data, and learning to generate knowledge graph representations of the entities;

[0060] S108, the text representation and the knowledge graph representation are fused into a vector representation to obtain a comprehensive knowledge information representation of the entity, the comprehensive knowledge information representation is used to fine-tune a preset multimodal large model, and the fine-tuned multimodal large model is used to perform pediatric nursing knowledge Q&A.

[0061] It should be noted that multimodal pediatric-related data are extracted through text, visual and voice data sources, among which text data sources include Chinese Pediatric Guidelines, nursing operation manuals, electronic medical records (after desensitization), etc., visual data sources include drug instructions illustrations, nursing operation video frames, etc., and voice data sources include patient parent consultation recordings, etc. The multimodal pediatric-related data are preprocessed according to a preset method, and PyPDF2 is used to parse the text-related pediatric data; key image areas of the visual pediatric-related data are cropped, such as the drug dosage table area is detected using OpenCV; the voice-related pediatric data are converted to text, and pediatric terminology correction is performed during the text conversion.

[0062] The preprocessed text data is encoded using the BERT pre-training model and converted into a matrix vector to obtain the semantic representation information of the sentences in the text data. The semantic representation information is divided according to random initial span values ​​to generate a span sequence, and the length of the span sequence does not exceed the set maximum span value. The division of the span sequence can effectively solve the problem of long-distance dependency restrictions between entities. The feature information in the span sequence is extracted through a convolutional neural network, the feature information is compared with the predefined entity type feature information, and the entity type in the span sequence is obtained according to the comparison result; the entity type is integrated into the corresponding span sequence, the span sequence is filtered, the span sequence that does not contain the entity is removed, the filtered span sequence is fused with the text data, and the fused text data is re-divided using the preset span to obtain an updated span sequence; the updated span sequence is learned using a BiLSTM network to obtain entity information, the corresponding hidden layer state is generated through forward propagation and backward propagation, the hidden layer state of the forward propagation and the hidden layer state of the backward propagation are spliced ​​to obtain the hidden layer state information output by the BiLSTM network; the hidden layer state information is fused with the entity type and input into the linear layer to obtain the knowledge entity category.

[0063] The pretrained CLIP model, consisting of an image encoder and a text encoder, has been trained on a large number of image-text pairs and possesses cross-modal matching capabilities. It uses the descriptive text data of each knowledge entity as input to the pretrained CLIP model's text encoder to generate a corresponding text feature vector. The preprocessed visual data is then used as input to the pretrained CLIP model's image encoder to generate an image feature vector. The cosine similarity between the text feature vector and the image feature vector is calculated. A cosine similarity closer to 1 indicates a closer match between the image and text semantics. Based on this similarity, the text entity is associated with the visual entity. The corresponding entity description text data is encoded using the RoBERTa model. The image features and text representations are then concatenated and aligned using dimensionality reduction to generate a supplementary text representation. Low-similarity or conflicting matches (e.g., a rash image matching multiple disease entities) are reviewed by medical experts to avoid misassociations. User feedback on the image-text matching results is also recorded, allowing for dynamic adjustment of thresholds or expansion of the CLIP model's training data.

[0064] Insert a relation classification layer after the BiLSTM network to obtain text data with knowledge entity category embeddings. Fusion the predefined relation types with the text data with knowledge entity category embeddings to generate a relation classification sequence. Introduce an attention mechanism to weight the relation classification sequence. Input the weighted relation classification sequence into the Softmax layer for classification to obtain the corresponding relation category. Build knowledge entity relation triples based on the knowledge entity category and relation category. Use a pre-trained large model for relation completion for incomplete sentences. Select ChatGLM-6B or Qwen-72. For large models with a high proportion of medical corpus, such as B, a structured prompt is designed to guide the large model in generating triples. For example, the input is: {disease} [neonatal jaundice] {relationship} [recommendation] → output: {nursing measure}. BeamSearch generates a preset number of candidate completion results, including {nursing measure: phototherapy}, {nursing measure: increase feeding frequency}, and {nursing measure: stop breastfeeding}. Rule filters are used to retain candidate completion results that meet the requirements to generate completed triples. For example, "stop breastfeeding" is excluded because it is not a first-line recommendation. All triples are then generated into a pediatric nursing knowledge base database. In addition, user corrections to the completion results (such as the expert's annotation "phototherapy requires combined bilirubin level") are recorded for fine-tuning the large model.

[0065] Figure 2 A flowchart for constructing a knowledge graph in a pediatric nursing knowledge base database is shown.

[0066] According to an embodiment of the present invention, the triples in the pediatric nursing knowledge base database are constructed into a knowledge graph, specifically:

[0067] S202, standardizing the triples extracted from the pediatric nursing knowledge base database, removing duplicate and contradictory triples, and mapping non-standard expressions to a medical terminology database for term alignment;

[0068] S204: Create independent nodes based on each knowledge entity, create edge structures based on relationship categories, generate a knowledge graph for storage, perform structured representation on the stored knowledge graph, and generate initial embedding vectors for entities and relationships in the knowledge graph;

[0069] S206, importing the initial embedding vector into the SACN model, encoding it using a weighted graph convolution layer, performing multi-hop neighbor aggregation on the head entity, outputting an enhanced entity embedding, concatenating the enhanced entity embedding with the relationship embedding, importing it into a convolutional attention layer, and extracting joint features using one-dimensional convolution;

[0070] S208, calculate the importance scores of the feature dimensions in the joint features through the attention mechanism, generate a final prediction embedding, retain the candidate relationships with scores higher than the threshold according to the final prediction embedding, verify the candidate relationships, and form the candidate relationships that pass the verification into triples and insert them into the knowledge graph.

[0071] It should be noted that standardization includes unified entity naming and correcting incorrect relationships. Neo4j or GraphDB is selected as the knowledge graph model, supporting attribute graph models and Cypher / SPARQL queries. Each knowledge entity is treated as a separate node with attributes, such as the "ibuprofen" node containing the dosage form: suspension. When creating edges, the relationship type strictly matches the schema, such as "Disease → [Needs Monitoring] → Nursing Indicator." Edge attributes record the source of evidence. The SACN model, combined with an attention mechanism, predicts potential relationships and enhances the knowledge graph. Initial embedding vectors for entities and relationships are generated using pre-trained models like TransE or ComplEx. The SACN model combines a graph convolutional network (GCN) with an attention mechanism. In pediatric care scenarios, the SACN's attention mechanism effectively captures age-sensitive relationships. Its core features include a weighted graph convolutional layer and a convolutional attention layer. The weighted graph convolutional layer aggregates entity neighbor information using a GCN, with weights dynamically adjusted by relationship type. For example, phototherapy and bilirubin monitoring, which are neighbors of neonatal jaundice, have different weights for the preferred treatment and monitoring indicators. The convolutional attention layer uses a 1D convolution kernel to capture local patterns in entity embeddings and uses the attention mechanism to focus on important feature dimensions. For example, the dosage form of a drug has a greater impact on the prediction of contraindication relationships. For any entity pair, all possible relationship types are enumerated and fed into the SACN model to calculate a score. Candidate relationships with scores above a threshold are retained. Supporting evidence is retrieved from clinical guidelines or literature to verify these candidate relationships. Verified triples are inserted into the knowledge graph to automatically complete logically valid associations that are not explicitly mentioned in the guidelines.

[0072] It should be noted that in the knowledge graph, all entities and relationships directly connected to the entity through a single edge are obtained as first-order neighbors. For example, the first-order neighbors of neonatal jaundice include: treatment category: (neonatal jaundice)-[preferred treatment]->(phototherapy), symptom category: (neonatal jaundice)-[accompanying symptoms]->(skin jaundice); entities connected to the entity through two edges are obtained as second-order neighbors. First, the first-order neighbor entity set is obtained, and then the first-order neighbors of each first-order neighbor are recursively queried. The second-order neighbors of neonatal jaundice may include: phototherapy→monitoring indicators→bilirubin level (path: jaundice→phototherapy→bilirubin), skin jaundice→differential diagnosis→breast milk jaundice; entities connected to the entity through three edges are obtained as third-order neighbors, the query of the second-order neighbors is expanded, and further jumps to the third-hop entity. The third-order neighbors of neonatal jaundice may include: bilirubin level→danger threshold→transfusion therapy (path: jaundice→phototherapy→bilirubin→transfusion).

[0073] Multi-order graph data is constructed based on the first-order, second-order, and third-order neighbors of an entity. This multi-order graph data is encoded using a graph attention network. A pre-trained embedding (such as TransE) is used to initialize the vector representation of each entity. The entity is updated by receiving the features of its first-order neighbors. The features of its second-order and third-order neighbors are gradually integrated through iterative aggregation to update the entity's own vector representation. An attention mechanism is used to assign attention scores to the neighbors, such as giving a higher weight to contraindications than related symptoms. The weighted neighbors are aggregated and updated during message passing to generate an entity embedding that incorporates multi-order context as the knowledge graph representation of the entity. By systematically extracting and aggregating multi-order neighbor information, the generated entity representation can capture both local direct associations and model global implicit logic, providing a richer semantic foundation for pediatric nursing knowledge reasoning.

[0074] It is important to note that the textual representation and knowledge graph representation of an entity are obtained, and a multi-layer perceptron is constructed to align and project the textual representation and knowledge graph representation. The projected vector is then fed into a gated attention layer for weighted attention, balancing the contributions of the text and graph. The weighted vectors are then added together, allowing the textual information and knowledge graph information to complement each other on the same dimension. This allows for a comprehensive knowledge representation of the entity, avoiding dimensionality expansion caused by splicing. For example, for the input entity "acetaminophen," a text vector is generated: "For children over 2 years old, dose 10-15 mg / kg..." and a graph vector is generated: "(Contraindications → G6PD deficiency)" and "(Indications → Fever)." An MLP projection is used to convert the text and graph vectors into 512-dimensional vectors, with the text side emphasizing dosage and the graph side emphasizing contraindications. This comprehensive representation includes both dosage details and contraindication warnings, enabling the generation of parent-friendly medication recommendations.

[0075] A pre-configured large model with multimodal input support, such as LLaVA-1.5 or Qwen-VL, is used as the base model. A LoRa adapter is constructed, and the base model's cross-attention layer and feedforward network are selected as the target layers for knowledge injection. These layers are responsible for multimodal feature fusion and knowledge retrieval. A low-rank factorization matrix is ​​added to the selected weight matrix, freezing the original parameters and preserving general capabilities while fine-tuning only pediatric-specific features. The comprehensive knowledge information representation is used as an additional input to the LoRa adapter and mapped to the parameter space of the weight matrix via a linear layer. During the fine-tuning phase of the pre-configured multimodal large model, plain text or image questions are input, allowing the large model to learn basic pediatric nursing response patterns. This allows for fine-tuning of general answer generation, strengthening multimodal alignment, and improving image-text association accuracy. The comprehensive knowledge information representation is concatenated with the input, and the activation values ​​of the feedforward network are adjusted using the LoRA adapter's weight matrix to bias knowledge-relevant input, achieving knowledge-enhanced fine-tuning. Using LoRA's lightweight knowledge injection, the large model achieves precise and controllable output in specialized domains while retaining its general language capabilities. For example, a user enters a question (e.g., "Does neonatal jaundice require phototherapy?"), generates a general answer ("Treatment for jaundice is determined by bilirubin levels"), retrieves a comprehensive representation of the entity "neonatal jaundice" from the knowledge graph, and inputs it into the LoRA adapter. The model then outputs a revised answer ("Phototherapy is required when serum bilirubin is >12 mg / dL, with a lower threshold for premature infants"). If the user uploads a photo of jaundice, the CLIP model extracts features and associates them with the skin yellowing node in the graph to further refine the answer.

[0076] Figure 3 A flowchart for answering pediatric nursing knowledge using the fine-tuned multimodal macro model is shown.

[0077] According to an embodiment of the present invention, a fine-tuned multimodal large model is used to perform pediatric nursing knowledge question answering, specifically:

[0078] S302: When a user asks a question, a general answer is generated by the fine-tuned pre-trained multimodal large model. The general answer is then represented by text. The keywords in the user's question and the keywords in the general answer are obtained from the knowledge graph, and the representations of N keywords are spliced ​​and fused to form a comprehensive knowledge information representation of the question answer content.

[0079] S304: Compare the comprehensive knowledge information representation of the question reply content with the text representation of the general answer for similarity. Use a threshold to determine the quality of the pre-trained multimodal large model's reply. If the reply exceeds the threshold, it is considered correct and provided directly to the user.

[0080] S306: If the threshold is not exceeded, the knowledge text corresponding to the above keywords is added to the user's question and re-input into the pre-trained multimodal large model as a prompt, requiring the large model to reply according to the knowledge text to obtain a new reply.

[0081] It should be noted that the output responses are comprehensively evaluated based on factors such as knowledge accuracy and multimodal alignment, and the large model is optimized based on this comprehensive evaluation. Furthermore, the breadth of entity neighbors retrieved from the knowledge graph (first-order, second-order, third-order, or more hops) is dynamically selected based on the complexity of the user's query. For simple questions, only directly connected first-order neighbors are queried for fast response. For complex questions, the query is expanded to third-order neighbors to cover implicit relationships. By dynamically adjusting the query depth, a balance is achieved between response speed and knowledge coverage.

[0082] Figure 4 The block diagram of the pediatric nursing knowledge question-answering system based on multimodal large model enhancement is shown.

[0083] The second embodiment of the present invention provides a pediatric nursing knowledge question-answering system 4 based on multimodal large model enhancement, comprising: a multimodal data acquisition and preprocessing unit 401, a knowledge graph construction unit 402, a multimodal representation fusion unit 403, a large model fine-tuning unit 404, and a knowledge enhancement reasoning unit 405;

[0084] The multimodal data acquisition and preprocessing unit acquires multimodal pediatric nursing knowledge, extracts knowledge entities and obtains descriptive text data of each knowledge entity to form a text representation of the entity;

[0085] The knowledge graph construction unit generates a pediatric nursing knowledge base database based on triple information of knowledge entities, constructs the triples in the pediatric nursing knowledge base database into a knowledge graph, extracts the first-order neighbors, second-order neighbors, and third-order neighbors of each entity in the knowledge graph to construct multi-order graph data, and learns to generate knowledge graph representations of the entities;

[0086] The multimodal representation fusion unit aligns the dimensions of the text representation and the knowledge graph representation through a multi-layer perceptron, adds the elements together to fuse the vector representation, and obtains a comprehensive knowledge information representation of the entity;

[0087] The large model fine-tuning unit selects an adapted base model as a preset multimodal large model, and uses a comprehensive knowledge information representation to perform knowledge injection and fine-tune the preset multimodal large model;

[0088] The knowledge-enhanced reasoning unit uses a fine-tuned multimodal large model to perform pediatric nursing knowledge questions and answers, parses user input to generate general answers, enhances and optimizes answers through knowledge retrieval, and synthesizes and outputs the final answer.

[0089] The third aspect of the present invention provides a computer-readable storage medium, which includes a pediatric nursing knowledge question and answer method program based on multimodal large model enhancement. When the pediatric nursing knowledge question and answer method program based on multimodal large model enhancement is executed by a processor, it implements the steps of the pediatric nursing knowledge question and answer method based on multimodal large model enhancement.

[0090] In the several embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. In addition, the functional units in the various embodiments of the present invention can all be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0091] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disks or optical disks, and other media that can store program codes.

[0092] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the scope of protection of the present invention.

Claims

1. A pediatric nursing knowledge question-answering method based on multimodal large model enhancement, characterized by: The following steps are involved: Acquire multimodal pediatric nursing-related data, extract knowledge entities, and obtain the descriptive text data of each knowledge entity to form a text representation of the entity; Generating a pediatric nursing knowledge base database based on triple information formed from the knowledge entities, and constructing the triples in the pediatric nursing knowledge base database into a knowledge graph; Extracting first-order neighbors, second-order neighbors, and third-order neighbors of each entity in the knowledge graph to construct multi-order graph data, and learning to generate knowledge graph representations of entities; The text representation and the knowledge graph representation are fused into a vector representation to obtain a comprehensive knowledge information representation of the entity. The comprehensive knowledge information representation is used to fine-tune a preset multimodal large model, and the fine-tuned multimodal large model is used to perform pediatric nursing knowledge questions and answers.

2. The pediatric nursing knowledge question-answering method based on multimodal large model enhancement according to claim 1 is characterized in that: Obtain multimodal pediatric nursing-related data, extract knowledge entities, and obtain the descriptive text data of each knowledge entity to form a text representation of the entity, specifically: Extracting multimodal pediatric data from text, visual, and speech data sources, preprocessing the multimodal pediatric data in a preset manner, performing PDF parsing on text-based pediatric data, cropping key image regions on visual pediatric data, and converting speech-based pediatric data into text; Use the BERT pre-trained model to encode the preprocessed text data, obtain semantic representation information of sentences in the text data, divide the semantic representation information according to random initial span values, and generate a span sequence; Extracting feature information from the span sequence through a convolutional neural network, comparing the feature information with predefined entity type feature information, and obtaining the entity type in the span sequence based on the comparison result; Integrating the entity type into a corresponding span sequence, filtering the span sequence, fusing the filtered span sequence with the text data, and re-dividing the fused text data using a preset span to obtain an updated span sequence; Use the BiLSTM network to learn the updated span sequence, generate the corresponding hidden layer state through forward propagation and back propagation, and concatenate the hidden layer state of the forward propagation and the hidden layer state of the back propagation to obtain the hidden layer state information output by the BiLSTM network; The hidden layer state information is fused with the entity type, and the resultant information is input into a linear layer to obtain the knowledge entity category. The descriptive text data of each knowledge entity is obtained as the text encoder input of the pre-trained CLIP model to generate a corresponding text feature vector. The pre-processed visual data is input into the image encoder of the pre-trained CLIP model to generate an image feature vector. Calculate the similarity between the text feature vector and the image feature vector, associate the text entity with the visual entity based on the similarity, encode the corresponding entity description text data using the RoBERTa model, use the image features and text representation to splice and align them after dimensionality reduction to generate a supplemented text representation.

3. The pediatric nursing knowledge question-answering method based on multimodal large model enhancement according to claim 2 is characterized in that: The pediatric nursing knowledge base database is generated based on the triple information formed by the knowledge entities, specifically: Insert a relation classification layer after the BiLSTM network to obtain text data with knowledge entity category embeddings. Then, fuse the predefined relation types with the text data with knowledge entity category embeddings to generate a relation classification sequence. An attention mechanism is introduced to perform attention weighting on the relationship classification sequence, and the weighted relationship classification sequence is input into the Softmax layer for classification to obtain the corresponding relationship category, and a knowledge entity relationship triple is formed according to the knowledge entity category and the relationship category; For incomplete sentences, a pre-trained large model is used to complete the relationship. A structured prompt is designed to guide the large model to generate triples. A preset number of candidate completion results are selected, and rule filters are used to retain the candidate completion results that meet the requirements to generate completion triples. All triples were used to generate a pediatric nursing knowledge base database.

4. The pediatric nursing knowledge question-answering method based on multimodal large model enhancement according to claim 1 is characterized in that: The triples in the pediatric nursing knowledge base database are constructed into a knowledge graph, specifically: Standardize the triples extracted from the pediatric nursing knowledge base database, remove duplicate and contradictory triples, and map non-standard expressions to the medical terminology database for term alignment; Create independent nodes based on each knowledge entity, create edge structures based on relationship categories, generate knowledge graphs for storage, perform structured representation of the stored knowledge graphs, and generate initial embedding vectors for entities and relationships in the knowledge graphs; Importing the initial embedding vector into the SACN model, encoding it using a weighted graph convolutional layer, performing multi-hop neighbor aggregation on the head entity, outputting an enhanced entity embedding, concatenating the enhanced entity embedding with the relation embedding, and importing it into the convolutional attention layer to extract joint features using one-dimensional convolution; The importance scores of the feature dimensions in the joint features are calculated through the attention mechanism to generate the final predicted embedding. According to the final predicted embedding, the candidate relations with scores higher than the threshold are retained, the candidate relations are verified, and the candidate relations that pass the verification are formed into triples and inserted into the knowledge graph.

5. The pediatric nursing knowledge question-answering method based on multimodal large model enhancement according to claim 1 is characterized in that: In the knowledge graph, the first-order neighbors, second-order neighbors, and third-order neighbors of each entity are extracted to construct multi-order graph data, and the knowledge graph representation of the entity is learned and generated, specifically: In the knowledge graph, all entities and relationships directly connected to the entity via a single edge are obtained as first-order neighbors, entities connected to the entity via two edges are obtained as second-order neighbors, and entities connected to the entity via three edges are obtained as third-order neighbors; Construct multi-order graph data based on the first-order, second-order, and third-order neighbors of an entity, encode the multi-order graph data using a graph attention network, initialize the vector representation of each entity using a pre-trained embedding, update the entity itself using the features of the first-order neighbors, and gradually integrate the features of the second-order and third-order neighbors through iterative aggregation to update the entity's own vector representation; The attention mechanism is used to assign attention scores to assign weights to neighbors, and the weighted neighbors are aggregated and updated during message passing to generate entity embeddings that integrate multi-order contexts as the knowledge graph representation of the entity.

6. The pediatric nursing knowledge question-answering method based on multimodal large model enhancement according to claim 1 is characterized in that: The text representation and the knowledge graph representation are fused into a vector representation to obtain a comprehensive knowledge information representation of the entity. The comprehensive knowledge information representation is used to fine-tune the preset multimodal large model, specifically: Obtain the text representation and knowledge graph representation of the entity, build a multi-layer perceptron to align and project the text representation and knowledge graph representation, import the projected vector into the gated attention layer for attention weighting, add the weighted vectors, and obtain the comprehensive knowledge information representation of the entity; Obtain a large model that supports multimodal input as the base model, build a LoRA adapter, select the cross-attention layer and feedforward network of the base model as the target layer for knowledge injection, add a low-rank decomposition matrix to the selected weight matrix, and use the comprehensive knowledge information representation as an additional input of the LoRA adapter, mapping it to the parameter space of the weight matrix through a linear layer; During the fine-tuning stage of the preset multimodal large model, plain text or image questions are input to enable the large model to learn the basic answer pattern of pediatric nursing, perform general answer generation fine-tuning, splice the comprehensive knowledge information representation with the input, and adjust the activation value of the feedforward network through the weight matrix of the LoRA adapter to bias towards knowledge-related input and achieve knowledge enhancement fine-tuning.

7. The pediatric nursing knowledge question-answering method based on multimodal large model enhancement according to claim 1 is characterized in that: The fine-tuned multimodal large model is used to answer questions about pediatric nursing knowledge, specifically: When a user asks a question, a fine-tuned pre-trained multimodal large model generates a general answer. The general answer is then represented by text. The keywords in the user's question and the keywords in the general answer are obtained from the knowledge graph. The representations of N keywords are spliced ​​and fused to form a comprehensive knowledge information representation of the question response content. The comprehensive knowledge information representation of the question response content is compared with the text representation of the general answer. The quality of the pre-trained multimodal large model's response is judged by a threshold. If the response exceeds the threshold, it is considered correct and provided directly to the user. If the threshold is not exceeded, the knowledge text corresponding to the above keywords is added to the user's question and re-input into the pre-trained multimodal large model as a prompt, requiring the large model to reply according to the knowledge text to obtain a new reply.

8. A pediatric nursing knowledge question-answering system based on multimodal large model enhancement, characterized by: Implementing the pediatric nursing knowledge question-answering method based on multimodal large model enhancement as described in any one of claims 1 to 7, comprising: a multimodal data acquisition and preprocessing unit, a knowledge graph construction unit, a multimodal representation fusion unit, a large model fine-tuning unit, and a knowledge enhancement reasoning unit; The multimodal data acquisition and preprocessing unit acquires multimodal pediatric nursing knowledge, extracts knowledge entities and obtains descriptive text data of each knowledge entity to form a text representation of the entity; The knowledge graph construction unit generates a pediatric nursing knowledge base database based on triple information of knowledge entities, constructs the triples in the pediatric nursing knowledge base database into a knowledge graph, extracts the first-order neighbors, second-order neighbors, and third-order neighbors of each entity in the knowledge graph to construct multi-order graph data, and learns to generate knowledge graph representations of the entities; The multimodal representation fusion unit aligns the dimensions of the text representation and the knowledge graph representation through a multi-layer perceptron, adds the elements together to fuse the vector representation, and obtains a comprehensive knowledge information representation of the entity; The large model fine-tuning unit selects an adapted base model as a preset multimodal large model, and uses a comprehensive knowledge information representation to perform knowledge injection and fine-tune the preset multimodal large model; The knowledge-enhanced reasoning unit uses a fine-tuned multimodal large model to perform pediatric nursing knowledge questions and answers, parses user input to generate general answers, enhances and optimizes answers through knowledge retrieval, and synthesizes and outputs the final answer.

Images