Generative manual pre-check system and method based on artificial intelligence

Abstract

The invention relates to the technical field of intelligent pre-examination, and discloses a generative artificial pre-examination system and method based on artificial intelligence, and the system comprises a multi-modal information collection module which is used for obtaining the text, face image and voice information of a patient; the collaborative reasoning analysis module is used for performing feature extraction and weighted fusion on the text, the face image and the voice information to form multi-modal feature representation, inputting the multi-modal feature representation into a generative large language model for reasoning, and outputting a preliminary triage scheme with scheme confidence; and the decision output module is used for generating inquiry prompt information when the confidence coefficient of the scheme is insufficient, performing reasoning again according to supplementary information fed back by the user, and outputting a final triage scheme. The method corresponds to the system. According to the invention, through multi-modal information fusion and intelligent interactive verification, medical staff are assisted to quickly and accurately complete pre-examination triage, and the efficiency and reliability of triage work can be improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent pre-inspection technology, specifically to a generative artificial pre-inspection system and method based on artificial intelligence. Background Technology

[0002] Currently, triage in hospital emergency departments relies heavily on nurses' manual experience and judgment. Faced with increasing patient volume and complex cases, nurses must make triage decisions quickly based on fragmented information such as patients' complaints, expressions, and physical signs, resulting in immense workload and significantly impacting triage accuracy due to individual experience, communication skills, and fatigue. Automated triage systems designed for adults are difficult to apply directly to children because they often cannot accurately describe their symptoms, and their symptoms (such as specific facial expressions and crying sounds) are unique, lacking effective identification and interpretation methods in existing systems. Furthermore, current technologies primarily focus on processing single information sources (such as textual complaints), failing to effectively integrate and collaboratively analyze multi-dimensional information such as facial expressions and tone of voice, leading to incomplete triage criteria and limited decision-making support.

[0003] Therefore, there is an urgent need for an automated triage solution that can intelligently integrate multimodal information, adapt to the characteristics of pediatrics, and efficiently interact and verify with medical staff, in order to assist nurses in their work and improve the efficiency and accuracy of triage. Summary of the Invention

[0004] The purpose of this invention is to provide a generative artificial pre-inspection system and method based on artificial intelligence to solve the technical problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention discloses the following technical solutions: In a first aspect, the present invention discloses a generative artificial pre-inspection system based on artificial intelligence, the system comprising: The multimodal information acquisition module is configured to acquire the patient's textual complaint information, facial image information, and voice information; The collaborative reasoning analysis module is connected to the multimodal information acquisition module, and the collaborative reasoning analysis module includes: The information feature extraction unit is configured to extract features from the textual complaint information, the facial image information, and the speech information to obtain text feature vectors, facial feature vectors, and speech feature vectors, respectively. The multimodal fusion inference unit is configured to perform weighted fusion of the text feature vector, the facial feature vector, and the speech feature vector to form a unified multimodal feature representation, and input the multimodal feature representation into a pre-trained generative large language model; wherein, the generative large language model performs collaborative inference based on the multimodal feature representation and outputs a preliminary triage plan with triage department, emergency level, and treatment suggestions, as well as the corresponding comprehensive confidence level of the plan; The decision output module, connected to the collaborative reasoning analysis module, is configured to generate and return at least one follow-up prompt message to the user based on the multimodal feature representation when the overall confidence level of the preliminary triage plan is lower than the confidence level threshold; and to receive supplementary information from the user based on the follow-up prompt message, and trigger the collaborative reasoning analysis module to perform further reasoning based on the supplementary information and the multimodal feature representation to generate and output the final triage plan.

[0006] Optionally, the information feature extraction unit includes a text feature extraction subunit, a facial feature extraction subunit, and a speech feature extraction subunit arranged in parallel; wherein: The facial feature extraction subunit employs a few-shot learning neural network based on an attention mechanism and is configured to receive the facial image information, extract the facial action unit encoding sequence, and encode it into the facial feature vector. The text feature extraction subunit adopts a clinical text embedding model and is configured to receive the text chief complaint information, extract semantic features and encode them into the text feature vector. The speech feature extraction subunit employs an audio spectrum feature extraction network and is configured to receive the speech information, extract acoustic features, and encode them into the speech feature vector.

[0007] Optionally, the weighted fusion process includes: The weight allocation model is invoked, which dynamically calculates the weight coefficients corresponding to the text feature vector, facial feature vector and speech feature vector based on the semantic completeness score of the text main complaint information, the key point clarity score of the facial image information and the signal-to-noise ratio score of the speech information. Based on the weighting coefficients, the text feature vector, the facial feature vector, and the speech feature vector are weighted and summed to generate the multimodal feature representation.

[0008] Optionally, the system further includes a retrieval enhancement generation module, which includes: The pediatric medical knowledge base stores structured symptom maps and triage rules for pediatric diseases; The knowledge retrieval module connects the pediatric medical knowledge base and the generative large language model. It is configured to retrieve relevant medical knowledge fragments from the pediatric medical knowledge base before the generative large language model performs collaborative reasoning, using the text feature vector and the facial feature vector as query criteria. The generative large language model is further configured to use the medical knowledge fragments as enhanced context and combine them with the multimodal feature representations for reasoning to generate the preliminary triage plan.

[0009] Optionally, the construction process of the pediatric medical knowledge base includes: From standard pediatric diagnosis and treatment guidelines and historical electronic medical records, symptom entities, disease entities, and logical relationships between entities are extracted to construct a pediatric symptom and disease knowledge graph. The entities, relationships, and associated text descriptions in the pediatric symptom and disease knowledge graph are converted into high-dimensional vectors and stored in a vector database for the retrieval enhancement unit to perform similarity retrieval.

[0010] Optionally, the knowledge retrieval module is further configured as follows: Using the text feature vector and the facial feature vector as the initial query vector, a first round of retrieval is performed from the pediatric medical knowledge base to obtain an initial set of medical knowledge fragments. The initial set of medical knowledge fragments and the multimodal feature representation are input into the generative large language model. The generative large language model evaluates the content relevance of each fragment in the initial set of medical knowledge fragments and generates one or more in-depth query vectors based on fragments whose content relevance is lower than a preset relevance threshold. Using the aforementioned enhanced query vector as a new query basis, a second round of retrieval is conducted from the pediatric medical knowledge base to obtain a supplementary set of medical knowledge fragments; The medical knowledge fragments ultimately used for reasoning by the generative large language model include the initial set of medical knowledge fragments and the supplementary set of medical knowledge fragments.

[0011] Optionally, the process of generating the follow-up question prompt information includes: The multimodal feature representation is subjected to feature dimension reliability analysis to identify feature dimensions with reliability lower than a preset dimension threshold or dimension combinations with logical conflicts. Based on feature dimensions with reliability below a preset dimensional threshold or combinations of dimensions with logical conflicts, question content is constructed to clarify or supplement dimensional information, serving as the follow-up question prompt information.

[0012] Optionally, the system also includes a continuous learning optimization module, configured as follows: The system collects the input data from the multimodal information acquisition module, the final triage plan output by the system, and the final diagnosis and treatment results of the patient recorded in the hospital information system. When the final triage plan is inconsistent with the final diagnosis and treatment result, the corresponding input data and the final diagnosis and treatment result are combined to form a training sample and added to the incremental training set; Using the incremental training set, the generative large language model is periodically fine-tuned using an efficient parameter fine-tuning method.

[0013] Optionally, the multimodal information acquisition module is further configured as follows: When a patient is identified as a child, a graphical interface for selecting the child's physical symptoms is provided to assist the guardian in describing the patient's symptoms. A preset dedicated audio processing channel for children's speech characteristics is also activated to optimize noise reduction and feature extraction for children's crying and slurred speech.

[0014] Secondly, the present invention provides a generative human pre-detection method based on artificial intelligence, the method comprising: Acquisition steps: Collect the patient's textual complaint information, facial image information, and voice information; Collaborative reasoning analysis steps: Features are extracted from the textual complaint information, facial image information, and voice information to obtain text feature vectors, facial feature vectors, and voice feature vectors, respectively; the text feature vectors, facial feature vectors, and voice feature vectors are weighted and fused to form a unified multimodal feature representation; the multimodal feature representation is input into a pre-trained generative large language model, which performs collaborative reasoning based on the multimodal feature representation to output a preliminary triage plan with triage department, emergency level, and treatment suggestions, along with its corresponding comprehensive confidence level. Decision output steps: Determine whether the overall confidence level of the preliminary triage plan is lower than the confidence level threshold; if it is lower, generate and return at least one follow-up prompt message to the user based on the multimodal feature representation; and receive supplementary information from the user based on the follow-up prompt message, perform further reasoning based on the supplementary information and the multimodal feature representation, generate the final triage plan and output it.

[0015] Beneficial Effects: The generative human pre-screening system and method based on artificial intelligence of this invention achieves automation and intelligent upgrading of the pre-screening and triage process by constructing a complete solution integrating multimodal information acquisition, feature fusion, generative collaborative reasoning, and intelligent interactive verification. Specifically, by fusing and analyzing textual complaints, facial images, and voice information, and utilizing a generative large language model for deep collaborative reasoning, it can more comprehensively and accurately understand the patient's (especially children's) condition and generate credible triage suggestions. Furthermore, through confidence assessment and intelligent follow-up questioning mechanisms, it proactively initiates interaction when the model's decision-making confidence is insufficient, guiding the supplementation of key information to optimize the decision, forming a closed loop of human-machine collaboration, thereby ensuring the accuracy of triage even in complex or incomplete situations. This reduces the cognitive burden on nurses, improves the efficiency of triage work, and provides objective and consistent auxiliary decision support, reducing the risk of triage errors caused by human factors. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A structural block diagram of an artificial intelligence-based generative pre-inspection system provided in an embodiment of the present invention; Figure 2 A flowchart illustrating the generative pre-inspection method based on artificial intelligence provided in this embodiment of the invention. Detailed Implementation

[0018] To facilitate understanding of the technical solutions provided in the embodiments of this application, the background technology involved in the embodiments of this application will be described below.

[0019] The current triage process in hospital emergency departments is essentially a rapid classification and decision-making process under conditions of incomplete information and time constraints. This process faces three core contradictions: First, the complexity of the decision-making subjects. Patients, especially infants and young children, often have their complaints relayed by their guardians, leading to information distortion or omissions. Simultaneously, children's diseases develop rapidly and exhibit specific symptoms (such as particular crying patterns and atypical facial expressions), making this non-textual information difficult to quantify and capture by traditional information systems. Second, the limitations of the decision-making body. This decision heavily relies on the clinical experience and instantaneous judgment of triage nurses. During peak hours, nurses must process massive amounts of heterogeneous information (hearing, seeing, asking) in a short period; differences in experience, workload, and communication barriers can all contribute to misjudgments. Third, the limitations of existing technological solutions. Automated triage tools used domestically and internationally are either based solely on structured questionnaires (ignoring facial expressions and voice) or are only single-modal classification models (such as text-only classification). Essentially, they represent a shallow matching of single-dimensional information and cannot simulate the comprehensive multi-source information analysis process of human medical care, such as observation, auscultation, inquiry, and palpation. More importantly, these systems lack the ability to proactively explore and collaboratively verify information when there are contradictions or insufficient evidence, and cannot form an effective human-computer interaction loop with nurses, resulting in limited effectiveness in complex and ambiguous scenarios where assistance is most needed.

[0020] Therefore, there is a significant gap in the existing technological landscape: an automated pre-screening and triage system that, like an experienced nurse, proactively integrates and deeply understands a patient's multi-dimensional presentation (what they say, their facial expressions, their voice), and intelligently guides the supplementation of key information when insufficient information is available, ultimately achieving stable and reliable decision support. This is precisely the problem that the embodiments of this application aim to solve.

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application. Secondly, in this document, the term "comprising" is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements, but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0022] Furthermore, it should be noted beforehand that the collection, transmission, storage, and processing of all patient information (including but not limited to textual complaints, facial images, voice information, and historical medical records) involved in the embodiments of this application are all conducted with the explicit informed consent of the patient or their guardian, and in compliance with relevant laws, regulations, and medical ethics. All data has been anonymized and desensitized before being used for model training or analysis to protect patient privacy.

[0023] In a first aspect, this embodiment provides a generative human pre-inspection system based on artificial intelligence, such as... Figure 1 As shown, the system includes: The multimodal information acquisition module is configured to acquire the patient's textual complaint information, facial image information, and voice information; The collaborative reasoning analysis module is connected to the multimodal information acquisition module. The collaborative reasoning analysis module includes: The information feature extraction unit is configured to extract features from textual narrative information, facial image information, and speech information, respectively, to obtain text feature vectors, facial feature vectors, and speech feature vectors. The multimodal fusion reasoning unit is configured to weight and fuse text feature vectors, facial feature vectors, and speech feature vectors to form a unified multimodal feature representation, and input the multimodal feature representation into a pre-trained generative large language model; wherein, the generative large language model performs collaborative reasoning based on the multimodal feature representation and outputs a preliminary triage plan with triage department, emergency level, and treatment suggestions, as well as the corresponding comprehensive confidence of the plan; The decision output module, connected to the collaborative reasoning analysis module, is configured to generate and return at least one follow-up prompt message to the user based on multimodal feature representation when the overall confidence level of the initial triage plan is lower than the confidence level threshold; and to receive supplementary information from the user based on the follow-up prompt message, and trigger the collaborative reasoning analysis module to perform further reasoning based on the supplementary information and multimodal feature representation, generate the final triage plan, and output it.

[0024] Specifically, the multimodal information acquisition module can be integrated into the terminal equipment of the triage station, including but not limited to: a keyboard or touchscreen for inputting textual complaints, a high-definition camera for capturing facial images of patients (especially children held by accompanying guardians), and a microphone for recording the voice of patients or guardians. The collaborative reasoning analysis module and decision output module can be deployed on the hospital's local server or a compliant cloud computing platform. The generative large language model can be a base model pre-trained on medical text (e.g., a model using the Transformer architecture). The overall confidence level of the proposed solution can be obtained by adding a confidence estimation head after the model's output layer. This estimation head is typically a fully connected layer that outputs a scalar value between 0 and 1, representing the model's overall grasp of the proposed solution. The confidence threshold is a configurable hyperparameter, for example, set to 0.7, and can be adjusted based on clinical validation results in practical applications. The interactive interface of the decision output module displays the generated follow-up prompts (such as "Please describe the specific location of the abdominal pain in more detail") to the nurse, who then asks the patient or guardian for further information and inputs the supplementary information into the system through the same interface.

[0025] Based on the above, this system integrates information from textual, visual, and auditory modalities to construct a comprehensive perceptual input layer that more closely resembles human senses. The information feature extraction unit transforms unstructured raw data into machine-understandable numerical feature vectors. The multimodal fusion reasoning unit synthesizes these features through a weighted approach, enabling the generative large language model to reason based on a richer and more comprehensive patient profile, thereby generating a more accurate preliminary triage plan. The decision output module's confidence-based interactive verification mechanism adds uncertainty perception capabilities to the system. When the system senses uncertainty in its decision-making, it proactively initiates precise follow-up questions to guide the supplementation of key information, optimizing the decision through a human-machine collaborative closed-loop process. This effectively solves the problem of the one-sidedness of decision-making from a single information source and compensates for deficiencies through interaction when information is ambiguous, thus improving the overall comprehensiveness and reliability of triage decisions.

[0026] Different types of information (text, images, audio) have drastically different data characteristics. Using a single or general feature extraction method is insufficient to fully extract the key features from various types of information, especially for children's facial expressions and clinical complaint texts, which require more specialized and refined feature extraction techniques. As an optional implementation of this embodiment, the information feature extraction unit includes parallel text feature extraction subunits, facial feature extraction subunits, and speech feature extraction subunits; wherein: The facial feature extraction subunit employs a few-shot learning neural network based on an attention mechanism, and is configured to receive facial image information, extract facial action unit encoding sequences, and encode them into facial feature vectors. The text feature extraction subunit uses a clinical text embedding model and is configured to receive the chief complaint information in the text, extract semantic features, and encode them into text feature vectors. The speech feature extraction subunit employs an audio spectrum feature extraction network, which is configured to receive speech information, extract acoustic features, and encode them into speech feature vectors.

[0027] In practical implementation, the three sub-units work in parallel to improve efficiency. The core of the facial feature extraction sub-unit is a few-shot learning network based on a meta-learning framework (such as MAML). Its backbone network can employ a lightweight convolutional neural network (CNN), embedding channel attention modules (such as SE modules) and spatial attention modules to enable the network to focus on key facial expression regions such as the eyes and mouth. During training, this network is meta-trained using a small dataset containing various children's pain, crying, and listless expressions, annotated with AU (Facial Action Unit) codes, allowing it to learn how to quickly adapt to and recognize newly emerging small samples of children's facial expressions. Finally, the network outputs a fixed-length feature vector representing a combination of AUs. The text feature extraction sub-unit can employ a BERT (Bidirectional Encoder Representations from Transformers) model further trained on large medical literature and electronic medical record corpora. In practical applications, this model can also be other clinical text embedding models such as BioBERT, ClinicalBERT, etc., and this embodiment does not limit this. The model transforms the input complaint text (e.g., "The child has a fever, cough, and is not feeling well") into a dense vector rich in clinical semantics. The speech feature extraction subunit can use convolutional neural networks or convolutional recurrent neural networks to extract deep patterns from acoustic features such as Mel-spectrograms or Mel-frequency cepstral coefficients (MFCCs) to characterize information such as emotion (e.g., anxiety, distress) and intensity in speech, outputting a speech feature vector.

[0028] Based on the above, by setting up parallel, domain-specific feature extraction subunits, the system can extract the most discriminative features from various types of raw data using the most suitable technical path. Addressing the difficulty in acquiring children's facial expression data, a few-shot learning mechanism enables the model to learn and generalize from limited samples, while an attention mechanism ensures the capture of subtle facial changes. A dedicated clinical text embedding model can better understand medical terminology and colloquial complaints. This parallel, specialized feature extraction architecture provides high-quality, high-information feature input for subsequent fusion and inference, effectively solving the problem of poor performance of general feature extraction methods in professional medical scenarios.

[0029] In multimodal fusion, the quality and reliability of information from different modalities change dynamically in each triage (e.g., the chief complaint is clear, but the child's facial expressions are not fully captured because their back is to the camera). Simple splicing or averaging fusion introduces noise and reduces the utilization rate of effective information. As an optional implementation method in this embodiment, the weighted fusion process includes: The weight allocation model is invoked. Based on the semantic completeness score of the text main complaint information, the key point clarity score of the facial image information, and the signal-to-noise ratio score of the speech information, the weight coefficients corresponding to the text feature vector, facial feature vector, and speech feature vector are dynamically calculated. Based on the weighting coefficients, the text feature vector, facial feature vector, and speech feature vector are weighted and summed to generate a multimodal feature representation.

[0030] Specifically, the weighting model can be a lightweight multilayer perceptron (MLP) that takes three quality scores as input: semantic completeness score and semantic completeness score. The assessment can be performed by calculating the match between the chief complaint text and a pre-defined keyword database for pediatric symptoms, combined with syntactic complexity. Key point clarity scoring. This can be achieved by first locating key facial points (such as eyelids and corners of the mouth) using facial detection algorithms (like RetinaFace), and then calculating the image contrast or gradient magnitude of these key point regions. Speech signal-to-noise ratio (SNR) scoring. It can be estimated by calculating the ratio of the energy of the speech segment to the energy of the silent segment. The weighted allocation model outputs three weight coefficients. , , ,satisfy The final multimodal feature representation We obtain the result through weighted summation: , in, , , These represent text, facial, and speech feature vectors, respectively. In practical deployment, the weight allocation model needs to be trained under supervision using a batch of training data labeled with ideal fusion weights (which can be labeled by experts based on the reliability of different modal information in the case) to learn the mapping relationship from quality score to optimal weight.

[0031] Based on the above, this dynamic weighted fusion mechanism endows the system with information quality perception capabilities. First, the data quality of each modal input is quantitatively evaluated, and then fusion weights are adaptively allocated based on the evaluation results. High-quality, reliable information modalities will occupy a larger proportion in the final integrated features, while low-quality, noisy modalities are suppressed. This intelligent fusion strategy ensures that the system can maximize the use of reliable information and suppress interference from unreliable information under various practical acquisition conditions (such as poor lighting, noisy environments, and brief subject reports), thereby stably producing high-quality multimodal feature representations and laying a solid foundation for subsequent accurate inference.

[0032] While generative large language models possess powerful language understanding and generation capabilities, their internal knowledge may be outdated or contain artifacts. Direct use in clinical scenarios requiring strict adherence to the latest and most authoritative medical knowledge for triage poses safety risks. As an optional implementation of this embodiment, the system further includes a retrieval-enhanced generation module, which includes: The pediatric medical knowledge base stores structured symptom maps and triage rules for pediatric diseases; The knowledge retrieval module connects the pediatric medical knowledge base with the generative large language model. It is configured to retrieve relevant medical knowledge fragments from the pediatric medical knowledge base based on text feature vectors and facial feature vectors before the generative large language model performs collaborative reasoning. Among them, the generative large language model is also configured to use medical knowledge fragments as enhanced context, and combine them with multimodal feature representations for reasoning to generate a preliminary triage plan.

[0033] In its implementation, the pediatric medical knowledge base is a collection of vector indexes stored in a vector database (such as Milvus or Pinecone). The knowledge retrieval module first concatenates or averages the query criteria (text feature vectors and facial feature vectors) to obtain a query vector. Then, by calculating cosine similarity, it retrieves the K (e.g., K=5) medical knowledge fragments most similar to the query vector from the vector database. These knowledge fragments are derived from authoritative treatment guidelines, medical textbooks, etc., and are stored in the form of structured or semi-structured text fragments such as "(symptoms / signs, possible disease, urgency, treatment suggestion)". When generating a response, the generative large language model constructs its input prompt in the following format: "Based on the following patient information: [natural language description converted from multimodal feature representation] and relevant medical knowledge: [K retrieved knowledge fragments], please provide triage suggestions." In this way, the model can use externally retrieved, reliable medical knowledge as a key reference during reasoning.

[0034] Based on the above, the Retrieval Enhanced Generation (RAG) architecture essentially equips the generative large language model with a real-time, accurate, and controllable external memory. During triage reasoning, the system does not simply rely on the model's intrinsic parameter knowledge, but actively retrieves the most relevant medical provisions or case fragments from an authoritative knowledge base and provides these fragments as context to the model. This enhances the accuracy, authority, and timeliness of the model's output, effectively curbing the risk of the model generating random results. It ensures that the generated triage plan strictly adheres to clinical guidelines and standards, solving the core problem of insufficient credibility of generative models in serious medical scenarios.

[0035] The effectiveness of enhanced retrieval depends heavily on the quality of the underlying knowledge base. An unstructured, simple text knowledge base is insufficient to support efficient and accurate semantic retrieval. As a further optional implementation method in this embodiment, the construction process of the pediatric medical knowledge base includes: From standard pediatric diagnosis and treatment guidelines and historical electronic medical records, symptom entities, disease entities, and logical relationships between entities are extracted to construct a pediatric symptom and disease knowledge graph. The entities, relationships, and associated text descriptions in the pediatric symptom and disease knowledge graph are transformed into high-dimensional vectors and stored in a vector database for similarity retrieval by the retrieval enhancement unit.

[0036] Specifically, knowledge graph construction can leverage information extraction techniques. From clinical guidelines (such as PDF documents), natural language processing tools (such as spaCy) combined with medical named entity recognition models are used to extract symptom and disease entities such as "fever," "cough," "pneumonia," and "febrile seizures." Simultaneously, relationship extraction models are used to identify relationships between entities, such as "symptom-indication-disease" ("cough" may indicate "pneumonia") and "disease-belongs to-department" ("pneumonia" belongs to "pediatric respiratory department"). For historical electronic medical records, entities and relationships in fields such as chief complaint, present illness, and diagnosis are extracted after anonymization. All extracted triples (head entity, relation, tail entity) constitute the initial form of the knowledge graph. Subsequently, knowledge graph embedding models (such as TransE) or text embedding models (such as Sentence-BERT) are used to transform each entity, relation, and its associated brief text description (such as "pneumonia: common symptoms include cough, fever, and difficulty breathing") into high-dimensional vectors. These vectors are stored in a vector database and indexed. In actual retrieval, the query vector is compared with these entity / description vectors for similarity, thereby finding the most semantically relevant medical knowledge.

[0037] Based on the above, by constructing a structured knowledge graph and vectorizing it, discrete medical knowledge is transformed into a continuous, computable semantic space. This makes knowledge retrieval no longer a simple keyword matching, but a deep semantic similarity matching. For example, when the system inputs the features "child's shortness of breath, nasal flaring," even if the query text does not directly contain the word "pneumonia," it can still retrieve knowledge fragments related to "difficulty breathing" and "pneumonia" through vector similarity. This semantic-based knowledge retrieval method greatly improves the recall and accuracy of knowledge search, provides a high-quality knowledge source for the retrieval enhancement generation module, and ensures that the contextual information provided to the large language model is highly relevant and reliable.

[0038] Single-round retrieval may fail to retrieve all key information at once due to insufficient precision of the query vector or limitations in knowledge base coverage, resulting in missing context provided to the model. As a further optional implementation in this embodiment, the knowledge retrieval module is also configured as follows: Using text feature vectors and facial feature vectors as initial query vectors, a first-round retrieval is performed from the pediatric medical knowledge base to obtain an initial set of medical knowledge fragments. The initial set of medical knowledge fragments and multimodal feature representations are input into the generative big language model. The generative big language model evaluates the content relevance of each fragment in the initial set of medical knowledge fragments and generates one or more in-depth query vectors based on fragments whose content relevance is lower than a preset relevance threshold. Using the enhanced query vector as a new query basis, a second round of retrieval was conducted from the pediatric medical knowledge base to obtain a supplementary set of medical knowledge fragments. Among them, the generative large language model ultimately uses medical knowledge fragments for reasoning, including an initial set of medical knowledge fragments and a supplementary set of medical knowledge fragments.

[0039] Specifically, the two-round iterative retrieval process described above involves the system concatenating the initially retrieved fragment set (e.g., 5 fragments) with multimodal feature representations after the first round of retrieval. This concatenation is then input into a large language model, and specific instructions are designed, such as: "Please evaluate the relevance of each of the following knowledge fragments to the current patient's condition, and analyze which key information is missing from the fragments with low relevance, thereby generating a new query statement." The model outputs the evaluation results and the new query statement (e.g., "The current information cannot distinguish between common enteritis and appendicitis; more information on 'migratory right lower quadrant abdominal pain' is needed"). Subsequently, the system uses this new query statement to transform it into a deeper query vector through the same text embedding model (e.g., Sentence-BERT) for the second round of retrieval. The preset relevance threshold can be set manually; for example, a score of 1-5 can be used during model evaluation, with fragments scoring below 3 considered insufficiently relevant. The results of the two rounds of retrieval are merged and used together as the final knowledge context. To improve efficiency, iteration termination conditions can be set, such as reaching the maximum number of rounds (e.g., 2 rounds) or the latest round of retrieval failing to obtain any new fragments.

[0040] Based on the above, this iterative retrieval mechanism achieves self-optimization of the retrieval process. The first round of retrieval provides basic context. The generative large language model acts as an information quality assessor and query optimizer, analyzing the deficiencies of the existing context and proactively generating more accurate and targeted in-depth queries. The second round of retrieval then fills in any gaps based on this optimized query. This closed loop of retrieval-evaluation-re-retrieval dynamically and purposefully enriches the knowledge context, significantly improving the completeness and accuracy of the retrieval results. This makes the medical evidence provided to the model for triage reasoning more sufficient and powerful, further reducing the risk of misjudgment due to knowledge gaps.

[0041] When the system's decision confidence is low, effective follow-up prompts need to be generated to guide supplementary information. The key to generating high-quality follow-up questions lies in automatically and accurately identifying which information dimensions(s) are causing the uncertainty. As an optional implementation method in this embodiment, the process of generating follow-up prompt information includes: Perform feature dimension reliability analysis on multimodal feature representations to identify feature dimensions with reliability below a preset dimension threshold or combination of dimensions with logical conflicts; Based on feature dimensions whose reliability is below a preset dimensional threshold or combinations of dimensions that have logical conflicts, question content is constructed to clarify or supplement dimensional information, serving as follow-up prompts.

[0042] In practice, feature dimension reliability analysis can be performed in the information feature extraction unit or the fusion inference unit. One approach is to utilize techniques such as ensemble learning or Monte Carlo Dropout to perform multiple forward propagations on the same input during inference, obtaining multiple estimates for the same feature dimension (e.g., the dimension representing "pain level"), and then calculating the variance of these estimates. As an inverse indicator of reliability, the larger the variance, the lower the reliability. Another approach is to add an auxiliary reliability prediction head to the key feature dimensions during feature extraction network training, directly outputting the reliability score for each dimension. (Between 0 and 1). Preset dimension threshold. The threshold can be set to 0.6. For logical conflicts, some medical rules can be preset, such as "the body temperature feature dimension shows high fever (>39℃), but the activity feature dimension shows alertness." When the values ​​of multiple feature dimensions simultaneously meet the conditions of the conflict rule, a logical conflict is determined to exist. When constructing follow-up questions, the system maintains a mapping table from feature dimensions to natural language questions. For example, if the reliability of the "abdominal pain location" dimension is low, the question "Please describe the specific location of the abdominal pain (e.g., upper abdomen, periumbilical region, right lower abdomen)" is generated; if a conflict is identified between "high fever" and "alertness," the question "Please confirm the child's specific mental state during the fever? Was the child lethargic and sleepy, or was the child still playing?" is generated.

[0043] Based on the above, the reasons behind low-confidence decisions are broken down into fine-grained steps. It's not simply about knowing the "result is unreliable," but rather about further analyzing "where the unreliability lies" or "which pieces of information are contradictory." By performing dimensional reliability analysis and logical consistency checks on the multimodal feature representations, the system can accurately pinpoint the specific sources of information ambiguity or contradiction. The resulting follow-up prompts are no longer vague supplementary information, but highly targeted, guiding questions that directly address the key issues. This significantly improves human-computer interaction efficiency, allowing nurses to quickly obtain the most valuable supplementary information, thereby efficiently assisting the system in optimizing decision-making.

[0044] Medical practices and knowledge are constantly evolving, and the performance of a static model system will decline over time, making it unable to adapt to new disease profiles or treatment models. As an optional implementation of this embodiment, the system also includes a continuous learning and optimization module, configured as follows: Collect input data from the multimodal information acquisition module, the final triage plan output by the system, and the final diagnosis and treatment results of patients recorded in the hospital information system; When the final triage plan is inconsistent with the final diagnosis and treatment result, the corresponding input data and the final diagnosis and treatment result are combined to form a training sample and added to the incremental training set; Using an incremental training set, a parameter-efficient fine-tuning method is employed to periodically fine-tune the parameters of the generative large language model.

[0045] In practice, the continuous learning and optimization module operates offline and asynchronously. Through the hospital's data bus, it anonymously collects closed-loop data of triage cases, adhering to privacy protocols. This data includes inputs (chief complaint text, images, audio), system outputs (final triage plan), and the patient's subsequent final diagnosis and treatment outcome at the hospital (as ground truth). The system automatically compares the triage plan with the final outcome. Cases where departmental classifications are inconsistent or urgency levels deviate beyond a certain range are identified as inconsistent. These cases are converted into instruction samples for model fine-tuning, for example, inputting "patient information: [original multimodal information summary]" and outputting "correct triage plan: [standard plan generated based on the final diagnosis and treatment outcome]". The accumulated incremental training set is used periodically (e.g., weekly or monthly) to fine-tune the large language model using Parametric Efficient Fine-Tuning (PEFT) techniques, such as LoRA. LoRA only trains the low-rank adaptation matrix injected into the model's attention layer without changing the original model parameters, thus achieving efficient and stable incremental learning and effectively avoiding catastrophic forgetting. In practical applications, other PEFT methods or full parameter fine-tuning can also be used, and this application does not limit the specific methods used.

[0046] Based on the above, this continuous learning mechanism endows the system with the ability to self-evolve. Actual clinical diagnosis and treatment results are used as supervisory signals to automatically collect error samples and form incremental training data. Through efficient parameter fine-tuning techniques, the system can continuously learn from new errors, correct its decision-making biases, and absorb new diagnostic and treatment model knowledge without affecting existing performance. This enables the system to keep pace with the development of clinical practice, maintain a high level of auxiliary triage performance over the long term, solve the problem of static models experiencing performance degradation due to knowledge aging, and achieve sustainable service.

[0047] Children, especially infants and toddlers, are unable to actively cooperate with information collection. Their complaints rely on guardians and are often inaccurate. Their vocal information, such as crying, also differs from that of adults, making conventional information collection methods ineffective. As an optional implementation method in this embodiment, the multimodal information collection module is further configured as follows: When a patient is identified as a child, a graphical interface for selecting the child's physical symptoms is provided to assist the guardian in describing the patient's symptoms. A preset dedicated audio processing channel for children's speech characteristics is also activated to optimize noise reduction and feature extraction for children's crying and slurred speech.

[0048] In practical implementation, the system can automatically determine if a patient is a child based on the entered age information or simple image age estimation, or the nurse can manually select the child mode. The graphical interface displays a standard frontal image of a child's body on the terminal touchscreen. Caregivers can quickly locate the area of ​​discomfort by tapping or smearing body parts (such as the head or abdomen) and selecting preset icons for common pediatric symptoms (such as "fever," "cough," "diarrhea," and "fussiness"). The system automatically converts these graphical operations into structured text descriptions, such as "Symptom location: abdomen; Symptom description: diarrhea, crying." A dedicated audio processing channel employs a set of pre-trained audio processing models tailored to the high-frequency, variable-pitch characteristics of children. For example, a deep learning-based noise reduction model, trained on a mixed audio dataset containing a large amount of environmental noise and children's cries, can more effectively separate clean children's speech or crying signals. The feature extraction network also uses a model trained on children's speech datasets to better capture their acoustic features. In practical applications, the interface design and audio processing algorithms can be implemented in various ways, and this application embodiment does not limit these methods.

[0049] Based on the above, customized optimizations were made to the front-end data collection process specifically for children. The graphical selection interface lowers the barrier for guardians to describe symptoms, transforming vague verbal descriptions into precise structured information and improving the accuracy and completeness of the chief complaint. A dedicated audio processing channel enhances the system's accuracy in extracting children's unique vocal signals, ensuring the effective acquisition of speech modal information. These two optimizations, addressing the pain points of difficult and low-quality data collection from pediatric patients at the source, provide a reliable data foundation for subsequent accurate analysis and are crucial prerequisites for the successful application of the entire system in pediatric settings.

[0050] In a second aspect, this embodiment provides a method applied to the artificial intelligence-based generative human pre-screening system described above (i.e., an artificial intelligence-based generative human pre-screening method), such as... Figure 2 As shown, the method includes the following steps in sequence: Acquisition steps: Collect the patient's textual complaint information, facial image information, and voice information; Collaborative reasoning analysis steps: Features are extracted from the textual complaint information, facial image information, and voice information to obtain text feature vectors, facial feature vectors, and voice feature vectors, respectively; these feature vectors are then weighted and fused to form a unified multimodal feature representation; this multimodal feature representation is input into a pre-trained generative large language model, which performs collaborative reasoning based on the multimodal feature representation, outputting a preliminary triage plan with triage department, emergency level, and treatment recommendations, along with its corresponding comprehensive confidence level. Decision output steps: Determine whether the overall confidence level of the preliminary triage plan is lower than the confidence threshold; if it is lower, generate and return at least one follow-up prompt message to the user based on the multimodal feature representation; receive supplementary information from the user based on the follow-up prompt message, and perform further reasoning based on the supplementary information and the multimodal feature representation to generate and output the final triage plan.

[0051] It should be noted that this method corresponds to the aforementioned generative artificial pre-inspection system based on artificial intelligence. Therefore, the parts of this method that are not specifically described (including but not limited to specific technical means and effects) can be referred to the relevant descriptions in the aforementioned generative artificial pre-inspection system based on artificial intelligence, and will not be repeated here.

[0052] In the embodiments provided by this invention, it should be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code, or any suitable combination thereof. For hardware implementation, the processor can be implemented in one or more of the following: application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to implement the functions described herein, or combinations thereof. For software implementation, some or all of the processes of the embodiments can be performed by a computer program instructing the associated hardware. During implementation, the program can be stored in a computer-readable storage medium or transmitted as one or more instructions or code on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media can be any available medium accessible to a computer. Computer-readable storage media can include, but are not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code having the form of instructions or data structures and accessible to a computer.

[0053] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A generative pre-inspection system based on artificial intelligence, characterized in that, The system includes: The multimodal information acquisition module is configured to acquire the patient's textual complaint information, facial image information, and voice information; The collaborative reasoning analysis module is connected to the multimodal information acquisition module, and the collaborative reasoning analysis module includes: The information feature extraction unit is configured to extract features from the textual complaint information, the facial image information, and the speech information to obtain text feature vectors, facial feature vectors, and speech feature vectors, respectively. The multimodal fusion inference unit is configured to perform weighted fusion of the text feature vector, the facial feature vector, and the speech feature vector to form a unified multimodal feature representation, and input the multimodal feature representation into a pre-trained generative large language model; wherein, the generative large language model performs collaborative inference based on the multimodal feature representation and outputs a preliminary triage plan with triage department, emergency level, and treatment suggestions, as well as the corresponding comprehensive confidence level of the plan; The decision output module, connected to the collaborative reasoning analysis module, is configured to generate and return at least one follow-up prompt message to the user based on the multimodal feature representation when the overall confidence level of the preliminary triage plan is lower than the confidence level threshold; and to receive supplementary information from the user based on the follow-up prompt message, and trigger the collaborative reasoning analysis module to perform further reasoning based on the supplementary information and the multimodal feature representation to generate and output the final triage plan.

2. The generative artificial intelligence pre-inspection system based on artificial intelligence according to claim 1, characterized in that, The information feature extraction unit includes parallel text feature extraction subunits, facial feature extraction subunits, and speech feature extraction subunits; wherein: The facial feature extraction subunit employs a few-shot learning neural network based on an attention mechanism and is configured to receive the facial image information, extract the facial action unit encoding sequence, and encode it into the facial feature vector. The text feature extraction subunit adopts a clinical text embedding model and is configured to receive the text chief complaint information, extract semantic features and encode them into the text feature vector. The speech feature extraction subunit employs an audio spectrum feature extraction network and is configured to receive the speech information, extract acoustic features, and encode them into the speech feature vector.

3. The generative artificial intelligence pre-inspection system based on artificial intelligence according to claim 1, characterized in that, The weighted fusion process includes: The weight allocation model is invoked, which dynamically calculates the weight coefficients corresponding to the text feature vector, facial feature vector and speech feature vector based on the semantic completeness score of the text main complaint information, the key point clarity score of the facial image information and the signal-to-noise ratio score of the speech information. Based on the weighting coefficients, the text feature vector, the facial feature vector, and the speech feature vector are weighted and summed to generate the multimodal feature representation.

4. The generative artificial intelligence pre-inspection system based on artificial intelligence according to claim 1, characterized in that, The system also includes a search enhancement generation module, which includes: The pediatric medical knowledge base stores structured symptom maps and triage rules for pediatric diseases; The knowledge retrieval module connects the pediatric medical knowledge base and the generative large language model. It is configured to retrieve relevant medical knowledge fragments from the pediatric medical knowledge base before the generative large language model performs collaborative reasoning, using the text feature vector and the facial feature vector as query criteria. The generative large language model is further configured to use the medical knowledge fragments as enhanced context and combine them with the multimodal feature representations for reasoning to generate the preliminary triage plan.

5. The generative artificial intelligence pre-inspection system based on artificial intelligence according to claim 4, characterized in that, The construction process of the pediatric medical knowledge base includes: From standard pediatric diagnosis and treatment guidelines and historical electronic medical records, symptom entities, disease entities, and logical relationships between entities are extracted to construct a pediatric symptom and disease knowledge graph. The entities, relationships, and associated text descriptions in the pediatric symptom and disease knowledge graph are converted into high-dimensional vectors and stored in a vector database for the retrieval enhancement unit to perform similarity retrieval.

6. The generative artificial intelligence pre-inspection system based on artificial intelligence according to claim 4, characterized in that, The knowledge retrieval module is also configured as follows: Using the text feature vector and the facial feature vector as the initial query vector, a first round of retrieval is performed from the pediatric medical knowledge base to obtain an initial set of medical knowledge fragments. The initial set of medical knowledge fragments and the multimodal feature representation are input into the generative large language model. The generative large language model evaluates the content relevance of each fragment in the initial set of medical knowledge fragments and generates one or more in-depth query vectors based on fragments whose content relevance is lower than a preset relevance threshold. Using the aforementioned enhanced query vector as a new query basis, a second round of retrieval is conducted from the pediatric medical knowledge base to obtain a supplementary set of medical knowledge fragments; The medical knowledge fragments ultimately used for reasoning by the generative large language model include the initial set of medical knowledge fragments and the supplementary set of medical knowledge fragments.

7. The generative artificial intelligence pre-inspection system based on artificial intelligence according to claim 1, characterized in that, The process of generating the follow-up question prompts includes: The multimodal feature representation is subjected to feature dimension reliability analysis to identify feature dimensions with reliability lower than a preset dimension threshold or dimension combinations with logical conflicts. Based on feature dimensions with reliability below a preset dimensional threshold or combinations of dimensions with logical conflicts, question content is constructed to clarify or supplement dimensional information, serving as the follow-up question prompt information.

8. The generative artificial intelligence pre-inspection system based on artificial intelligence according to claim 1, characterized in that, The system also includes a continuous learning optimization module, configured as follows: The system collects the input data from the multimodal information acquisition module, the final triage plan output by the system, and the final diagnosis and treatment results of the patient recorded in the hospital information system. When the final triage plan is inconsistent with the final diagnosis and treatment result, the corresponding input data and the final diagnosis and treatment result are combined to form a training sample and added to the incremental training set; Using the incremental training set, the generative large language model is periodically fine-tuned using an efficient parameter fine-tuning method.

9. The generative artificial pre-inspection system based on artificial intelligence according to claim 1, characterized in that, The multimodal information acquisition module is also configured to: When a patient is identified as a child, a graphical interface for selecting the child's physical symptoms is provided to assist the guardian in describing the patient's symptoms. A preset dedicated audio processing channel for children's speech characteristics is also activated to optimize noise reduction and feature extraction for children's crying and slurred speech.

10. A generative pre-inspection method based on artificial intelligence, characterized in that, The method includes: Acquisition steps: Collect the patient's textual complaint information, facial image information, and voice information; Collaborative reasoning analysis steps: Features are extracted from the textual complaint information, facial image information, and voice information to obtain text feature vectors, facial feature vectors, and voice feature vectors, respectively; the text feature vectors, facial feature vectors, and voice feature vectors are weighted and fused to form a unified multimodal feature representation; the multimodal feature representation is input into a pre-trained generative large language model, which performs collaborative reasoning based on the multimodal feature representation to output a preliminary triage plan with triage department, emergency level, and treatment suggestions, along with its corresponding comprehensive confidence level. Decision output steps: Determine whether the overall confidence level of the preliminary triage plan is lower than the confidence level threshold; if it is lower, generate and return at least one follow-up prompt message to the user based on the multimodal feature representation; and receive supplementary information from the user based on the follow-up prompt message, perform further reasoning based on the supplementary information and the multimodal feature representation, generate the final triage plan and output it.

Images