An ai health service system based on multi-modal perception and multi-source knowledge base

Abstract

The application discloses an AI health service system based on multi-modal perception and multi-source knowledge base, relates to the technical field of medical informatization, and comprises the following steps: acquiring a patient identifier and calling historical health data, generating a patient-specific knowledge base and a department public knowledge base after field normalization; processing current text, voice, picture and video inquiry data to generate a current inquiry context; recalling candidate historical health data records based on the current complaint text to generate a patient dynamic portrait and a target inquiry field table; performing consistency verification on the field evidence set to generate a to-be-verified field, and combining missing field calculation to generate a follow-up question priority value to generate a follow-up question order; generating a to-be-confirmed electronic medical record draft with a source mark according to the inquiry result, and writing back the inquiry strategy library after doctor confirmation, modification or return, and synchronizing to the patient end, so that the individuality of health inquiry, the traceability of the medical record draft and the follow-up inquiry optimization capability are improved.

Description

Technical Field

[0001] This invention relates to the field of medical information technology, specifically to an AI health service system based on multimodal perception and a multi-source knowledge base. Background Technology

[0002] With the development of internet hospitals and smart healthcare systems, pre-diagnosis health inquiries, medical history collection, and electronic medical record generation are increasingly being applied in outpatient clinics, follow-up visits, and chronic disease management. Existing health inquiry systems typically generate questions based on patient input of their chief complaint, combined with pre-set questionnaires, disease knowledge bases, or natural language processing models. Electronic medical record content is then generated based on patient feedback to reduce repetitive questioning and record-writing work for doctors.

[0003] Chinese invention patent application CN116028603A, published on April 28, 2023, discloses an intelligent pre-consultation method, device, system, and storage medium based on big data. This method simulates a question-and-answer interaction between a doctor and a user before consultation, extracting chief complaint symptom information from the user's input text. Based on the symptoms, it retrieves symptom-related attribute-based consultation questions from a question knowledge base and a medical knowledge graph. The method collects and summarizes the user's consultation information through a consultation path, ultimately automatically generating a standardized medical record. However, this invention patent application mainly focuses on chief complaint symptom identification, related question generation, and question... The current approach to processing patient information still has room for improvement in the unified organization and collaborative utilization of existing data, such as patients' historical electronic medical records, examination and testing records, medication records, allergy history, and past medical history, as well as in the current inquiry process. For information provided by patients in different formats, such as text, voice, images, or videos, the current solution focuses more on completing the question-and-answer collection process, lacking the identification and prompting of missing, contradictory, or time-inconsistent information from different sources. Furthermore, the correspondence between the generated medical record content and the original inquiry data is not intuitive enough, requiring doctors to review multiple information sources during the review process, and the results of doctor modifications or rejections are difficult to continuously use for improving subsequent inquiry processes.

[0004] Therefore, this invention provides an AI health service system based on multimodal perception and a multi-source knowledge base. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides an AI-powered health service system based on multimodal perception and a multi-source knowledge base. It retrieves and normalizes historical health data through patient identification to generate a patient-specific knowledge base and a departmental public knowledge base. By processing current query data such as text, voice, images, and videos, it generates the current query context. It identifies missing, contradictory, and time-inconsistent information through field evidence sets and generates a follow-up question order based on question priority values. By binding source tags to draft electronic medical records awaiting confirmation and writing doctor confirmation, modification, or rejection results into the query strategy library, it achieves personalized processing of health queries, traceability of medical record field sources, and feedback optimization of subsequent query strategies.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: an AI health service system based on multimodal perception and a multi-source knowledge base, comprising: S1. Obtain patient basic information, historical electronic medical record data, historical examination and test information, historical medication data, allergy history data, and past medical history data to construct historical health data; S2. Generate a patient-specific knowledge base and a departmental public knowledge base based on historical health data; S3. Process the current query data and generate the current query context; S4. Based on the historical health data retrieved from the current chief complaint text, generate a dynamic patient profile and a target query field table; S5. Establish a set of field evidence and perform consistency verification to generate fields to be verified; S6. Sort candidate follow-up question fields based on follow-up question priority values ​​and generate follow-up question order; S7. Generate a draft electronic medical record to be confirmed based on the source tag; S8. The doctor confirms that the draft electronic medical record to be confirmed is written into the electronic medical record database and synchronized with the patient.

[0007] Preferably, the patient's account number, medical card number, electronic medical insurance voucher number, or hospital-side patient unique number is obtained, and the obtained number is converted into a patient identifier; After successful identity verification, historical health data is retrieved based on the patient's identifier. This historical health data includes basic patient information, historical electronic medical record data, historical examination and testing data, historical medication data, allergy history data, and past medical history data. Basic patient information includes name, gender, age, and contact information. Historical electronic medical record data includes historical chief complaints, historical diagnoses, medical records, treatment records, and discharge summaries. Historical examination and testing data includes examination items, test indicators, indicator values, units, reference ranges, and examination / test dates. Historical medication data includes drug names, dosages, frequency of use, and time of use. Allergy history data includes allergy items, descriptions of allergic reactions, and the time of occurrence. Past medical history data includes the name of past illnesses, treatment dates, and the source of the records.

[0008] Preferably, the historical health data obtained by S1 is subjected to field normalization processing; the field normalization processing is performed according to the field normalization rule table, which records the original field name, unified field name, data type, unit format and field meaning description; for historical health data with the same field meaning description but different original field names, they are written into the same unified field. After completing the field normalization process, a patient-specific knowledge base is generated according to data type. For basic patient information, patient identifier, name, gender, age, and contact information are saved as index information. For historical electronic medical record data, historical chief complaints, progress notes, treatment records, and discharge summaries with the same visit date, department name, and historical diagnosis name are grouped together and arranged from earliest to latest according to the medical record creation time to generate historical electronic medical record text segments. Each historical electronic medical record text segment is bound to patient identifier, data source, visit date, department name, historical diagnosis name, and unified field name. For historical examination and test data, historical medication data, allergy history data, and past medical history data, the corresponding data source, source time, and unified field name are recorded respectively. Medical entities are extracted from historical health data containing medical content. These medical entities include symptom entities, disease entities, drug entities, examination entities, test indicator entities, treatment entities, and allergy entities. The extraction of medical entities is achieved through a medical named entity recognition model trained based on the Transformers framework. The input is the text content corresponding to historical electronic medical record text segments, historical examination and test data, historical medication data, allergy history data, or past history data. The output is the medical entity type, entity text, and the position of the entity in the text. Data containing medical entities is organized into historical health description text, which includes unified field names, field content, data source, and source time. The semantic vector of the historical health description text is obtained through the Sentence-Transformers model, with the input being the historical health description text and the output being the semantic vector. The semantic vector is then bound to the patient identifier, data source, source time, and unified field name and written into the patient-specific knowledge base. The patient's basic information is only used for patient identifier association and retrieval scope limitation and is not used as the object of medical entity extraction. At the same time, a departmental public knowledge base is generated according to the department type; the departmental public knowledge base includes target inquiry field tables under different departments, association rules between symptoms and examination items, follow-up visit fields, medication contraindication rules, and health education content.

[0009] Preferably, the patient selects either a text / image inquiry mode or a video inquiry mode during the current health inquiry, and inputs the current inquiry data. The text content in the current chief complaint text and the voice text record is merged to obtain the current chief complaint text. If either the current chief complaint text or the voice text record does not exist, the existing text content is used as the current chief complaint text. The current inquiry data includes the current chief complaint text, voice content, uploaded images, and video data. When a patient selects the text and image inquiry mode, the system receives the patient's current chief complaint text and uploaded image; it uses a ResNet-50 model to classify the uploaded image, with the input being the uploaded image and the output being the classification result of the affected area image or the report image; When the image type is a lesion image, the uploaded image is normalized in size, brightness, and noise is filtered to obtain a preprocessed lesion image. The preprocessed lesion image is then input into the U-Net dual-channel segmentation model to obtain a lesion region mask and a visible body part region mask. A lesion image with the outlines of the lesion region and the visible body part region already labeled is obtained. The U-Net dual-channel segmentation model uses the lesion image with the outlines of the lesion region and the visible body part region labeled as training samples, with the preprocessed lesion image as input and the lesion region mask and the visible body part region mask as output. The ResNet-50 model is used to identify body part categories. The input is a preprocessed image of the affected area, and the output is one of the following body part categories: head, neck, chest, abdomen, back, upper limb, lower limb, hand, or foot. Color features and current area ratio are extracted from the mask of the affected area. The color features are obtained from the hue, saturation, and brightness values ​​of the pixels in the affected area in the HSV color space. The current area ratio is the ratio of the number of pixels in the mask of the affected area to the number of pixels in the mask of the visible body part area. When there are historical images of lesions from the same patient and the same body part category, the historical area ratio corresponding to that historical image is read, and the difference between the current area ratio and the historical area ratio is calculated to obtain a description of the area change; when there are no historical images of lesions from the same patient and the same body part category, only the current area ratio is recorded. Affected area image records are generated based on body part category, color features, current area proportion, and area change description. The affected area image records include structured fields and image description text. The structured fields include body part category, color features, current area proportion, and area change description. The image description text is generated by concatenating the structured fields in a preset field order. When the image type is a report image, PaddleOCR is used to perform text recognition on the report image to obtain the report recognition text. The inspection and testing item name, indicator name, indicator value, unit, and reference range are extracted from the report recognition text. Anomaly labels are generated according to whether the indicator value is below, above, or within the reference range, respectively. The currently uploaded report image record is generated based on the inspection and testing item name, indicator name, indicator value, unit, reference range, and anomaly label. The currently uploaded report image record includes structured fields and report description text, which is generated by concatenating the structured fields according to the preset field order. The current chief complaint text, the image record of the affected area, and the image record of the currently uploaded report are written into the current query context; the BERT sequence labeling medical entity recognition model is used to extract medical entities from the current chief complaint text, image description text, and report description text, respectively, to obtain symptom entities, body part entities, test indicator entities, numerical entities, and time entities; the Sentence-Transformers text vector model is used to convert the current chief complaint text, image description text, and report description text into semantic vectors respectively; When a patient selects the video consultation mode, an audio-visual connection is established; the Whisper speech recognition model is used to perform speech recognition on the audio stream to obtain a speech-text record. The difference value between adjacent frames is calculated for the video stream. The difference value between adjacent frames is the average of the absolute values ​​of the gray level difference between two adjacent grayscale images at the same pixel position. When the difference value of the next adjacent frame is greater than the average of the difference values ​​of the previous ten adjacent frames, the next frame is taken as a key frame. When the difference values ​​of the previous ten adjacent frames are less than ten, the average of all the difference values ​​of the adjacent frames that have been obtained is used as the comparison benchmark. Video representation records are extracted from keyframes; OpenCV is used to identify face regions and the mean and proportion of preset color channels in the face regions are calculated to obtain facial color records; MediaPipe Face Landmarker is used to obtain facial key points and generate expression records; MediaPipe Pose Landmarker is used to obtain human pose key points and generate pose records; MediaPipe Hand Landmarker is used to obtain hand key points and generate hand pointing area records based on the relative position between hand key points and human body parts. Based on the hand pointing area, posture recording, and facial expression recording, the video representation records are mapped to target query fields: when the hand pointing area is located in the abdominal area, or the posture recording includes bending over and is accompanied by frowning features in the facial expression recording, it corresponds to the abdominal discomfort field; when the hand pointing area is located in the foot area, or there is a repetitive rubbing of the feet, it corresponds to the foot discomfort field or the extremity numbness field; when the hand pointing area is located in the head area and the facial expression recording includes frowning features, it corresponds to the head discomfort field. The video representation records are converted into video representation description text, which includes facial color records, expression records, posture records, hand pointing area records, and corresponding target query fields. The Sentence-Transformers text vector model is used to convert the video representation description text into semantic vectors. The speech text records, video representation records, video representation description text, and their corresponding target query fields are written into the current query context.

[0010] Preferably, the current chief complaint text and voice text record in the current query context are read, and candidate historical health data records are retrieved from the patient-specific knowledge base using the current chief complaint text, department type, and patient identifier as search conditions; the current chief complaint text and candidate historical health data records are converted into semantic vectors using the Sentence-Transformers text vector model, the cosine similarity between the two is calculated, and the top K candidate historical health data records are selected from high to low according to the cosine similarity, where K is a positive integer not less than 1; A dynamic patient profile is generated based on the recalled candidate historical health data records; the dynamic patient profile includes basic patient information, historical diagnosis names related to the current chief complaint text, historical medication data, allergy history data, past medical history data, historical examination and test data, and follow-up visit fields; A target query field table is generated based on the current chief complaint text, department type, and department public knowledge base. The target query field table includes field name, field type, field source, and field sorting value. The current query context, patient dynamic profile, and target query field table are compared, and filled fields and missing fields are marked. Fields that already exist in the patient dynamic profile and do not conflict with the current query context are designated as referenceable fields. For referenceable fields that conflict with the current query context, the process is transferred to S5 to generate fields to be verified.

[0011] Preferably, consistency checks are performed on the current chief complaint text, voice text record, affected area image record, currently uploaded report image record, video representation record, and patient dynamic profile in the current inquiry context; a field evidence set is established for the same target inquiry field, and the field evidence set includes field name, source type, source content, source time, symptom entity, numerical entity, time entity, and content polarity; the source type includes current chief complaint text, voice text record, affected area image record, currently uploaded report image record, video representation record, and historical health data record; Content polarity is generated according to the following rules: when the source content contains "have," "appear," "exist," "accompany," or "occurs," and the character distance between this word and the symptom entity in the same source content does not exceed L characters, the content polarity is marked as "existent"; when the source content contains "none," "absent," "deny," "not seen," or "not appeared," and the character distance between this word and the symptom entity in the same source content does not exceed L characters, the content polarity is marked as "non-existent"; all other cases are marked as "uncertain"; L is a positive integer not less than 1. Based on the set of field evidence, fields to be verified are generated. For the same target query field, if the content polarity of different sources is existence and non-existence respectively, a field to be verified with semantic antonyms is generated. If different sources correspond to the same test indicator, and the indicator values ​​after unit normalization are different, a field to be verified with numerical inconsistencies is generated. If different sources correspond to the same symptom entity, but the time entity falls within the time range of D days prior to the current health query initiation time and outside the time range respectively, a field to be verified with time inconsistencies is generated, where D is a positive integer not less than 1. When the source content required by the target query field is not covered by the current query context, patient dynamic profile, or field evidence set, the target query field is marked as the field corresponding to the supplementary evidence.

[0012] Preferably, the missing fields, fields to be verified, and fields corresponding to evidence to be supplemented are collectively used as candidate follow-up questions; for each candidate follow-up question field, the main complaint correlation value, historical correlation value, image correlation value, video correlation value, and corrected correlation value are calculated respectively; The main complaint association value is calculated as follows: The candidate follow-up question fields and the current main complaint text are converted into semantic vectors using the Sentence-Transformers text vector model. The cosine similarity between the two semantic vectors is calculated, and the cosine similarity is mapped from [-1, 1] to [0, 1] to obtain the main complaint association value. The historical association value is calculated as follows: The Sentence-Transformers text vector model is used to convert the candidate follow-up fields and the historical diagnosis names in the patient dynamic profile into semantic vectors, respectively. The cosine similarity between the two semantic vectors is calculated, and the cosine similarity is mapped from [-1, 1] to [0, 1] to obtain the historical association value. The image association value is calculated as follows: The candidate follow-up question field and the image description text or report description text are converted into semantic vectors using the Sentence-Transformers text vector model, the cosine similarity between the two semantic vectors is calculated, and the cosine similarity is mapped from [-1, 1] to [0, 1] to obtain the image association value. The video association value is calculated as follows: The candidate follow-up question field and the video representation description text are converted into semantic vectors using the Sentence-Transformers text vector model. The cosine similarity between the two semantic vectors is calculated and the cosine similarity is mapped from [-1, 1] to [0, 1] to obtain the video association value. The corrected correlation value is the field correction rate of the candidate follow-up question field in similar health inquiries; similar health inquiries refer to historical health inquiries and current health inquiries being the same in at least two aspects: department type, disease name, and inquiry scenario; the disease name comes from the historical diagnosis name in the patient dynamic profile or the candidate disease name recalled by the department's public knowledge base based on the current chief complaint text; the inquiry scenario is one of the following: initial visit, follow-up visit, chronic disease follow-up, or report interpretation; the field correction rate is the ratio of the number of times the field has been modified by the doctor in the most recent N similar health inquiries to the total number of the most recent N similar health inquiries, where N is a positive integer not less than 1; The priority value for follow-up questions is the sum of the main complaint correlation value, historical correlation value, image correlation value, video correlation value, corrected correlation value, and verification flag value; the verification flag value of the field to be verified is 1, and the verification flag value of the field not to be verified is 0; when a source of data does not exist, the correlation value corresponding to that source is recorded as 0; The order of follow-up questions is generated from high to low priority value. If two candidate follow-up question fields have the same priority value, the order of follow-up questions is determined in the following order: fields to be verified first, fields whose source is a disease inquiry rule first, and fields with smaller sort values ​​first. After the patient answers follow-up questions, the answer is written into the current query context, and the missing field judgment, consistency check, and follow-up question priority value calculation are re-executed. For fields that the patient clearly cannot provide, where the answer to two consecutive follow-up questions does not contain the symptom entity, numerical entity, or time entity corresponding to the candidate follow-up question field, where the candidate follow-up question field does not match the patient's basic information, patient dynamic profile, or current query context, or where the patient requests manual intervention, the field is written into the "to be supplemented by doctor" column or transferred to doctor intervention. The above follow-up questioning process is repeated until all candidate follow-up question fields are processed, or the current health query is transferred to doctor intervention.

[0013] Preferably, after completing the follow-up questioning process in S6, a draft electronic medical record to be confirmed is generated based on the current inquiry context, patient dynamic profile, image record of the affected area, image record of the currently uploaded report, video characterization record, and historical examination and test data; the draft electronic medical record to be confirmed includes subjective data, objective data, assessment opinions, and treatment suggestions; Bind a source tag to each field in the draft electronic medical record to be confirmed; the source tag includes source type, source content, source time and source record number; the source type includes patient text input, voice text recording, affected area image recording, currently uploaded report image recording, video representation recording, historical health data recording and doctor correction recording; Content without a source marker is not written into the main text of the draft electronic medical record to be confirmed, but is written into the column to be supplemented by the doctor; for medical record fields that have been generated for verification in S5 and have been confirmed by follow-up in S6, the original source of difference and the patient's confirmation content are recorded in the source marker; when the draft electronic medical record to be confirmed is generated for the first time, the source marker does not include the doctor's correction record; historical doctor correction records are only used as source markers when the content of the medical record field actually references the historical correction results; after the doctor modifies the draft electronic medical record to be confirmed, the doctor's correction record is added to the modified field.

[0014] Preferably, the draft electronic medical record to be confirmed is pushed to the doctor's workstation and set to a status pending doctor's confirmation; the doctor confirms, modifies, or returns the draft electronic medical record to be confirmed. When a doctor modifies a draft electronic medical record to be confirmed, the original content, the modified content, the modified field name, the doctor's identifier, and the modification time are saved to generate a doctor's correction record for this time. This doctor's correction record is then written into the inquiry strategy library. Subsequent similar health inquiries read the historical doctor's correction records in the inquiry strategy library and participate in the calculation of follow-up inquiry priority value according to the field correction rate calculation method in S6. When a doctor returns a draft electronic medical record pending confirmation, the reason for the return is recorded. If the reason for the return corresponds to a target query field, the reason for the return is recorded as a doctor's correction to that target query field and written into the query strategy library. If the reason for the return does not correspond to a target query field, a record for a field to be expanded is generated. The record for a field to be expanded includes the reason for the return, the name of the field to be added by the doctor, the department type, the disease name, the query scenario, and the source time. After the record for a field to be expanded is manually reviewed and approved, it is written into the department's public knowledge base as a new target query field or a field alias. Records for a field to be expanded that do not pass manual review are only retained as records and do not participate in the subsequent generation of target query fields. When a doctor confirms a draft electronic medical record pending confirmation, the draft status is changed to "confirmed and entered into the database," and the draft is written into the electronic medical record database. A medical record update notification is sent to the patient, allowing the patient to view the electronic medical record content, medical orders, and health education content confirmed by the doctor. At the same time, a closed-loop record is maintained, which includes the patient's identifier, inquiry start time, inquiry end time, inquiry mode, patient-specific knowledge base version, departmental public knowledge base version, draft number, list of fields modified by the doctor, doctor's confirmation time, and electronic medical record entry number.

[0015] (III) Beneficial Effects This invention provides an AI-powered health service system based on multimodal perception and a multi-source knowledge base, which has the following beneficial effects: 1. This invention normalizes fields in historical health data and generates a patient-specific knowledge base and a departmental public knowledge base, enabling the health inquiry process to combine the patient's historical electronic medical record data, historical examination and test data, historical medication data, allergy history data, and past medical history data, thereby improving the matching degree between the inquiry content and the patient's personal health status.

[0016] 2. This invention processes current query data such as text, voice, images, and videos, and generates the current query context, enabling query data from different sources to participate in subsequent field judgments in a unified manner, thereby reducing information omissions caused by single text queries.

[0017] 3. This invention establishes a set of field evidence and generates fields to be verified. Then, it calculates the priority value of follow-up questions by combining the missing fields and the corresponding fields of the evidence to be supplemented. This allows the order of follow-up questions to be dynamically determined based on the current chief complaint text, historical health data, image records, video representation records, and doctor's correction results, thereby reducing the probability of repeated follow-up questions and omission of key fields.

[0018] 4. This invention binds a source tag to each field in the draft electronic medical record to be confirmed, and writes the doctor's confirmation, modification or rejection results into the inquiry strategy library, so that the source of the medical record field can be traced, and the doctor's correction results can participate in the subsequent health inquiry process, thereby improving the continuity of medical record draft review and inquiry strategy optimization. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the process of an AI health service system based on multimodal perception and multi-source knowledge base according to the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 This invention provides an AI-powered health service system based on multimodal perception and a multi-source knowledge base, comprising the following steps: S1. Obtain patient basic information, historical electronic medical record data, historical examination and test information, historical medication data, allergy history data, and past medical history data to construct historical health data.

[0022] Obtain the patient's account number, medical card number, electronic medical insurance voucher number, or hospital-side unique patient number, and convert the obtained number into a patient identifier. The patient identifier is used to limit the scope of patients for subsequent data retrieval, knowledge base search, medical record draft generation, and status transition. After successful identity verification, historical health data is retrieved based on the patient's identifier. This historical health data includes basic patient information, historical electronic medical record data, historical examination and testing data, historical medication data, allergy history data, and past medical history data. Basic patient information includes name, gender, age, and contact information. Historical electronic medical record data includes historical chief complaints, historical diagnoses, medical records, treatment records, and discharge summaries. Historical examination and testing data includes examination items, test indicators, indicator values, units, reference ranges, and examination / test dates. Historical medication data includes drug names, dosages, frequency of use, and time of use. Allergy history data includes allergy items, descriptions of allergic reactions, and the time of occurrence. Past medical history data includes the name of past illnesses, treatment dates, and the source of the records.

[0023] S2. Generate a patient-specific knowledge base and a departmental public knowledge base based on historical health data.

[0024] The historical health data obtained by S1 is processed for field normalization. Field normalization is performed according to the field normalization rule table, which records the original field name, unified field name, data type, unit format, and field meaning description. Historical health data with the same field meaning description but different original field names are written into the same unified field. After completing the field normalization process, a patient-specific knowledge base is generated according to data type. For basic patient information, patient identifier, name, gender, age, and contact information are saved as index information. For historical electronic medical record data, historical chief complaints, progress notes, treatment records, and discharge summaries with the same visit date, department name, and historical diagnosis name are grouped together and arranged from earliest to latest according to the medical record creation time to generate historical electronic medical record text segments. Each historical electronic medical record text segment is bound to patient identifier, data source, visit date, department name, historical diagnosis name, and unified field name. For historical examination and test data, historical medication data, allergy history data, and past medical history data, the corresponding data source, source time, and unified field name are recorded respectively. Medical entities are extracted from historical health data containing medical content. These medical entities include symptom entities, disease entities, drug entities, examination entities, test indicator entities, treatment entities, and allergy entities. The extraction of medical entities is achieved through a medical named entity recognition model trained based on the Transformers framework. The input is the text content corresponding to historical electronic medical record text segments, historical examination and test data, historical medication data, allergy history data, or past history data. The output is the medical entity type, entity text, and the position of the entity in the text. Data containing medical entities is organized into historical health description text, which includes unified field names, field content, data source, and source time. The semantic vector of the historical health description text is obtained through the Sentence-Transformers model, with the input being the historical health description text and the output being the semantic vector. The semantic vector is then bound to the patient identifier, data source, source time, and unified field name and written into the patient-specific knowledge base. The patient's basic information is only used for patient identifier association and retrieval scope limitation and is not used as the object of medical entity extraction. Simultaneously, a departmental public knowledge base is generated according to departmental type; the departmental public knowledge base includes target inquiry field tables for different departments, association rules between symptoms and examination items, follow-up visit fields, medication contraindication rules, and health education content; the patient-specific knowledge base is used to subsequently recall the patient's historical health data records, and the departmental public knowledge base is used to subsequently generate target inquiry field tables, determine the applicability of fields, and provide health education content.

[0025] S3. Process the current query data and generate the current query context.

[0026] Patients select either text / image inquiry mode or video inquiry mode in the current health inquiry and enter the current inquiry data. The text content in the current chief complaint text and voice text record is merged to obtain the current chief complaint text. If either the current chief complaint text or the voice text record does not exist, the existing text content is used as the current chief complaint text. The current inquiry data includes the current chief complaint text, voice content, uploaded pictures and video data. When a patient selects the image and text inquiry mode, the system receives the patient's current chief complaint text and uploaded image; it uses a ResNet-50 model to classify the uploaded image, with the input being the uploaded image and the output being the classification result of the affected area image or the report image; When the image type is a lesion image, the uploaded image is normalized in size, brightness, and noise is filtered to obtain a preprocessed lesion image. The preprocessed lesion image is then input into the U-Net dual-channel segmentation model to obtain a lesion region mask and a visible body part region mask. A lesion image with the outlines of the lesion region and the visible body part region already labeled is obtained. The U-Net dual-channel segmentation model uses the lesion image with the outlines of the lesion region and the visible body part region labeled as training samples, with the preprocessed lesion image as input and the lesion region mask and the visible body part region mask as output. The ResNet-50 model is used to identify body part categories. The input is a preprocessed image of the affected area, and the output is one of the following body part categories: head, neck, chest, abdomen, back, upper limb, lower limb, hand, or foot. Color features and current area ratio are extracted from the mask of the affected area. The color features are obtained from the hue, saturation, and brightness values ​​of the pixels in the affected area in the HSV color space. The current area ratio is the ratio of the number of pixels in the mask of the affected area to the number of pixels in the mask of the visible body part area. When there are historical images of lesions from the same patient and the same body part category, the historical area ratio corresponding to that historical image is read, and the difference between the current area ratio and the historical area ratio is calculated to obtain a description of the area change; when there are no historical images of lesions from the same patient and the same body part category, only the current area ratio is recorded. Affected area image records are generated based on body part category, color features, current area proportion, and area change description. The affected area image records include structured fields and image description text. The structured fields include body part category, color features, current area proportion, and area change description. The image description text is generated by concatenating the structured fields in a preset field order. When the image type is a report image, PaddleOCR is used to perform text recognition on the report image to obtain the report recognition text. The inspection and testing item name, indicator name, indicator value, unit, and reference range are extracted from the report recognition text. Anomaly labels are generated according to whether the indicator value is below, above, or within the reference range, respectively. The currently uploaded report image record is generated based on the inspection and testing item name, indicator name, indicator value, unit, reference range, and anomaly label. The currently uploaded report image record includes structured fields and report description text, which is generated by concatenating the structured fields according to the preset field order. The current chief complaint text, the image record of the affected area, and the image record of the currently uploaded report are written into the current query context; the BERT sequence labeling medical entity recognition model is used to extract medical entities from the current chief complaint text, image description text, and report description text, respectively, to obtain symptom entities, body part entities, test indicator entities, numerical entities, and time entities; the Sentence-Transformers text vector model is used to convert the current chief complaint text, image description text, and report description text into semantic vectors respectively; When a patient selects the video consultation mode, an audio-visual connection is established; the Whisper speech recognition model is used to perform speech recognition on the audio stream to obtain a speech-text record. The difference value between adjacent frames is calculated for the video stream. The difference value between adjacent frames is the average of the absolute values ​​of the gray level difference between two adjacent grayscale images at the same pixel position. When the difference value of the next adjacent frame is greater than the average of the difference values ​​of the previous ten adjacent frames, the next frame is taken as a key frame. When the difference values ​​of the previous ten adjacent frames are less than ten, the average of all the difference values ​​of the adjacent frames that have been obtained is used as the comparison benchmark. Video representation records are extracted from keyframes; OpenCV is used to identify face regions and the mean and proportion of preset color channels in the face regions are calculated to obtain facial color records; MediaPipe Face Landmarker is used to obtain facial key points and generate expression records; MediaPipe Pose Landmarker is used to obtain human pose key points and generate pose records; MediaPipe Hand Landmarker is used to obtain hand key points and generate hand pointing area records based on the relative position between hand key points and human body parts. Based on the hand pointing area, posture recording, and facial expression recording, the video representation records are mapped to target query fields: when the hand pointing area is located in the abdominal area, or the posture recording includes bending over and is accompanied by frowning features in the facial expression recording, it corresponds to the abdominal discomfort field; when the hand pointing area is located in the foot area, or there is a repetitive rubbing of the feet, it corresponds to the foot discomfort field or the extremity numbness field; when the hand pointing area is located in the head area and the facial expression recording includes frowning features, it corresponds to the head discomfort field. The video representation records are converted into video representation description text, which includes facial color records, expression records, posture records, hand pointing area records, and corresponding target query fields. The Sentence-Transformers text vector model is used to convert the video representation description text into semantic vectors. The speech text records, video representation records, video representation description text, and their corresponding target query fields are written into the current query context.

[0027] S4. Based on the historical health data retrieved from the current chief complaint text, generate a dynamic patient profile and a target query field table.

[0028] The system reads the current chief complaint text and voice text record from the current query context. Using the current chief complaint text, department type, and patient identifier as search criteria, it retrieves candidate historical health data records from the patient-specific knowledge base. The Sentence-Transformers text vector model is used to convert the current chief complaint text and candidate historical health data records into semantic vectors respectively. The cosine similarity between the two is calculated, and the top K candidate historical health data records are selected from high to low according to the cosine similarity, where K is a positive integer not less than 1. A dynamic patient profile is generated based on the recalled candidate historical health data records; the dynamic patient profile includes basic patient information, historical diagnosis names related to the current chief complaint text, historical medication data, allergy history data, past medical history data, historical examination and test data, and follow-up visit fields; A target query field table is generated based on the current chief complaint text, department type, and department public knowledge base. The target query field table includes field name, field type, field source, and field sorting value. The current query context, patient dynamic profile, and target query field table are compared, and filled fields and missing fields are marked. Fields that already exist in the patient dynamic profile and do not conflict with the current query context are designated as referenceable fields. For referenceable fields that conflict with the current query context, the process is transferred to S5 to generate fields to be verified.

[0029] S5. Establish a set of field evidence and perform consistency verification to generate fields to be verified.

[0030] Consistency checks are performed on the current chief complaint text, voice text recording, affected area image recording, currently uploaded report image recording, video representation recording, and patient dynamic profile in the current inquiry context; a field evidence set is established for the same target inquiry field, which includes field name, source type, source content, source time, symptom entity, numerical entity, time entity, and content polarity; the source type includes current chief complaint text, voice text recording, affected area image recording, currently uploaded report image recording, video representation recording, and historical health data recording; Content polarity is generated according to the following rules: when the source content contains "have," "appear," "exist," "accompany," or "occurs," and the character distance between this word and the symptom entity in the same source content does not exceed L characters, the content polarity is marked as "existent"; when the source content contains "none," "absent," "deny," "not seen," or "not appeared," and the character distance between this word and the symptom entity in the same source content does not exceed L characters, the content polarity is marked as "non-existent"; all other cases are marked as "uncertain"; L is a positive integer not less than 1. Based on the set of field evidence, fields to be verified are generated. For the same target query field, if the content polarity of different sources is existence and non-existence respectively, a field to be verified with semantic antonyms is generated. If different sources correspond to the same test indicator, and the indicator values ​​after unit normalization are different, a field to be verified with numerical inconsistencies is generated. If different sources correspond to the same symptom entity, but the time entity falls within the time range of D days prior to the current health query initiation time and outside the time range respectively, a field to be verified with time inconsistencies is generated, where D is a positive integer not less than 1. When the source content required by the target query field is not covered by the current query context, patient dynamic profile, or field evidence set, the target query field is marked as the field corresponding to the supplementary evidence.

[0031] S6. Sort the candidate follow-up question fields based on the follow-up question priority value and generate the follow-up question order.

[0032] The missing fields, fields to be verified, and fields corresponding to evidence to be supplemented are collectively used as candidate follow-up questions; for each candidate follow-up question field, the main complaint correlation value, historical correlation value, image correlation value, video correlation value, and corrected correlation value are calculated respectively; The main complaint association value is calculated as follows: The candidate follow-up question fields and the current main complaint text are converted into semantic vectors using the Sentence-Transformers text vector model. The cosine similarity between the two semantic vectors is calculated, and the cosine similarity is mapped from [-1, 1] to [0, 1] to obtain the main complaint association value. The historical association value is calculated as follows: The Sentence-Transformers text vector model is used to convert the candidate follow-up fields and the historical diagnosis names in the patient dynamic profile into semantic vectors, respectively. The cosine similarity between the two semantic vectors is calculated, and the cosine similarity is mapped from [-1, 1] to [0, 1] to obtain the historical association value. The image association value is calculated as follows: The candidate follow-up question field and the image description text or report description text are converted into semantic vectors using the Sentence-Transformers text vector model, the cosine similarity between the two semantic vectors is calculated, and the cosine similarity is mapped from [-1, 1] to [0, 1] to obtain the image association value. The video association value is calculated as follows: The candidate follow-up question field and the video representation description text are converted into semantic vectors using the Sentence-Transformers text vector model. The cosine similarity between the two semantic vectors is calculated and the cosine similarity is mapped from [-1, 1] to [0, 1] to obtain the video association value. The corrected correlation value is the field correction rate of the candidate follow-up question field in similar health inquiries; similar health inquiries refer to historical health inquiries and current health inquiries being the same in at least two aspects: department type, disease name, and inquiry scenario; the disease name comes from the historical diagnosis name in the patient dynamic profile or the candidate disease name recalled by the department's public knowledge base based on the current chief complaint text; the inquiry scenario is one of the following: initial visit, follow-up visit, chronic disease follow-up, or report interpretation; the field correction rate is the ratio of the number of times the field has been modified by the doctor in the most recent N similar health inquiries to the total number of the most recent N similar health inquiries, where N is a positive integer not less than 1; The priority value for follow-up questions is the sum of the main complaint correlation value, historical correlation value, image correlation value, video correlation value, corrected correlation value, and verification flag value; the verification flag value of the field to be verified is 1, and the verification flag value of the field not to be verified is 0; when a source of data does not exist, the correlation value corresponding to that source is recorded as 0; The order of follow-up questions is generated from high to low priority value. If two candidate follow-up question fields have the same priority value, the order of follow-up questions is determined in the following order: fields to be verified first, fields whose source is a disease inquiry rule first, and fields with smaller sort values ​​first. After the patient answers follow-up questions, the answer is written into the current query context, and the missing field judgment, consistency check, and follow-up question priority value calculation are re-executed. For fields that the patient clearly cannot provide, where the answer to two consecutive follow-up questions does not contain the symptom entity, numerical entity, or time entity corresponding to the candidate follow-up question field, where the candidate follow-up question field does not match the patient's basic information, patient dynamic profile, or current query context, or where the patient requests manual intervention, the field is written into the "to be supplemented by doctor" column or transferred to doctor intervention. The above follow-up questioning process is repeated until all candidate follow-up question fields are processed, or the current health query is transferred to doctor intervention.

[0033] S7. Generate a draft electronic medical record to be confirmed based on the source tag.

[0034] After completing the follow-up questioning process in S6, a draft electronic medical record to be confirmed is generated based on the current inquiry context, patient dynamic profile, image records of the affected area, image records of currently uploaded reports, video characterization records, and historical examination and test data. The draft electronic medical record to be confirmed includes subjective data, objective data, assessment opinions, and treatment suggestions. Bind a source tag to each field in the draft electronic medical record to be confirmed; the source tag includes source type, source content, source time and source record number; the source type includes patient text input, voice text recording, affected area image recording, currently uploaded report image recording, video representation recording, historical health data recording and doctor correction recording; Content without a source marker is not written into the main text of the draft electronic medical record to be confirmed, but is written into the column to be supplemented by the doctor; for medical record fields that have been generated for verification in S5 and have been confirmed by follow-up in S6, the original source of difference and the patient's confirmation content are recorded in the source marker; when the draft electronic medical record to be confirmed is generated for the first time, the source marker does not include the doctor's correction record; historical doctor correction records are only used as source markers when the content of the medical record field actually references the historical correction results; after the doctor modifies the draft electronic medical record to be confirmed, the doctor's correction record is added to the modified field.

[0035] S8. The doctor confirms that the draft electronic medical record to be confirmed is written into the electronic medical record database and synchronized with the patient.

[0036] The draft electronic medical record to be confirmed is pushed to the doctor's workstation and set to the status of awaiting doctor's confirmation; the doctor confirms, modifies or returns the draft electronic medical record to be confirmed. When a doctor modifies a draft electronic medical record to be confirmed, the original content, the modified content, the modified field name, the doctor's identifier, and the modification time are saved to generate a doctor's correction record for this time. This doctor's correction record is then written into the inquiry strategy library. Subsequent similar health inquiries read the historical doctor's correction records in the inquiry strategy library and participate in the calculation of follow-up inquiry priority value according to the field correction rate calculation method in S6. When a doctor returns a draft electronic medical record pending confirmation, the reason for the return is recorded. If the reason for the return corresponds to a target query field, the reason for the return is recorded as a doctor's correction to that target query field and written into the query strategy library. If the reason for the return does not correspond to a target query field, a record for a field to be expanded is generated. The record for a field to be expanded includes the reason for the return, the name of the field to be added by the doctor, the department type, the disease name, the query scenario, and the source time. After the record for a field to be expanded is manually reviewed and approved, it is written into the department's public knowledge base as a new target query field or a field alias. Records for a field to be expanded that do not pass manual review are only retained as records and do not participate in the subsequent generation of target query fields. When a doctor confirms a draft electronic medical record pending confirmation, the draft status is changed to "confirmed and entered into the database," and the draft is written into the electronic medical record database. A medical record update notification is sent to the patient, allowing the patient to view the electronic medical record content, medical orders, and health education content confirmed by the doctor. At the same time, a closed-loop record is maintained, which includes the patient's identifier, inquiry start time, inquiry end time, inquiry mode, patient-specific knowledge base version, departmental public knowledge base version, draft number, list of fields modified by the doctor, doctor's confirmation time, and electronic medical record entry number.

[0037] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0038] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0039] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. An AI-powered health service method based on multimodal perception and a multi-source knowledge base, characterized in that, include: S1. Obtain patient basic information, historical electronic medical record data, historical examination and test information, historical medication data, allergy history data, and past medical history data to construct historical health data; S2. Generate a patient-specific knowledge base and a departmental public knowledge base based on historical health data; S3. Process the current query data and generate the current query context; S4. Based on the historical health data retrieved from the current chief complaint text, generate a dynamic patient profile and a target query field table; S5. Establish a set of field evidence and perform consistency verification to generate fields to be verified; S6. Sort candidate follow-up question fields based on follow-up question priority values ​​and generate follow-up question order; S7. Generate a draft electronic medical record to be confirmed based on the source tag; S8. The doctor confirms that the draft electronic medical record to be confirmed is written into the electronic medical record database and synchronized with the patient.

2. The AI ​​health service method based on multimodal perception and multi-source knowledge base according to claim 1, characterized in that: In S1, historical health data is constructed by acquiring basic patient information, historical electronic medical record data, historical examination and testing information, historical medication data, allergy history data, and past medical history data, as follows: Obtain the patient's account number, medical card number, electronic medical insurance voucher number, or the hospital's unique patient number, and convert the obtained number into a patient identifier; After successful identity verification, historical health data is retrieved based on the patient's identifier. This historical health data includes basic patient information, historical electronic medical record data, historical examination and testing data, historical medication data, allergy history data, and past medical history data. Basic patient information includes name, gender, age, and contact information. Historical electronic medical record data includes historical chief complaints, historical diagnoses, medical records, treatment records, and discharge summaries. Historical examination and testing data includes examination items, test indicators, indicator values, units, reference ranges, and examination / test dates. Historical medication data includes drug names, dosages, frequency of use, and time of use. Allergy history data includes allergy items, descriptions of allergic reactions, and the time of occurrence. Past medical history data includes the name of past illnesses, treatment dates, and the source of the records.

3. The AI ​​health service method based on multimodal perception and multi-source knowledge base according to claim 1, characterized in that: In S2, a patient-specific knowledge base and a departmental public knowledge base are generated based on historical health data, as follows: The historical health data obtained by S1 is processed for field normalization. Field normalization is performed according to the field normalization rule table, which records the original field name, unified field name, data type, unit format, and field meaning description. Historical health data with the same field meaning description but different original field names are written into the same unified field. After completing the field normalization process, a patient-specific knowledge base is generated according to data type. For basic patient information, patient identifier, name, gender, age, and contact information are saved as index information. For historical electronic medical record data, historical chief complaints, progress notes, treatment records, and discharge summaries with the same visit date, department name, and historical diagnosis name are grouped together and arranged from earliest to latest according to the medical record creation time to generate historical electronic medical record text segments. Each historical electronic medical record text segment is bound to patient identifier, data source, visit date, department name, historical diagnosis name, and unified field name. For historical examination and test data, historical medication data, allergy history data, and past medical history data, the corresponding data source, source time, and unified field name are recorded respectively. Medical entities are extracted from historical health data containing medical content. These medical entities include symptom entities, disease entities, drug entities, examination entities, test indicator entities, treatment entities, and allergy entities. The extraction of medical entities is achieved through a medical named entity recognition model trained based on the Transformers framework. The input is the text content corresponding to historical electronic medical record text segments, historical examination and test data, historical medication data, allergy history data, or past history data. The output is the medical entity type, entity text, and the position of the entity in the text. Data containing medical entities is organized into historical health description text, which includes unified field names, field content, data source, and source time. The semantic vector of the historical health description text is obtained through the Sentence-Transformers model, with the input being the historical health description text and the output being the semantic vector. The semantic vector is then bound to the patient identifier, data source, source time, and unified field names and written into the patient-specific knowledge base. Basic patient information is used only for patient identification and search scope limitation, and is not used as a medical entity extraction object; At the same time, a departmental public knowledge base is generated according to the department type; the departmental public knowledge base includes target inquiry field tables under different departments, association rules between symptoms and examination items, follow-up visit fields, medication contraindication rules, and health education content.

4. The AI ​​health service method based on multimodal perception and multi-source knowledge base according to claim 1, characterized in that: In S3, the current query data is processed and the current query context is generated as follows: Patients select either text / image inquiry mode or video inquiry mode in the current health inquiry and enter the current inquiry data. The text content in the current chief complaint text and voice text record is merged to obtain the current chief complaint text. If either the current chief complaint text or the voice text record does not exist, the existing text content is used as the current chief complaint text. The current inquiry data includes the current chief complaint text, voice content, uploaded pictures and video data. When the patient selects the text and image inquiry mode, the system receives the patient's current chief complaint text and uploaded images. The ResNet-50 model is used to classify uploaded images. The input is the uploaded image, and the output is the classification result of the affected area image or report image. When the image type is a picture of a lesion, the uploaded image is normalized in size, normalized in brightness, and noise is filtered out to obtain a preprocessed picture of the lesion. The preprocessed affected area image is input into the U-Net dual-channel segmentation model to obtain the affected area region mask and the visible body part region mask; the affected area image with the outline of the affected area region and the outline of the visible body part region is obtained. The U-Net dual-channel segmentation model uses the affected area image with the outline of the affected area region and the outline of the visible body part region as training samples, with the preprocessed affected area image as input and the affected area region mask and the visible body part region mask as output. The ResNet-50 model is used to identify body part categories. The input is a preprocessed image of the affected area, and the output is one of the following body part categories: head, neck, chest, abdomen, back, upper limb, lower limb, hand, or foot. Color features and current area ratio are extracted from the mask of the affected area. The color features are obtained from the hue, saturation, and brightness values ​​of the pixels in the affected area in the HSV color space. The current area ratio is the ratio of the number of pixels in the mask of the affected area to the number of pixels in the mask of the visible body part area. When there are historical images of lesions from the same patient and the same body part category, the historical area ratio corresponding to that historical image is read, and the difference between the current area ratio and the historical area ratio is calculated to obtain a description of the area change; when there are no historical images of lesions from the same patient and the same body part category, only the current area ratio is recorded. Affected area image records are generated based on body part category, color features, current area proportion, and area change description. The affected area image records include structured fields and image description text. The structured fields include body part category, color features, current area proportion, and area change description. The image description text is generated by concatenating the structured fields in a preset field order. When the image type is a report image, PaddleOCR is used to perform text recognition on the report image to obtain the report recognition text. The inspection and testing item name, indicator name, indicator value, unit, and reference range are extracted from the report recognition text. Anomaly labels are generated according to whether the indicator value is below, above, or within the reference range, respectively. The currently uploaded report image record is generated based on the inspection and testing item name, indicator name, indicator value, unit, reference range, and anomaly label. The currently uploaded report image record includes structured fields and report description text, which is generated by concatenating the structured fields according to the preset field order. The current chief complaint text, the image record of the affected area, and the image record of the currently uploaded report are written into the current query context; the BERT sequence labeling medical entity recognition model is used to extract medical entities from the current chief complaint text, image description text, and report description text, respectively, to obtain symptom entities, body part entities, test indicator entities, numerical entities, and time entities; the Sentence-Transformers text vector model is used to convert the current chief complaint text, image description text, and report description text into semantic vectors respectively; When a patient selects the video consultation mode, an audio-visual connection is established; the Whisper speech recognition model is used to perform speech recognition on the audio stream to obtain a speech-text record. The difference value between adjacent frames is calculated for the video stream. The difference value between adjacent frames is the average of the absolute values ​​of the gray level difference between two adjacent grayscale images at the same pixel position. When the difference value of the next adjacent frame is greater than the average of the difference values ​​of the previous ten adjacent frames, the next frame is taken as a key frame. When the difference values ​​of the previous ten adjacent frames are less than ten, the average of all the difference values ​​of the adjacent frames that have been obtained is used as the comparison benchmark. Video representation records are extracted from keyframes; OpenCV is used to identify face regions and the mean and proportion of preset color channels in the face regions are calculated to obtain facial color records; MediaPipe Face Landmarker is used to obtain facial key points and generate expression records; MediaPipe Pose Landmarker is used to obtain human pose key points and generate pose records; MediaPipe Hand Landmarker is used to obtain hand key points and generate hand pointing area records based on the relative position between hand key points and human body parts. Based on the hand pointing area, posture recording, and facial expression recording, the video representation records are mapped to target query fields: when the hand pointing area is located in the abdominal area, or the posture recording includes bending over and is accompanied by frowning features in the facial expression recording, it corresponds to the abdominal discomfort field; when the hand pointing area is located in the foot area, or there is a repetitive rubbing of the feet, it corresponds to the foot discomfort field or the extremity numbness field; when the hand pointing area is located in the head area and the facial expression recording includes frowning features, it corresponds to the head discomfort field. The video representation records are converted into video representation description text, which includes facial color records, expression records, posture records, hand pointing area records, and corresponding target query fields. The Sentence-Transformers text vector model is used to convert the video representation description text into semantic vectors. The speech text records, video representation records, video representation description text, and their corresponding target query fields are written into the current query context.

5. The AI ​​health service method based on multimodal perception and multi-source knowledge base according to claim 1, characterized in that: In S4, based on historical health data recall from the current chief complaint text, a dynamic patient profile and a target query field table are generated, as follows: The system reads the current chief complaint text and voice text record from the current query context. Using the current chief complaint text, department type, and patient identifier as search criteria, it retrieves candidate historical health data records from the patient-specific knowledge base. The Sentence-Transformers text vector model is used to convert the current chief complaint text and candidate historical health data records into semantic vectors respectively. The cosine similarity between the two is calculated, and the top K candidate historical health data records are selected from high to low according to the cosine similarity, where K is a positive integer not less than 1. Dynamic patient profiles are generated based on the recalled candidates' historical health data records. The patient dynamic profile includes basic patient information, historical diagnosis names related to the current chief complaint text, historical medication data, allergy history data, past medical history data, historical examination and test data, and follow-up visit fields; Generate a target query field table based on the current chief complaint text, department type, and department public knowledge base; The target query field table includes field name, field type, field source, and field sorting value; compare the current query context, patient dynamic profile, and target query field table, and mark filled fields and missing fields; for fields that already exist in the patient dynamic profile and do not conflict with the current query context, they are designated as referenceable fields; for referenceable fields that conflict with the current query context, the process is transferred to S5 to generate fields to be verified.

6. The AI ​​health service method based on multimodal perception and multi-source knowledge base according to claim 1, characterized in that: In S5, a set of field evidence is established and consistency verification is performed to generate fields to be verified, as follows: Perform consistency checks on the current chief complaint text, voice text record, affected area image record, currently uploaded report image record, video representation record, and patient dynamic profile in the current inquiry context; For the same target inquiry field, establish a field evidence set. The field evidence set includes field name, source type, source content, source time, symptom entity, numerical entity, time entity, and content polarity. The source type includes current chief complaint text, voice text record, affected area image record, currently uploaded report image record, video representation record, and historical health data record. Content polarity is generated according to the following rules: when the source content contains "have," "appear," "exist," "accompany," or "occurs," and the character distance between this word and the symptom entity in the same source content does not exceed L characters, the content polarity is marked as "existent"; when the source content contains "none," "absent," "deny," "not seen," or "not appeared," and the character distance between this word and the symptom entity in the same source content does not exceed L characters, the content polarity is marked as "non-existent"; all other cases are marked as "uncertain"; L is a positive integer not less than 1. Based on the set of field evidence, fields to be verified are generated. For the same target query field, if the content polarity of different sources is existence and non-existence respectively, a field to be verified with semantic antonyms is generated. If different sources correspond to the same test indicator, and the indicator values ​​after unit normalization are different, a field to be verified with numerical inconsistencies is generated. If different sources correspond to the same symptom entity, but the time entity falls within the time range of D days prior to the current health query initiation time and outside the time range respectively, a field to be verified with time inconsistencies is generated, where D is a positive integer not less than 1. When the source content required by the target query field is not covered by the current query context, patient dynamic profile, or field evidence set, the target query field is marked as the field corresponding to the supplementary evidence.

7. The AI ​​health service method based on multimodal perception and multi-source knowledge base according to claim 1, characterized in that: In S6, candidate follow-up question fields are sorted based on follow-up question priority values, and a follow-up question order is generated as follows: The missing fields, fields to be verified, and fields corresponding to evidence to be supplemented are collectively used as candidate follow-up questions; for each candidate follow-up question field, the main complaint correlation value, historical correlation value, image correlation value, video correlation value, and corrected correlation value are calculated respectively; The main complaint association value is calculated as follows: The candidate follow-up question fields and the current main complaint text are converted into semantic vectors using the Sentence-Transformers text vector model. The cosine similarity between the two semantic vectors is calculated, and the cosine similarity is mapped from [-1, 1] to [0, 1] to obtain the main complaint association value. The historical association value is calculated as follows: The Sentence-Transformers text vector model is used to convert the candidate follow-up fields and the historical diagnosis names in the patient dynamic profile into semantic vectors, respectively. The cosine similarity between the two semantic vectors is calculated, and the cosine similarity is mapped from [-1, 1] to [0, 1] to obtain the historical association value. The image association value is calculated as follows: The candidate follow-up question field and the image description text or report description text are converted into semantic vectors using the Sentence-Transformers text vector model, the cosine similarity between the two semantic vectors is calculated, and the cosine similarity is mapped from [-1, 1] to [0, 1] to obtain the image association value. The video association value is calculated as follows: The candidate follow-up question field and the video representation description text are converted into semantic vectors using the Sentence-Transformers text vector model. The cosine similarity between the two semantic vectors is calculated and the cosine similarity is mapped from [-1, 1] to [0, 1] to obtain the video association value. The corrected correlation value is the field correction rate of the candidate follow-up question field in similar health inquiries; similar health inquiries refer to historical health inquiries and current health inquiries being the same in at least two aspects: department type, disease name, and inquiry scenario; the disease name comes from the historical diagnosis name in the patient dynamic profile or the candidate disease name recalled by the department's public knowledge base based on the current chief complaint text; the inquiry scenario is one of the following: initial visit, follow-up visit, chronic disease follow-up, or report interpretation; the field correction rate is the ratio of the number of times the field has been modified by the doctor in the most recent N similar health inquiries to the total number of the most recent N similar health inquiries, where N is a positive integer not less than 1; The priority value for follow-up questions is the sum of the main complaint correlation value, historical correlation value, image correlation value, video correlation value, corrected correlation value, and verification flag value; the verification flag value of the field to be verified is 1, and the verification flag value of the field not to be verified is 0; when a source of data does not exist, the correlation value corresponding to that source is recorded as 0; The order of follow-up questions is generated from high to low priority value. If two candidate follow-up question fields have the same priority value, the order of follow-up questions is determined in the following order: fields to be verified first, fields whose source is a disease inquiry rule first, and fields with smaller sort values ​​first. After the patient answers the follow-up questions, the answer is written into the current query context, and the missing field judgment, consistency check and follow-up question priority value calculation are re-executed; for fields that the patient clearly cannot provide, whose answers to two consecutive follow-up questions do not contain the symptom entity, numerical entity or time entity corresponding to the candidate follow-up question field, whose candidate follow-up question field does not match the patient's basic information, patient dynamic profile or current query context, or whose patient requests manual intervention, the field is written into the "to be supplemented by doctor" column or transferred to doctor intervention; Repeat the above follow-up questioning process until all candidate follow-up question fields have been processed, or the current health inquiry is transferred to the doctor's control.

8. The AI ​​health service method based on multimodal perception and multi-source knowledge base according to claim 1, characterized in that: In S7, a draft electronic medical record to be confirmed is generated based on the source tag, as follows: After completing the follow-up questioning process in S6, a draft electronic medical record to be confirmed is generated based on the current inquiry context, patient dynamic profile, image records of the affected area, image records of currently uploaded reports, video characterization records, and historical examination and test data. The draft electronic medical record to be confirmed includes subjective data, objective data, assessment opinions, and treatment suggestions. Bind a source tag to each field in the draft electronic medical record to be confirmed; the source tag includes source type, source content, source time, and source record number; Source types include patient text input, voice and text recordings, affected area image records, currently uploaded report image records, video representation records, historical health data records, and doctor correction records; Content without a source marker is not written into the main text of the draft electronic medical record to be confirmed, but is written into the column to be supplemented by the doctor; for medical record fields that have been generated in S5 and have been confirmed by follow-up questioning in S6, the original source of difference and the patient's confirmation content are recorded in the source marker. When generating the first draft of an electronic medical record to be confirmed, the source marker does not include the doctor's revision record. Historical doctor correction records are only used as source markers when the content of medical record fields actually references historical correction results; After the doctor modifies the draft electronic medical record to be confirmed, the doctor will add the doctor's correction record to the modified fields.

9. The AI ​​health service method based on multimodal perception and multi-source knowledge base according to claim 1, characterized in that: In S8, the doctor confirms that the draft electronic medical record to be confirmed is written into the electronic medical record database and synchronized with the patient, as follows: The draft electronic medical record to be confirmed is pushed to the doctor's workstation and set to the status of awaiting doctor's confirmation; the doctor confirms, modifies or returns the draft electronic medical record to be confirmed. When a doctor modifies a draft electronic medical record to be confirmed, the original content, the modified content, the modified field names, the doctor's identifier, and the modification time are saved, a doctor's modification record is generated, and the doctor's modification record is written to the query strategy database. Subsequent similar health inquiries read historical doctor correction records from the inquiry strategy library and participate in the calculation of follow-up inquiry priority value according to the field correction rate calculation method in S6; When a doctor returns a draft electronic medical record pending confirmation, the reason for the return is recorded. If the reason for the return corresponds to a target query field, the reason for the return is recorded as a doctor's correction to that target query field and written into the query strategy library. If the reason for the return does not correspond to a target query field, a record for a field to be expanded is generated. The record for a field to be expanded includes the reason for the return, the name of the field to be added by the doctor, the department type, the disease name, the query scenario, and the source time. After the record for a field to be expanded is manually reviewed and approved, it is written into the department's public knowledge base as a new target query field or a field alias. Records with fields that fail manual review will only be retained and will not be used to generate subsequent target query fields; When a doctor confirms a draft electronic medical record pending confirmation, the draft status is changed to "confirmed and entered into the database," and the draft is written into the electronic medical record database. A medical record update notification is sent to the patient, allowing the patient to view the electronic medical record content, medical orders, and health education content confirmed by the doctor. At the same time, a closed-loop record is maintained, which includes the patient's identifier, inquiry start time, inquiry end time, inquiry mode, patient-specific knowledge base version, departmental public knowledge base version, draft number, list of fields modified by the doctor, doctor's confirmation time, and electronic medical record entry number.

Images