Sampling guidance and detection task generation method and system based on multi-modal large model

Abstract

The application discloses a sampling guide and detection task generation method and system based on a multi-modal large model, and the application acquires multi-modal input data such as patient voice, identity information and historical detection information, recognizes the voice, extracts symptoms, parts and accompanying information to form structured patient information, dynamically generates follow-up contents according to information integrity conditions and performs multi-round interaction, inputs information meeting the conditions, POCT equipment executable items, sample types and detection equipment states into a task generation module together to generate structured detection tasks, generates individualized sampling guide information according to the detection tasks and patient characteristics and outputs the information through a multi-modal mode, and sends the output detection tasks and guide information to POCT equipment for execution after safety checking. The application solves the problems of rigid traditional POCT interaction processes, insufficient symptom collection and missing sampling operation guide, and realizes intelligent and individualized detection task generation and operation guide.

Description

Technical Field

[0001] This invention relates to the fields of medical human-computer interaction, POCT (point-of-care testing) task management, and artificial intelligence information processing. Specifically, it relates to a method, system, electronic device, and computer-readable storage medium deployed on a POCT mobile robot or POCT smart terminal, based on a multimodal large model for patient information collection, dynamic interaction, testing task generation, and sampling operation guidance. This invention generates POCT testing tasks, provides testing process prompts and sampling operation guidance, but does not directly diagnose or treat patients. Background Technology

[0002] Before performing a testing task, a POCT (point-of-care testing) robot typically needs to complete a series of human-computer interaction processes, including patient identification, symptom collection, test selection, and sample collection. However, the current human-computer interaction methods of POCT devices have the following prominent problems: First, the interaction method mainly relies on fixed touch screen menu selection, which is not user-friendly for elderly patients, patients with impaired consciousness in the emergency room, or patients with limited mobility, and the interaction process is rigid.

[0003] Second, symptom collection relies on fixed consultation templates and cannot dynamically follow up with questions based on the patient's answers. For example, after a patient complains of "chest pain," the system cannot automatically ask for key differential information such as the nature of the pain, duration, and location of radiation, resulting in incomplete information collection and difficulty in supporting the reasonable selection of subsequent testing items.

[0004] Third, the selection of testing items relies heavily on the experience and judgment of the operators. Medical staff with different years of experience may recommend different combinations of testing items for the same patient, lacking data-driven decision support.

[0005] Fourth, the sampling process lacks personalized guidance tailored to individual patient characteristics (such as age, vascular condition, and cooperation level) and the requirements of the testing items. Operators' mastery of sampling sites, techniques, and sample processing steps varies from person to person.

[0006] Fifth, if no safety constraints are set for direct voice-based human-computer dialogue, it may output unverified conclusions, which could mislead patients.

[0007] Therefore, there is a need for a human-computer interaction method that can achieve full collection of patient information, dynamic follow-up questioning, intelligent detection task generation, and personalized sampling guidance through multimodal interaction, while also having a safety constraint mechanism. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a sampling guidance and detection task generation method and system based on a multimodal large model that can achieve full collection of patient information, dynamic follow-up questioning, intelligent detection task generation and personalized sampling guidance through multimodal interaction, while having a safety constraint mechanism.

[0009] A sampling guidance and detection task generation method based on a multimodal large model includes: S1. Acquire patient-related features and at least two types of multimodal input data from POCT-executable detection items; wherein, the patient-related features specifically include: patient voice, patient identity information, patient interactive input, and historical detection information; S2. Perform speech recognition on the patient's speech, and extract at least one of the following from the recognition results: symptoms, location, time, severity, and accompanying information, to form structured patient information; S3. Based on the structured patient information, preset information integrity conditions, and interaction history, determine the information items to be supplemented, and generate follow-up questions corresponding to the information items to be supplemented through a multimodal large model; S4. Obtain the patient's answers to the follow-up questions and update the structured patient information until the structured patient information meets the conditions for generating the detection task; S5. Based on the structured patient information, POCT executable tests, sample type, sample requirements, and testing equipment status that meet the conditions for generating the testing task, a structured POCT testing task is generated; wherein, the candidate tests for the structured POCT testing task are screened according to the list of POCT executable tests and the testing equipment status; S6. Generate sampling guidance information based on the structured POCT detection task and patient-related characteristics, and output it through at least one of voice, display, animation, light or interactive confirmation interface; S7. Perform detection item matching, sample type matching, equipment capability matching, and safety output verification on the structured POCT detection task and the sampling guidance information, and send them to the POCT for execution after the verification is passed; S8. Before sending the execution, display the patient's identity, test items, sample requirements and operation prompts through the interactive interface, and obtain the operator's confirmation result.

[0010] Furthermore, the patient's identity information is obtained through at least one of the following methods: facial recognition, electronic medical insurance voucher, ID card, barcode, or QR code.

[0011] Furthermore, based on the obtained patient identity information, at least one of the following can be obtained: medical history, allergy history, previous test results, or previous test times associated with the patient identity information.

[0012] Furthermore, in step S2, the speech recognition specifically employs at least one of the following processing methods: far-field microphone array acquisition, noise reduction, speech activity detection, and automatic speech recognition.

[0013] Furthermore, step S4 includes a step for determining whether the structured patient information meets the conditions for generating the detection task. Specifically, determining whether the structured patient information meets the conditions for generating the detection task includes: determining whether the patient's identity has been confirmed, whether the detection object has been determined, whether the sample type can be determined, and whether the necessary information affecting the selection of detection items is complete.

[0014] Furthermore, the structured POCT testing task in step S5 specifically includes at least one of the following: testing items, testing package, sample type, sample volume, testing priority, estimated testing time, required reagent tray, sampling requirements, and manual confirmation identifier.

[0015] Furthermore, the security output verification in step S7 specifically includes at least one of the following: whitelist verification of detection items, equipment capability verification, sample type matching verification, reagent tray availability verification, and filtering of prohibited diagnostic statements.

[0016] Furthermore, when the generated interactive output contains a definitive diagnosis of a disease, a definitive treatment instruction, or a test item that is incompatible with the capabilities of the testing equipment, the interactive output is replaced, intercepted, or transferred to manual confirmation.

[0017] A sampling guidance and detection task generation system based on a multimodal large model includes: The multimodal acquisition module is used to acquire patient voice, identity information, interactive input, historical test information, and POCT equipment information; The speech recognition module is used to recognize the patient's speech and generate text content; An information structuring module is used to extract symptoms, location, time, severity, and accompanying information from the text content; The dynamic follow-up question module is used to generate follow-up questions based on structured patient information and information integrity conditions; The task generation module is used to generate structured POCT testing tasks based on structured patient information, device-executable items, and sample requirements. The sampling guidance module is used to generate sampling operation and sample processing guidance information based on the structured POCT detection task; The security verification module is used to perform project matching, sample matching, equipment capability matching, and security output verification on the detection task and sampling guidance information; The task execution interface is used to send the verified inspection tasks to the POCT inspection equipment or robot task management module.

[0018] A computer-readable storage medium having a computer program stored thereon, which, when run on a processor, causes the processor to perform the method.

[0019] This invention provides a method and system for generating sampling guidance and detection tasks based on a multimodal large model, which has the following beneficial effects: First, by integrating multimodal input data such as voice, identity information, and historical test data, and using a multimodal large model as the core of interactive decision-making, it has achieved intelligent processing of the entire process from patient information collection, dynamic follow-up questioning, test task generation to sampling guidance, breaking through the limitations of fixed menu selection and fixed template consultation of traditional POCT equipment.

[0020] Second, through a dynamic follow-up questioning mechanism, the multimodal big model judges missing information and generates targeted follow-up questions in real time based on the currently collected structured patient information, information integrity conditions, and interaction history. It simulates the doctor's consultation logic to carry out in-depth information collection, ensuring that the key information required for the generation of the detection task is fully acquired.

[0021] Third, during the generation of detection tasks, the semantic understanding and recommendation capabilities of the multimodal large model are combined with the actual execution capabilities of the POCT device (list of executable items, reagent tray status, sample type constraints) to ensure that the generated detection tasks are executable at the device level and to avoid generating detection suggestions that exceed the device's capabilities.

[0022] Fourth, it automatically generates personalized sampling operations and sample processing guidance based on the testing task and individual patient characteristics, and outputs them through various means such as voice and screen animation, reducing the difficulty for operators to operate different testing items and patient types.

[0023] Fifth, by setting up a security output verification mechanism, multiple filters are applied to the detection items and guidance information, including whitelists, equipment capabilities, sample matching, and prohibited statements, to ensure that the human-computer interaction output does not contain inappropriate content such as definitive disease diagnosis conclusions, thereby improving the security and reliability of the system. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0025] Figure 1 A flowchart of the method provided by the present invention; Figure 2 This is a schematic diagram of the system structure provided by the present invention. Detailed Implementation

[0026] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0027] The following detailed description of the implementation method of the present invention is in conjunction with the accompanying drawings. The description is only a partial embodiment and not all embodiments. For clarity, representations and descriptions unrelated to the present invention are omitted in the drawings and description.

[0028] To provide a clearer understanding of the technical features, objectives, and beneficial effects of this invention, the following detailed description of the technical solution is provided. Obviously, the described embodiments are only a portion of the embodiments of this invention, not all of them, and should not be construed as limiting the scope of implementation of this invention. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort are within the protection scope of this invention.

[0029] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a sampling guidance and detection task generation method and system based on a multimodal large model that can achieve full collection of patient information, dynamic follow-up questioning, intelligent detection task generation and personalized sampling guidance through multimodal interaction, while having a safety constraint mechanism.

[0030] A sampling guidance and detection task generation method based on a multimodal large model, such as Figure 1 As shown, it includes: S1. Acquire patient-related features and at least two types of multimodal input data from POCT-executable detection items; wherein, the patient-related features specifically include: patient voice, patient identity information, patient interactive input, and historical detection information; S2. Perform speech recognition on the patient's speech, and extract at least one of the following from the recognition results: symptoms, location, time, severity, and accompanying information, to form structured patient information; S3. Based on the structured patient information, preset information integrity conditions, and interaction history, determine the information items to be supplemented, and generate follow-up questions corresponding to the information items to be supplemented through a multimodal large model; S4. Obtain the patient's answers to the follow-up questions and update the structured patient information until the structured patient information meets the conditions for generating the detection task; S5. Based on the structured patient information, POCT executable tests, sample type, sample requirements, and testing equipment status that meet the conditions for generating the testing task, a structured POCT testing task is generated; wherein, the candidate tests for the structured POCT testing task are screened according to the list of POCT executable tests and the testing equipment status; S6. Generate sampling guidance information based on the structured POCT detection task and patient-related characteristics, and output it through at least one of voice, display, animation, light or interactive confirmation interface; S7. Perform detection item matching, sample type matching, equipment capability matching, and safety output verification on the structured POCT detection task and the sampling guidance information, and send them to the POCT for execution after the verification is passed; S8. Before sending the execution, display the patient's identity, test items, sample requirements and operation prompts through the interactive interface, and obtain the operator's confirmation result.

[0031] Patient identity information is obtained through at least one of the following methods: facial recognition, electronic medical insurance voucher, ID card, barcode, or QR code.

[0032] The obtained patient identity information includes at least one of the following: medical history, allergy history, previous test results, or previous test times associated with the patient identity information.

[0033] In step S2, speech recognition specifically employs at least one of the following processing methods: far-field microphone array acquisition, noise reduction, speech activity detection, and automatic speech recognition.

[0034] Step S4 includes a step for determining whether the structured patient information meets the conditions for generating the detection task. Specifically, determining whether the structured patient information meets the conditions for generating the detection task includes: determining whether the patient's identity has been confirmed, whether the detection object has been determined, whether the sample type can be determined, and whether the necessary information affecting the selection of detection items is complete.

[0035] The structured POCT testing task in step S5 specifically includes at least one of the following: testing items, testing package, sample type, sample volume, testing priority, estimated testing time, required reagent tray, sampling requirements, and manual confirmation mark.

[0036] The security output verification in step S7 specifically includes at least one of the following: whitelist verification of detection items, equipment capability verification, sample type matching verification, reagent tray availability verification, and filtering of prohibited diagnostic statements.

[0037] When the generated interactive output contains a definitive diagnosis of a disease, a definitive treatment instruction, or a test item that is incompatible with the capabilities of the testing equipment, the interactive output is replaced, intercepted, or transferred to manual confirmation.

[0038] A sampling guidance and detection task generation system based on a multimodal large model, such as Figure 2 As shown, it includes: The multimodal acquisition module is used to acquire patient voice, identity information, interactive input, historical test information, and POCT equipment information; The speech recognition module is used to recognize the patient's speech and generate text content; An information structuring module is used to extract symptoms, location, time, severity, and accompanying information from the text content; The dynamic follow-up question module is used to generate follow-up questions based on structured patient information and information integrity conditions; The task generation module is used to generate structured POCT testing tasks based on structured patient information, device-executable items, and sample requirements. The sampling guidance module is used to generate sampling operation and sample processing guidance information based on the structured POCT detection task; The security verification module is used to perform project matching, sample matching, equipment capability matching, and security output verification on the detection task and sampling guidance information; The task execution interface is used to send the verified inspection tasks to the POCT inspection equipment or robot task management module.

[0039] A computer-readable storage medium having a computer program stored thereon, which, when run on a processor, causes the processor to perform the method.

[0040] Example: Detection task generation and sampling guidance for emergency chest pain patients This embodiment describes the complete interactive process of an emergency chest pain patient using a POCT robot to generate test tasks and guide the sampling operation.

[0041] Step S1: Acquire multimodal input data. The patient arrives at the POCT robot and completes identity verification by scanning their electronic medical insurance voucher. The POCT robot retrieves the patient's past medical history (5-year history of hypertension) and allergy history (no known drug allergies) from the hospital information system. The robot's far-field microphone array is in standby mode.

[0042] Step S2: Speech Recognition and Information Structuring. The patient states their chief complaint: "I've had chest tightness for about half an hour, and my left shoulder is also a bit uncomfortable. I've been sweating a lot." The robot's speech recognition module performs noise reduction and automatic speech recognition to obtain the recognized text. The information structuring module extracts the following from the text: chief complaint: "chest tightness," location: "chest," duration: "about 30 minutes," and accompanying symptoms: "left shoulder discomfort" and "sweating." The initial structured patient information record is: {Chief complaint: chest tightness, location: chest, duration: about 30 minutes, accompanying symptoms: left shoulder discomfort, sweating, information completeness: to be supplemented}.

[0043] Step S3: Dynamically generate follow-up questions. Based on the current structured patient information and combined with the chest pain differential diagnosis knowledge graph, the multimodal big data model determines that the following key information is still missing: the nature of the pain (dull / stabbing / squeezing), aggravating or relieving factors, and a history of similar attacks. The big data model generates the following follow-up question: "Do you feel this discomfort as a dull, throbbing pain, or a stabbing pain like being pricked with needles?" Step S4: Obtain answers and update information. The patient answers: "A dull, pressing feeling." The system appends this answer to the structured patient information and updates the information completeness assessment. The system continues to ask: "Have you had a cold or fever recently? Have you had similar symptoms before?" The patient answers: "No cold. I've had chest tightness before, but never as severe as this." After three rounds of follow-up questions, the information completeness conditions are met (chief complaint clear, nature of pain clear, duration known, accompanying symptoms obtained), and the system enters the task generation stage.

[0044] Step S5: Generate a structured testing task. The testing task generation module matches the structured patient information with the list of currently executable items on the POCT device. The current device is equipped with a cardiac marker testing reagent tray (capable of detecting cTnI, Myo, and CK-MB), a coagulation function reagent tray (capable of detecting D-Dimer), and a basic biochemistry reagent tray. The large model, combining patient symptom characteristics and device capabilities, generates the following structured testing task: {Test items: combined detection of three cardiac markers, D-Dimer detection; Sample type: venous whole blood; Sample volume: 5 mL; Detection priority: high; Estimated detection time: 15 minutes; Required reagent trays: cardiac marker tray, coagulation function tray; Manual confirmation: yes}.

[0045] The detection task generation module performs device-side screening of candidate detection items. This screening is based at least on the list of detection items that the POCT device can perform, the current status of the detection device, the reagent tray type, and the validity status of the reagent tray, so that the generated structured detection task corresponds to the detection items and sample requirements that the current device can perform. Step S6: Generate sampling guidance information. Based on the testing task requirements and patient characteristics (age 65 years, previous blood collection records), the sampling guidance module generates personalized guidance information: "Please select the patient's median cubital vein for blood collection. The recommended needle angle is 15 to 30 degrees. Collect approximately 5 mL of venous whole blood, inject it into the purple anticoagulant tube, and gently invert and mix 8 to 10 times. For blood gas analysis samples, please be sure to expel air bubbles from the syringe." The guidance information is broadcast via the POCT robot's speaker, while an animated guidance screen is displayed simultaneously.

[0046] Step S7: Security Output Verification. The security verification module verifies the generated detection task and guidance information. The detection items "three myocardial markers" and "D-Dimer" are both on the device whitelist and match the currently loaded reagent tray. The sample type "venous whole blood" matches the device's capabilities. The guidance information does not contain a definitive disease diagnosis conclusion, thus passing the security output verification.

[0047] Step S8: Task Confirmation and Execution. After successful verification, the testing task is displayed on the POCT robot's touchscreen, including patient identity, testing items, sample requirements, and a "Confirm" button. The operator (nurse) confirms the information is correct and clicks "Confirm," sending the testing task to the POCT module for execution. The system automatically records the voice-recognized text of this interaction, structured patient information, follow-up questions, the generated testing task, and the operator's confirmation result.

[0048] After the security verification is passed, the system does not send the testing task directly. Instead, it displays the patient's identity, testing items, sample requirements and operation prompts on the interactive interface. After obtaining the operator's confirmation, the structured POCT testing task is sent to the POCT testing device or robot task management module.

[0049] It should be noted that the specific scenarios, disease names, testing items, and operating steps in the above embodiments are only used to illustrate the technical solutions of the present invention and do not constitute a limitation on the scope of protection of the present invention. In practical applications, the follow-up questioning strategy, testing item recommendation logic, and sampling guidance content can be adaptively adjusted according to specific clinical scenarios and POCT equipment configurations.

[0050] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A sampling guidance and detection task generation method based on a multimodal large model, characterized in that, include: S1. Acquire patient-related features and at least two types of multimodal input data from POCT-executable detection items; wherein, the patient-related features specifically include: patient voice, patient identity information, patient interactive input, and historical detection information; S2. Perform speech recognition on the patient's speech, and extract at least one of the following from the recognition results: symptoms, location, time, severity, and accompanying information, to form structured patient information; S3. Construct a multimodal large model, determine the information items to be supplemented based on the structured patient information, preset information integrity conditions and interaction history, and generate follow-up questions corresponding to the information items to be supplemented through the multimodal large model; S4. Obtain the patient's answers to the follow-up questions and update the structured patient information until the structured patient information meets the conditions for generating the detection task; S5. Based on the structured patient information, POCT executable tests, sample type, sample requirements, and testing equipment status that meet the conditions for generating the testing task, a structured POCT testing task is generated; wherein, the candidate tests for the structured POCT testing task are screened according to the list of POCT executable tests and the testing equipment status; S6. Generate sampling guidance information based on the structured POCT detection task and patient-related characteristics, and output it through at least one of voice, display, animation, light or interactive confirmation interface; S7. Perform detection item matching, sample type matching, equipment capability matching, and safety output verification on the structured POCT detection task and the sampling guidance information, and send them to the POCT for execution after the verification is passed; S8. Before sending the execution, display the patient's identity, test items, sample requirements and operation prompts through the interactive interface, and obtain the operator's confirmation result.

2. The method according to claim 1, characterized in that, The patient's identity information is obtained through at least one of the following methods: facial recognition, electronic medical insurance voucher, ID card, barcode, or QR code.

3. The method according to claim 2, characterized in that, Based on the obtained patient identity information, at least one of the following can be obtained: medical history, allergy history, previous test results, or previous test times associated with the patient identity information.

4. The method according to claim 1, characterized in that, In step S2, speech recognition specifically employs at least one of the following processing methods: far-field microphone array acquisition, noise reduction, speech activity detection, and automatic speech recognition.

5. The method according to claim 1, characterized in that, Step S4 includes a step for determining whether the structured patient information meets the conditions for generating the detection task. Specifically, determining whether the structured patient information meets the conditions for generating the detection task includes: determining whether the patient's identity has been confirmed, whether the detection object has been determined, whether the sample type can be determined, and whether the necessary information affecting the selection of detection items is complete.

6. The method according to claim 1, characterized in that, The structured POCT testing task in step S5 specifically includes at least one of the following: testing items, testing package, sample type, sample volume, testing priority, estimated testing time, required reagent tray, sampling requirements, and manual confirmation mark.

7. The method according to claim 1, characterized in that, The security output verification in step S7 specifically includes at least one of the following: whitelist verification of detection items, equipment capability verification, sample type matching verification, reagent tray availability verification, and filtering of prohibited diagnostic statements.

8. The method according to claim 1, characterized in that, When the generated interactive output contains a definitive diagnosis of a disease, a definitive treatment instruction, or a test item that is incompatible with the capabilities of the testing equipment, the interactive output is replaced, intercepted, or transferred to manual confirmation.

9. A sampling guidance and detection task generation system for implementing the method of any one of claims 1 to 8 for a multimodal large model, characterized in that, include: The multimodal acquisition module is used to acquire patient voice, identity information, interactive input, historical test information, and POCT equipment information; The speech recognition module is used to recognize the patient's speech and generate text content; An information structuring module is used to extract symptoms, location, time, severity, and accompanying information from the text content; The dynamic follow-up question module is used to generate follow-up questions based on structured patient information and information integrity conditions; The task generation module is used to generate structured POCT testing tasks based on structured patient information, device-executable items, and sample requirements. The sampling guidance module is used to generate sampling operation and sample processing guidance information based on the structured POCT detection task; The security verification module is used to perform project matching, sample matching, equipment capability matching, and security output verification on the detection task and sampling guidance information; The task execution interface is used to send the verified inspection tasks to the POCT inspection equipment or robot task management module.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run on a processor, it causes the processor to perform the method according to any one of claims 1 to 8.

Images