Nuclear emergency medical rescue question and answer method and device based on VR scene and medium

Abstract

The invention belongs to the technical field of data processing, and provides a nuclear emergency medical rescue question answering method and device based on a VR scene and a medium, and the method comprises the steps: firstly obtaining an original question proposed by a user, VR visual data and a current medical rescue task; inputting the original problem and the current medical rescue task into a retrieval enhancement generation module to obtain initial context information; inputting the initial context information, the VR visual data and the current medical rescue task into an adaptive adjustment module to obtain a retrieval result; and finally, inputting the original question and the retrieval result into the large language model to obtain a standard answer corresponding to the original question. According to the method, the information is retrieved from the private knowledge base by using the RAG to expand the LLM knowledge boundary, and meanwhile, the VR visual data and the current medical rescue task information are introduced, so that the question answering system can perform adaptive adjustment according to a real-time scene, the pertinence and the practicability of answering are improved, and the efficiency and the accuracy of the question answering system are improved.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a nuclear emergency medical rescue question-and-answer method, device, and medium based on a VR scenario. Background Art

[0002] In the field of nuclear emergency medical rescue, traditional training methods are unable to meet the demands of rehearsing complex rescue scenarios due to the high risk and non-repeatability of real-world environments. VR technology, by creating immersive virtual scenes, can simulate the emergency response process at a nuclear emergency medical rescue site, including key operations such as on-site command, on-site inspection, rapid triage, decontamination, medical treatment, and casualty evacuation, providing a safe and controllable training environment for rescue personnel. However, trainees may encounter various problems during operations, requiring quick access to technical support and operational guidance through a Q&A database.

[0003] With the rapid development of artificial intelligence and natural language processing technologies, large language models (LLMs), trained through deep learning, possess powerful natural language processing capabilities and are widely used in various fields. However, because their knowledge is limited to the training data, they are prone to "hallucinations" when handling specialized domain questions, resulting in responses lacking authenticity and accuracy.

[0004] To address this issue, Retrieval-Augmented Generation (RAG) technology has emerged. It expands the knowledge boundaries of LLMs by incorporating information retrieved from private knowledge bases into prompts, showing broad application prospects in NLP fields such as text summarization and dialogue systems. Existing large-model question-answering generation technologies based on retrieval augmentation, while combining knowledge bases with large language models for domain knowledge question answering, have significant drawbacks. The retrieved content often contains invalid information, which interferes with the large model's responses, affecting answer quality and making it difficult to meet users' demand for accurate answers. Summary of the Invention

[0005] In view of this, the present invention provides a nuclear emergency medical rescue question-and-answer method, device, and medium based on a VR scenario, aiming to solve the problem of low answer accuracy in the prior art.

[0006] A first aspect of an embodiment of the present invention provides a nuclear emergency medical rescue question-and-answer method based on a VR scenario, comprising:

[0007] Obtain original questions raised by users, VR visual data, and current medical rescue tasks;

[0008] Input the original question and the current medical rescue task into the retrieval enhancement generation module to obtain the initial context information;

[0009] Input the initial context information, VR visual data and current medical rescue task into the adaptive adjustment module to obtain the retrieval results;

[0010] The original question and retrieval results are input into the large language model to obtain the standard answer corresponding to the original question.

[0011] In one possible implementation, the initial context information, VR visual data, and the current medical rescue task are input into the adaptive adjustment module to obtain retrieval results, including:

[0012] Analyze VR visual data based on current medical rescue tasks to obtain user-focused information;

[0013] The initial context information is processed according to the user's attention information to obtain the retrieval results.

[0014] In one possible implementation, VR visual data is analyzed based on the current medical rescue mission to obtain user-focused information, including:

[0015] According to the current medical rescue mission requirements, the features of the injured, equipment and environment are extracted from the VR visual data as static features;

[0016] Taking the current medical rescue mission requirements into consideration, operational features are extracted from VR visual data as dynamic features;

[0017] Determine the user's attention information based on static and dynamic features.

[0018] In one possible implementation, the initial context information, VR visual data, and the current medical rescue task are input into the adaptive adjustment module to obtain retrieval results, including:

[0019] Analyze VR visual data based on current medical rescue tasks to obtain user attention and non-attention information;

[0020] The initial context information is processed according to the user's attention information and non-attention information to obtain the retrieval results.

[0021] In one possible implementation, VR visual data is analyzed based on the current medical rescue mission to obtain the user's attention information and non-attention information, including:

[0022] According to the current medical rescue mission requirements, the features of the injured, equipment and environment are extracted from the VR visual data as static features;

[0023] Taking the current medical rescue mission requirements into consideration, operational features are extracted from VR visual data as dynamic features;

[0024] Determine the user's attention information based on dynamic features;

[0025] Determine the user's non-attention information based on static features and attention information.

[0026] In one possible implementation, the initial context information, VR visual data, and the current medical rescue task are input into the adaptive adjustment module to obtain retrieval results, including:

[0027] Perform eye movement analysis on VR visual data to obtain eye movement scanning paths;

[0028] According to the current medical rescue task and eye movement scanning path, VR visual data is analyzed to obtain the user's strong attention information, weak attention information and non-attention information;

[0029] The initial context information is processed according to the strong attention information, weak attention information and non-attention information to obtain the retrieval results.

[0030] In one possible implementation, VR visual data is analyzed based on the current medical rescue mission and eye movement scanning path to obtain the user's strong attention information, weak attention information, and non-attention information, including:

[0031] According to the current medical rescue mission requirements, the features of the injured, equipment and environment are extracted from the VR visual data as static features;

[0032] Taking the current medical rescue mission requirements into consideration, operational features are extracted from VR visual data as dynamic features;

[0033] Determine the user's strong focus information based on the eye movement scanning path and dynamic characteristics;

[0034] Determine the user's weak attention information based on dynamic features and strong attention information;

[0035] The user's non-attention information is determined based on static features, strong attention information and weak attention information.

[0036] In one possible implementation, the original question and the current medical rescue task are input into the retrieval enhancement generation module to obtain initial context information, including:

[0037] Convert the original question and the current medical rescue task text into question word vectors;

[0038] Perform semantic retrieval in the knowledge base based on the question word vector to obtain context information.

[0039] A second aspect of an embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the first aspect are implemented.

[0040] A third aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the first aspect are implemented.

[0041] The embodiments of the present invention provide a method, device, and medium for question-and-answering nuclear emergency medical rescue in a VR scenario. The method first obtains the original question posed by the user, VR visual data, and the current medical rescue task; then, the original question and the current medical rescue task are input into a retrieval enhancement generation module to obtain initial context information; the initial context information, VR visual data, and the current medical rescue task are then input into an adaptive adjustment module to obtain retrieval results; and finally, the original question and retrieval results are input into a large language model to obtain a standard answer corresponding to the original question. The present invention utilizes RAG to retrieve information from a private knowledge base to expand the LLM knowledge boundary, while simultaneously introducing VR visual data and current medical rescue task information, enabling the question-and-answer system to perform adaptive adjustments based on the real-time scenario, improving the pertinence and practicality of the answers, and enhancing the efficiency and accuracy of the question-and-answer system. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 This is a diagram illustrating an application scenario of a nuclear emergency medical rescue question-and-answer method based on a VR scenario provided by an embodiment of the present invention;

[0044] Figure 2 This is a flowchart of an implementation of a nuclear emergency medical rescue question-and-answer method based on a VR scenario provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In the following description, specific details such as particular system structures and techniques are provided for purposes of illustration, not limitation, to facilitate a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0046] Figure 1 This is an application scenario diagram of the nuclear emergency medical rescue question-and-answer method based on VR scenario provided by an embodiment of the present invention. The method of the present invention can be applied to this scenario, including but not limited to. Figure 1 As shown, the scenario includes: VR equipment, motion capture system and server.

[0047] Among them, the VR device 11 includes VR glasses and a VR backpack; the motion capture system 12 includes a motion capture suit provided with marker points and inertial sensors, and a camera for shooting the marker points.

[0048] During nuclear emergency medical rescue simulation training, trainees wear motion capture clothing 11, VR glasses 12, and a VR backpack 13. Data collected by the inertial sensors on the motion capture clothing 11 and the marker data captured by the camera 14 are reported in real time to a server 15. Server 15 processes this data using a corresponding algorithm to capture the trainee's actual movements, which are then displayed on the VR glasses via the VR backpack. A language / text input device 16 is provided on the VR backpack 13 or VR glasses 12. Through this input device 16, users can ask questions to the question-and-answer database system. This input device transmits the user's questions to server 15 via the VR backpack 13. The question-and-answer database system on server 15 analyzes the questions using the nuclear emergency medical rescue question-and-answer method based on VR scenarios, obtains standard answers to the questions, and provides feedback to the trainees.

[0049] Figure 2 This is a flowchart of the implementation of the nuclear emergency medical rescue question-answering method based on VR scenario provided by an embodiment of the present invention. Figure 2 As shown, in some embodiments, a nuclear emergency medical rescue question-answering method based on a VR scenario includes:

[0050] S210, obtaining the original question raised by the user, VR visual data, and the current medical rescue task;

[0051] In an embodiment of the present invention, in a VR training scenario for nuclear emergency medical rescue, trainees can ask questions based on concerns they encounter during operation, such as, "When conducting contamination testing on a casualty, the instrument displays abnormal values. What should I do?" The system collects these questions through various input interfaces, such as voice recognition devices or text input boxes. Voice recognition technology converts the trainee's voice content into text format and performs pre-processing operations such as noise reduction and speech enhancement to improve recognition accuracy. If text input is used, the input content is formatted and grammatically checked to ensure that the system accurately understands the question, laying the foundation for subsequent processing.

[0052] Based on the pre-set nuclear emergency rescue drill process and the actual on-site situation, the system will identify the current medical rescue task. For example, at one stage of the drill, the task might be to quickly triage and provide initial treatment for a group of simulated casualties, prioritizing the seriously injured and ensuring that all patients receive basic medical care. This task information is communicated to the Q&A system through the system's task scheduling module. Dynamic changes on-site, such as new casualties, allow for real-time updates and adjustments to the task, ensuring the Q&A system consistently supports the current core mission.

[0053] Traditional training methods have limitations when dealing with complex, real-world rescue scenarios. While VR technology can create a simulated training environment, trainees still face numerous challenges operating within it, requiring precise technical support and operational guidance. In nuclear emergency medical rescue scenarios, VR equipment captures rich visual data, such as the victim's injuries, the on-site environment, and medical equipment. After receiving the user's question, the system inputs the VR visual data along with the question into the visual question-answering module. For example, when assessing the victim's injury, the system can more intuitively and accurately understand the context surrounding the question, enabling more precise analysis and answers. The virtual scene simulates the entire nuclear emergency rescue process, including on-site command, on-site inspection, rapid classification, decontamination, medical treatment, and patient evacuation. Visual question-answering technology provides answers to questions encountered by trainees during their operations, combining real-time visual information from the VR scene. For example, during protective clothing donning and doffing training, if a trainee has questions about a particular step, the visual question-answering system can provide targeted guidance based on the donning and doffing movements displayed in the VR scene and the current scene.

[0054] The present invention uses RAG to retrieve information from a private knowledge base to expand the LLM knowledge boundary, and at the same time introduces VR visual data and current medical rescue mission information, so that the question-answering system can be adaptively adjusted according to the real-time scenario, thereby improving the pertinence and practicality of the answers, and improving the efficiency and accuracy of the question-answering system.

[0055] S220, inputting the original question and the current medical rescue task into the retrieval enhancement generation module to obtain initial context information;

[0056] In some embodiments, the original question and the current medical rescue task are input into a retrieval enhancement generation module to obtain initial context information, including: converting the original question and the current medical rescue task text into question word vectors; performing semantic retrieval in the knowledge base based on the question word vectors to obtain context information.

[0057] In an embodiment of the present invention, the original question and the current medical rescue task enter the system in the form of natural language text. Before converting them into word vectors, the system will first preprocess these texts. The preprocessed text will be sent to a special processing model to map discrete data (such as text, images) into low-dimensional continuous vectors to capture the relationship between texts, such as an Embedding model. The Embedding model will convert each word in it, such as "trauma", "initial treatment", "injury assessment", "classification", etc., into a vector with a specific dimension (such as 300 dimensions). These vectors not only contain the semantic information of the words, but also reflect the semantic association between words through the relative position and distance relationship between vectors. Combining these word vectors in the order in the text forms the question word vector and the task word vector, which completely encode the semantic features of the original question and the current medical rescue task, and provide an effective data form for subsequent semantic retrieval.

[0058] The knowledge base stores a vast amount of knowledge related to nuclear emergency medical rescue. To ensure efficient retrieval, the system pre-indexes this knowledge. Typically, data structures such as inverted indexes are used to associate each document and each knowledge point in the knowledge base with corresponding keywords. For example, for a document on the treatment of injured personnel, all keywords involved are extracted and indexed with the document. This allows the system to quickly locate potentially relevant documents or knowledge points when a question word vector is entered.

[0059] The system calculates semantic matching between question word vectors and indexes in the knowledge base. Common calculation methods include cosine similarity and Euclidean distance. Taking cosine similarity as an example, it measures semantic similarity between the question word vector and each document vector in the knowledge base by calculating the cosine of the angle between them. The closer the cosine value is to 1, the more similar the directions of the two vectors are, and the more similar the corresponding text semantics are. The system calculates all relevant vectors in the knowledge base and sorts the search results in descending order of similarity. If the cosine similarity between the question word vector and the vector corresponding to a particular knowledge point is high, that knowledge point is prioritized for retrieval. The system then selects the most relevant documents or knowledge points based on a preset threshold or ranking. If the threshold is set to 0.8, only documents with a cosine similarity greater than 0.8 are retained. This filtered information may come from different documents or different knowledge fragments. The system integrates it, removes duplication, and organizes it according to a certain logical order (such as relevance or importance) to form the initial context information. This information may include the initial diagnosis method of the injured, the treatment principles of different types of wounds, the medical resources available on site, etc., providing a rich knowledge base for subsequent analysis and answers of the large language model.

[0060] In some embodiments, the original question and the current medical rescue task are input into a retrieval enhancement generation module to obtain initial context information, including: performing natural language processing on the original question to extract core entities; based on the nuclear emergency medical rescue knowledge graph, according to the current medical rescue task, through the entity triple relationship and preset rules, querying the corresponding related entities associated with the core entity to form a multimodal entity set; converting the multimodal entity set into a word vector in a unified vector space, performing semantic retrieval in the knowledge base, and obtaining context information.

[0061] In some embodiments, based on the nuclear emergency medical rescue knowledge graph, according to the current medical rescue task, through the entity triple relationship and preset rules, the corresponding related entities associated with the core entity are queried to form a multimodal entity set, including: parsing the type of the current medical rescue task and extracting task keywords; based on the task keywords and the nuclear emergency medical rescue knowledge graph, the related entities corresponding to the core entity are extracted, and the dynamic weight of each related entity is calculated according to the association relationship in the nuclear emergency medical rescue knowledge graph; according to the dynamic weight, the core entity of the multimodal entity set and the corresponding related entities are formed into a multimodal entity set.

[0062] In an embodiment of the present invention, when the original question and the current medical rescue task are input into the search enhancement generation module to obtain initial context information, natural language processing technology is first used to deeply analyze the original question. Through steps such as word segmentation, part-of-speech tagging, and named entity recognition, core entities are accurately extracted from the user's question. For example, for the question "Treatment plan for nuclear radiation casualties experiencing dyspnea and skin redness and swelling," key entities such as "dyspnea" and "Treatment plan" can be identified. Then, based on the pre-constructed nuclear emergency medical rescue knowledge graph and the current medical rescue task type, the core entities are multi-dimensionally expanded through entity triple relationships (such as "symptoms - belongs to - injury manifestations" and "treatment process - includes - first aid steps") and preset rules (such as "multiple symptoms → initiate consultation process") to identify related entities. Each expanded entity is then assigned an initial weight using association strength parameters in the graph (such as co-occurrence frequency and expert annotation priority). Subsequently, a multimodal entity set consisting of text entities (such as treatment specification clauses), visual entities (such as VR image features of the injured), and operational entities (such as equipment operation steps) is converted into a semantic vector of unified dimension through a cross-modal pre-training model. A hybrid retrieval strategy of "vector retrieval + keyword retrieval" is implemented in the private knowledge base: vector retrieval matches semantically related knowledge fragments based on cosine similarity, while keyword retrieval performs precise matching on standardized entities. Finally, the retrieval results are fused and sorted according to the dynamic weights of the entities (for example, graph relevance accounts for 40%, modality matching accounts for 30%, and task urgency accounts for 30%), duplicate information is removed, and initial context information containing task background, symptom analysis, operation procedures, risk warnings, etc. is generated.

[0063] S230, inputting the initial context information, VR visual data, and the current medical rescue task into the adaptive adjustment module to obtain a retrieval result;

[0064] In this embodiment of the present invention, the adaptive adjustment module simultaneously receives initial context information, VR visual data, and information about the current medical rescue mission. The initial context information includes knowledge content related to the original question and mission, retrieved from a knowledge base. The VR visual data provides real-time images of the rescue scene, which is of primary interest to the user. The module uses image recognition and computer vision technologies to extract features from the VR visual data. For example, this can identify the location, size, and severity of a patient's wounds and determine the model and operating status of medical equipment. Furthermore, the module provides an in-depth interpretation of the initial context information in conjunction with the current medical rescue mission.

[0065] S240: Input the original question and the search results into the large language model to obtain the standard answer corresponding to the original question.

[0066] In some embodiments, the original question and the search results are input into the large language model to obtain a standard answer corresponding to the original question, including: unifying the format of the original question and the search results to obtain a tokenized sequence, and adding scene labels; parsing the semantic structure of the original question according to the intent decomposition model, identifying the core requirement type, and generating a corresponding prompt word template; inputting the tokenized original question, search results, scene labels, task priority labels and prompt word templates into the large language model, triggering the model's conditional generation mechanism, and performing logical reasoning in combination with the entity relationship of the nuclear emergency medical rescue knowledge graph; and performing consistency verification on the answer output by the large language model, performing content screening by comparing the matching degree between the key entities in the search results and the answer content, and generating a standard answer.

[0067] In some embodiments, the semantic structure of the original question is parsed according to the intent decomposition model, the core requirement type is identified, and a corresponding prompt word template is generated, including: parsing the semantic structure of the original question through the intent decomposition model to identify the core requirement type; retrieving a predefined template library according to the core requirement type to obtain multiple prompt words; constructing a template based on the multiple prompt words, and filling the entity, scenario label, and priority corresponding to the prompt word into the template placeholder to obtain a prompt word template.

[0068] In an embodiment of the present invention, first, the original question and the retrieval results are formatted in a unified manner, the text is split into sequences by a word segmentation tool, and converted into a tokenized sequence that can be recognized by the model, and a scene label is attached to the sequence to limit the question and answer domain. Then, the original question is deeply semantically parsed using the intent decomposition model, key entities are extracted by named entity recognition technology, and the core requirement type is identified in combination with the pre-trained domain classification model. Subsequently, a set of suitable prompt words (such as "treatment steps", "instrument use", "drug dosage") is retrieved from the predefined template library according to the type, and based on the scene label, task priority label and extracted entity, it is filled into the template placeholder to construct a structured prompt word template (for example, "In {scene label}, for {symptom}, please provide {prompt word} with {priority} response, and it is necessary to clarify {equipment} operating parameters and {drug} usage specifications").

[0069] After inputting the tokenized original question, search results, scenario labels, task priority labels, and prompt word templates into the large language model, the model's conditional generation mechanism is triggered. The model combines the entity relationships in the nuclear emergency medical rescue knowledge graph and enhances the use of domain knowledge through the graph attention mechanism to generate a preliminary answer containing key operational steps (such as "Immediately perform chest compressions at a rate of 100-120 times / minute, and simultaneously prepare the defibrillator to charge to 200J"). Finally, a consistency verification module compares the answer content with the key entities in the search results, eliminates unverified fictitious content, and adjusts the answer's details based on real-time operational data in the VR scene, ultimately generating a standard answer that meets professional standards and meets user needs.

[0070] In embodiments of the present invention, the original question and search results may differ in structure and expression, requiring format unification and standardization. The original question may be a casual expression in natural language, while the search results may contain data from different sources and formats, such as text snippets and graphical information. The system converts this information into a standard format that can be understood by the large language model. This typically involves tokenizing the text, breaking it down into individual words, and adding necessary identifiers to distinguish between the different parts of the question and search results.

[0071] Extract key information from the original question and search results, such as the core concepts and task requirements in the question, and important knowledge points and operation steps in the search results. If key information is found to be missing or incomplete, the system will attempt to supplement it from other relevant data sources.

[0072] After receiving the preprocessed information, the large language model first conducts a deep understanding and semantic analysis of the original question. Using its internal language understanding mechanisms, it analyzes the question's grammatical structure, semantic intent, and underlying domain knowledge. For questions related to nuclear emergency medical rescue, the model identifies key terms and understands the relationships between them, determining the core requirement of the question: seeking an emergency treatment for a specific casualty condition.

[0073] The model uses the search results as a crucial reference, integrating the extensive knowledge acquired during pre-training to perform logical reasoning and analysis. It comprehensively considers retrieved medical knowledge, operational specifications, case studies, and other information to comprehensively address the problem. When addressing the aforementioned casualty issue, the model draws on the search results for specific steps and precautions for various nuclear emergency medical rescue missions, as well as its own understanding of medical knowledge, such as the mechanisms of action of different medications and the applicability of various wound treatment methods, to construct a logically coherent solution.

[0074] In some cases, the large language model may generate multiple candidate answers. The system will screen and sort the answers based on factors such as accuracy, relevance, completeness, and degree of match with the question. Priority is given to those answers that are most accurate, comprehensive, and closely related to the question as the final output. If there are multiple answers that meet certain requirements, the system may present them to the user in order of quality, or select them based on the user's specific needs (such as conciseness and level of detail). After the above series of rigorous processing steps, the answer finally output by the large language model can accurately and comprehensively answer the original question, provide reliable information support for users in nuclear emergency medical rescue scenarios, and help them better deal with various complex rescue situations.

[0075] S230 is described below through three embodiments, but is not intended to be limiting.

[0076] Example 1:

[0077] In some embodiments, the initial context information, VR visual data and the current medical rescue task are input into the adaptive adjustment module to obtain retrieval results, including: analyzing the VR visual data according to the current medical rescue task to obtain the user's attention information; processing the initial context information according to the user's attention information to obtain the retrieval results.

[0078] In an embodiment of the present invention, the current medical rescue mission is an important basis for analyzing VR visual data. In a nuclear emergency medical rescue scenario, if the current mission is to provide emergency treatment to the wounded in a specific area, the system will use advanced image recognition technology and computer vision algorithms to focus on the visual elements related to the mission in that area. Target detection algorithms are used to identify the location, number, and general condition of the wounded, such as whether there are obvious wounds or bleeding. At the same time, medical equipment on site, such as testing instruments, first aid kits, stretchers, etc., is identified, and the working status of the equipment is determined, such as whether the equipment is operating normally and the display status of the indicator lights on the equipment.

[0079] In addition to identifying visual elements, the system also analyzes the user's behavior in the VR scene. By tracking information such as the user's gaze direction, dwell time, and operating actions, the user's focus can be inferred. If a user stares at a detection instrument with a flashing alarm light for a long time and attempts to operate on the operating panel, the system can infer that the user may have questions about the abnormality of the instrument and the correct operation method. In addition, the user's interaction with other rescuers can also provide clues. If the user asks a companion for advice on how to deal with a certain person's injury, then the injury information of the injured person is likely to be the focus of the user's attention.

[0080] Combining the results of visual element recognition and user behavior analysis, the system generates a detailed list of information of interest to the user. This list may include the patient's specific injuries (such as wound location, type, and severity), abnormal conditions of medical equipment (such as error messages and faulty components), and operational procedures closely related to the current task (such as whether the first aid steps being performed are correct and what the next steps should be). This information serves as key evidence for subsequent processing of the initial context information.

[0081] In order to more accurately filter the initial context information, the system will set corresponding filtering thresholds and weights based on different types of information and task requirements. A higher weight is given to keywords or key phrases that are directly related to the information that the user is concerned about; a lower weight is set for some general information with weaker relevance. For example, if the user is concerned about the failure of a specific model of testing equipment, the weight of information related to the instrument model and troubleshooting will be increased, while the weight of general introductions to other types of medical equipment will be lowered. At the same time, a relevance threshold is set, and only when the degree of match between information and the information that the user is concerned about exceeds this threshold will it be retained in the screening results.

[0082] The system filters the initial context information according to the set thresholds and weights. During the screening process, each information fragment in the initial context information will be evaluated. By calculating the degree of match between the keywords in the information fragment and the keywords in the user's information list, combined with the pre-set weights, a comprehensive relevance score is obtained. Only information fragments with scores exceeding the threshold will be retained. For example, a piece of information about common faults and solutions of the detection instrument is likely to be retained because its keywords highly match the information that the user is concerned about and have a high weight; while a piece of information about the use of equipment in other medical scenarios will be excluded because of its low relevance and score that does not meet the threshold.

[0083] After screening, the system integrates the remaining information. During this integration process, the information is arranged according to a certain logical sequence, such as the order in which the problem should be solved and the level of importance of the information. If the filtered information covers multiple aspects, such as analysis of the cause of an instrument failure, specific solution steps, and precautions, the system will organize this information into a coherent search result. This search result will cover the user's questions as completely as possible, providing high-quality input for the subsequent large language model to generate accurate answers, effectively reducing the impact of invalid information on the answers and ensuring the effectiveness of the question-and-answer process.

[0084] In some embodiments, VR visual data is analyzed according to the current medical rescue mission to obtain user attention information, including: extracting wounded characteristics, equipment characteristics and environmental characteristics from VR visual data as static features according to the task requirements of the current medical rescue mission; extracting operation characteristics from VR visual data as dynamic features according to the task requirements of the current medical rescue mission; and determining user attention information based on static features and dynamic features.

[0085] In an embodiment of the present invention, different medical rescue tasks focus on different characteristics of the injured. In a nuclear emergency medical rescue scenario, if the current task is to perform emergency triage of the injured, the system will focus on extracting features such as the severity of the injured person's injury and the location of the injury from the VR visual data. Through image recognition technology, a pre-trained model is used to identify the type of wound of the injured person, such as open wound or closed wound, and to determine the size, depth and bleeding of the wound. At the same time, the body posture and expression of the injured person are analyzed to assist in determining their state of consciousness and degree of pain. If the task is to perform a detailed injury diagnosis on a specific injured person, the system will further focus on the characteristics related to the injured person's vital signs, such as respiratory rate, pulse, etc. This information can be obtained through medical monitoring equipment simulated in the VR scene. For example, the system obtains the heart rate data of the injured person by identifying the waveforms and values ​​on the screen of the electrocardiogram monitor in the VR scene.

[0086] The current medical rescue mission also determines the equipment features that need to be paid attention to. If the mission is to conduct inspections, the system will extract features such as the model, working status, and displayed values ​​of the inspection equipment from the VR visual data. By identifying the appearance of the equipment, the model of the equipment is determined, and then the corresponding equipment knowledge base is called to obtain the normal operating parameter range of the equipment of this model. Observe the display screen of the equipment, extract the displayed values ​​and warning information, and determine whether the equipment is operating normally. If abnormal warning lights or error messages are found on the device display, the system will record these as key equipment features. If the mission is to use first aid equipment to treat the wounded, the system will pay attention to the availability of the equipment, such as whether the first aid kit is intact, whether the medicines and equipment are complete, etc.

[0087] Environmental characteristics are also closely related to medical rescue missions. In nuclear emergency rescue scenarios, if the mission involves conducting rescue operations within a specific area, the system will extract the layout characteristics of that area, including the size of the rescue site, the location of passageways, and the presence of obstacles. This information is crucial for rationally planning rescue procedures and personnel routes. Furthermore, the system also monitors potential hazards in the environment. By analyzing environmental elements in VR visual data, it determines whether the environment poses a threat to the rescue operation and whether special protective measures are necessary.

[0088] Based on the current medical rescue mission requirements, the system extracts operational features from VR visual data. These features reflect the dynamic information during the rescue process. If the current task is to treat the wounds of the wounded, the system will extract the rescuer's operating steps, such as the order and method of disinfection and bandaging, the medical tools used, and other information. By identifying the rescuer's hand movements and the tools they hold, combined with motion capture technology in the VR scene, the operation process can be accurately recorded. For example, the system can determine whether the rescuer has followed the correct process to disinfect the wound first and then bandage it, and whether the bandaging technique is in compliance with the specifications. If the task is to operate the detection equipment, the system will extract the dynamic process of the equipment operation, including the order of pressing the operation buttons, the adjustment of parameter settings, and other information, to determine whether the operation is correct.

[0089] The system conducts a comprehensive analysis of the extracted static and dynamic features. For example, if static features indicate that a patient's injuries are severe and contaminated, while dynamic features indicate that the rescuers' handling of the patient's wounds is questionable, the user's focus is likely on how to properly treat the patient's wound while preventing further spread of contamination. If device features indicate an abnormality in the detection equipment, while operational features indicate that the rescuers are attempting to adjust the equipment parameters, the user's focus may be on the cause of the abnormality and the correct adjustment method. By combining static and dynamic features, the system can more comprehensively and accurately infer the user's focus.

[0090] Based on the comprehensive analysis, the system generates a user-focused information set. This set includes information closely related to the questions users might ask. For example, in the example above, this might include optimal treatment options for specific injuries, protective measures against contamination, and solutions to equipment anomalies. This information provides a strong basis for subsequent contextual information retrieved by the enhanced generation module, thereby reducing the impact of invalid information and improving the accuracy and effectiveness of the question-answering system.

[0091] Example 2:

[0092] In some embodiments, the initial context information, VR visual data and the current medical rescue task are input into the adaptive adjustment module to obtain retrieval results, including: analyzing the VR visual data according to the current medical rescue task to obtain the user's attention information and non-attention information; processing the initial context information according to the user's attention information and non-attention information to obtain retrieval results.

[0093] In this embodiment of the present invention, "non-attention information" refers to critical information that is highly relevant to the core task but that the user is not paying attention to in the current scenario. The system mines this information in various ways. First, based on the task's specialized knowledge and logic, the system determines the importance of certain elements to task execution. For example, during a patient transport mission, an ambulance is present on-site, and its medicine inventory directly impacts subsequent treatment. However, the user may be focused on the patient and overlook this information. In this case, the ambulance's medicine inventory information is considered "non-attention information." Second, the system identifies "non-attention information" by comparing the user's visual focus and operational behavior with the task requirements. For example, when conducting contamination testing on a patient, if the user only focuses on the instrument's readings but fails to notice the calibration information displayed next to the instrument, which is crucial for the accuracy of the test results, this calibration information is considered "non-attention information." Furthermore, the system combines past experience and common problem scenarios to identify critical information that the user may have overlooked. For example, when treating a group of patients, while the user is currently focused on some seriously injured patients, they may not be aware of the potential risk of sudden deterioration among other less seriously injured patients. This potential risk information is also considered "non-attention information."

[0094] The system performs a preliminary screening of the initial context information based on the user's focus. Through keyword matching and semantic analysis, it selects content highly relevant to the information in question. If the user is interested in wound treatment for a particular patient, the system searches the initial context for information containing relevant information such as the steps for wound treatment, appropriate medications, and precautions for handling that type of wound. Information directly related to specific procedures such as wound disinfection, bandaging, and suturing is highlighted and sorted by relevance and importance, prioritizing the information fragments that are closely related to the user's focus.

[0095] Unfocused information plays an essential role in optimizing search results. The system compares unfocused information with the initial context information to ensure that information critical to the task, though not previously considered by the user, is included in the search results. For example, if the VR visual data reveals that the user overlooked information about low ambulance drug supplies, but the initial context information includes information about drug alternatives or nearby refill points, the system will add this information to the search results to supplement the potentially missed key knowledge. For important operating procedures or potential risks that the user may have overlooked, the system will enhance their presentation in the search results. For example, if the user overlooked the calibration requirements for the instrument during a test, and the initial context information contains detailed steps and instructions on the importance of calibration, the system will highlight this information to remind the user. Furthermore, for initial context information that is not strongly relevant to unfocused information or is of little value to the current task, the system will appropriately downgrade or exclude it, reducing the impact of redundant information on the search results.

[0096] After filtering and optimizing information based on both relevant and non-relevant information, the system integrates and organizes the remaining information. The information is organized into coherent search results based on a logical structure and the order of user understanding. If the search results involve multiple aspects, such as wound treatment methods, drug storage issues, testing precautions, etc., the system will summarize these contents separately. The core solution is presented first, and then the relevant explanations and supplementary information are gradually expanded to ensure that the search results can not only accurately answer the user's potential questions, but also provide users with comprehensive and clear knowledge support, so that they can be subsequently input into the large language model to generate high-quality answers.

[0097] In some embodiments, VR visual data is analyzed according to the current medical rescue mission to obtain the user's attention information and non-attention information, including: extracting the characteristics of the injured, equipment and environment as static features from the VR visual data according to the task requirements of the current medical rescue mission; extracting operation characteristics as dynamic features from the VR visual data according to the task requirements of the current medical rescue mission; determining the user's attention information based on the dynamic features; and determining the user's non-attention information based on the static features and attention information.

[0098] In an embodiment of the present invention, within a complex nuclear emergency medical rescue VR scenario, user actions can intuitively reflect their focus. The system determines the user's attention through in-depth analysis of dynamic features. Leveraging advanced motion capture and image analysis technologies, the system comprehensively tracks the user's behavior within the VR scenario. For example, when performing CPR on a casualty, the system accurately records the position, frequency, and depth of the user's hand pressure, as well as the details of artificial respiration. If the user's pressure position is inaccurate or the frequency is unstable, and they repeatedly adjust their movements, this indicates that the user may be unsure about the correct compression technique. Therefore, "correct CPR compression guidelines" are highly likely the user's focus. In addition to their own actions, the user's interactions with other elements within the scenario also contain important clues. In a nuclear emergency rescue scenario, if a user frequently clicks or views a specific part of a medical device, such as repeatedly checking the display of a tester's readings and communicating with nearby rescuers, this suggests that the user is likely concerned with the meaning and accuracy of the device's test data, as well as the impact of the current test results on rescue decisions. By identifying these interactions, the system can accurately infer the user's focus within the current task.

[0099] The order of operations during task execution is also key to identifying information of interest. For example, when treating and bandaging a wounded patient, if the user hesitates about the choice of bandaging materials and method after completing the disinfection steps, pauses for a long time, or attempts different operations, the system can determine that the user may have questions about the subsequent bandaging steps. For example, "the best bandaging materials and methods for this type of wound" will be identified as information of interest. By analyzing abnormal or hesitant steps in the operation process, the system can effectively capture the user's information of interest, which reflects the difficulties encountered by the user during task execution and the problems that need to be solved.

[0100] After identifying static features and information of interest, the system conducts detailed comparison and analysis to identify critical information that is highly relevant to the core mission but often overlooked by the user—information of no interest. In a nuclear emergency rescue scenario, static features for a casualty include information such as injury severity, location, and contamination. If a user is interested in the treatment of a casualty's surface wound, and static features indicate a potential risk of internal contamination, but the user's actions and attention do not address this, then "methods for detecting and treating internal contamination" would be considered information of no interest. This information is closely related to the current rescue mission and, if not addressed promptly, could have serious consequences for the casualty, yet the user is unaware of it during their current operations. From a device perspective, let's assume the current task is to monitor a casualty's vital signs using a specific model of medical equipment. Equipment features include the device's operating status, various parameters, and other indicators. If a user is focused on the heart rate data displayed by the device, but the device's static features indicate that its battery is running low, and the user is unaware of this information, but the battery issue could affect the continued use of the device and the accuracy of the monitoring results, then the "Impact of Low Device Battery on Monitoring Tasks and Countermeasures" will be identified as non-concern information. By comparing the key features of the device with the user's focus, the system can identify this type of easily overlooked but mission-critical information.

[0101] Environmental characteristics are also crucial in rescue missions. For example, in a nuclear accident rescue scenario, environmental characteristics include information such as the structural stability of buildings on site. If a user is focused on providing emergency treatment to the injured at the current location, and the static environmental characteristics of the scene indicate a risk of nearby building collapse, but the user fails to address this potential hazard during operation, then the "impact of the risk of nearby building collapse on rescue operations and preventive measures" becomes irrelevant information. By comprehensively analyzing environmental characteristics and user-focused information, the system identifies these potential factors that may affect the smooth progress of the rescue mission. Even though the user may not be currently aware of these factors, they are crucial for the comprehensive and safe completion of the rescue mission.

[0102] Example 3:

[0103] In some embodiments, the initial context information, VR visual data and the current medical rescue task are input into the adaptive adjustment module to obtain retrieval results, including: performing eye movement analysis on the VR visual data to obtain an eye movement scanning path; analyzing the VR visual data according to the current medical rescue task and the eye movement scanning path to obtain the user's strong attention information, weak attention information and non-attention information; processing the initial context information according to the strong attention information, weak attention information and non-attention information to obtain retrieval results.

[0104] In an embodiment of the present invention, in a VR scenario, the user's eye movement data is collected in real time using the VR device's built-in eye tracking technology. This data records information such as the user's gaze location, duration, and scan trajectory while performing the current medical rescue task. For example, when treating a wounded patient, the user's gaze may frequently shift between the patient's wound, the medical equipment being used, and the operating manual. Eye tracking technology can accurately record the trajectory and duration of these movements.

[0105] Based on the collected eye movement data, the system generates an eye movement scanning path. It can reflect the visualization or digital representation of the user's visual attention distribution in the VR scene. The eye movement data is processed by an algorithm, and the user's gaze points are connected to form a continuous scanning path. For example, when dealing with a complex nuclear detection device, the user's eye movement scanning path may first focus on the device's display screen to check various data indicators, then move to the operation button area, and then look at the device's connection lines. The path formed in this process is fully recorded by the system.

[0106] Analyze eye movement scanning paths in conjunction with current medical rescue tasks. If, during a wound treatment task, a user gazes at a specific wound site for an extended period of time, frequently lingers and focuses on that site, and simultaneously performs actions related to the wound, such as disinfection and bandaging, then information related to that wound, such as its type, severity, and treatment methods, is identified as strongly focused information. This is because the user's prolonged gaze and related actions indicate that this is the area of ​​focus and treatment, and is closely related to the core operation of the task.

[0107] For some areas or elements, although the user's eyes scan over them, the stay time is short and the operation behavior is not closely related to them. These corresponding information are determined to be weak attention information. In the process of treating the wounded, the user may occasionally look at the medical equipment next to them, but do not perform actual operation, just a brief look. At this time, the information of the medical equipment (such as the device name and basic functions) is weak attention information relative to the key information of wound treatment. It is related to the current task to a certain extent, but it is not the focus of the user's current operation.

[0108] Non-focused information is highly relevant to the current medical rescue task, but the user's eye scan path doesn't indicate their focus. The system identifies this important information by comparing the key information required for the task with the information covered by the user's eye scan path.

[0109] In some embodiments, VR visual data is analyzed according to the current medical rescue task and eye movement scanning path to obtain the user's strong attention information, weak attention information and non-attention information, including: extracting the characteristics of the injured, equipment and environment as static features from the VR visual data according to the task requirements of the current medical rescue task; extracting the operation characteristics as dynamic features from the VR visual data according to the task requirements of the current medical rescue task; determining the user's strong attention information according to the eye movement scanning path and dynamic features; determining the user's weak attention information according to the dynamic features and strong attention information; determining the user's non-attention information according to the static features, strong attention information and weak attention information.

[0110] In this embodiment of the present invention, the eye movement scan path reflects the user's gaze trajectory and focus area in the VR scene. In a nuclear emergency medical rescue scenario, the system leverages the VR device's eye tracking technology to obtain detailed user eye movement data, including gaze point, gaze duration, and scan direction. For example, in a casualty care scenario, if a user's eye movement scan path is focused on the patient's wound for a prolonged period, with frequent gaze and scans around the wound, this indicates that the user is paying close attention to the wound.

[0111] Dynamic features include information about the user's operational behavior in the VR scene. When the system finds that the user not only focuses their eyes on the wound, but also performs corresponding operational actions, such as disinfecting and bandaging the wound, it can further confirm that the information related to the wound is the user's strong attention information. Because the user's operational behavior matches the focus of their vision, it means that this is the object that the user is currently focusing on and paying attention to. For example, while the user stares at the wound for a long time and wipes the wound with a sterilized cotton ball, the location of the wound, the severity of the injury, the treatment method, and other information directly related to the wound can be identified as strong attention information. This information is crucial for the user to complete the current rescue mission and is the core problem that the user is actively seeking to solve.

[0112] After identifying strong attention information, the system further analyzes the user's dynamic characteristics, observing what other operations the user performs besides those directly related to the strong attention information. For example, when treating a wounded patient, the user's primary action may be to treat the wound, but they may occasionally look at nearby medical equipment, such as checking certain displayed data or operating buttons on the device. These objects that are somewhat related to the primary action but are not involved in the core action may contain weak attention information.

[0113] The information involved in these operations is compared with the strongly focused information. If it is found that this information has a relatively weak correlation with the core of the current task, and the user's attention to it is short-lived and the degree of interaction is low, then it can be determined to be weakly focused information. For example, when a user is treating a wounded person's wound, he occasionally glances at the vital signs monitor next to him, only briefly obtaining data such as heart rate and blood pressure, and does not perform in-depth operations on the monitor or pay long-term attention to it. At this time, the data displayed by the monitor and the basic functions of the monitor are weakly focused information relative to the strongly focused information of wound treatment. Although this information is not the core of the user's current task, it may also assist the user in completing the rescue task to a certain extent, or reflect the user's general understanding and attention to the entire rescue scene.

[0114] Static features cover a wide range of information in VR scenes, including features of the injured, equipment, and environment. In nuclear emergency rescue scenarios, this information includes the overall physical condition of the injured (other potential injuries besides wounds, whether there is a history of other diseases, etc.), detailed parameters of medical equipment (other functions and states besides those operated and focused on by the user), and various details of the rescue environment. Static features are compared with the identified strong and weak attention information. Information that is highly relevant to the core of the current medical rescue mission, but is not paid attention to in the user's eye movement scanning path and does not reflect related operational behavior in the dynamic features, is identified as non-attention information. Although this information is ignored by the user, it is crucial to the safety and integrity of the entire rescue mission. It can be subsequently attracted to the user's attention through system prompts and other means to ensure the smooth progress of the rescue mission.

[0115] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0116] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A nuclear emergency medical rescue question-answering method based on VR scenario, characterized in that: include: Obtain original questions raised by users, VR visual data, and current medical rescue tasks; Inputting the original question and the current medical rescue task into the retrieval enhancement generation module to obtain initial context information; Inputting the initial context information, the VR visual data, and the current medical rescue task into an adaptive adjustment module to obtain a retrieval result; The original question and the search results are input into the large language model to obtain a standard answer corresponding to the original question.

2. The nuclear emergency medical rescue question-answering method based on VR scenario according to claim 1 is characterized in that: Inputting the initial context information, the VR visual data, and the current medical rescue task into an adaptive adjustment module to obtain a retrieval result, including: Analyzing the VR visual data according to the current medical rescue mission to obtain user-focused information; The initial context information is processed according to the user attention information to obtain a retrieval result.

3. The nuclear emergency medical rescue question-answering method based on VR scenario according to claim 2 is characterized in that: According to the current medical rescue mission, the VR visual data is analyzed to obtain user's attention information, including: Extracting features of the injured, equipment, and environment from the VR visual data as static features according to the task requirements of the current medical rescue task; Extracting operational features from the VR visual data as dynamic features based on the task requirements of the current medical rescue task; The user's attention information is determined according to the static features and the dynamic features.

4. The nuclear emergency medical rescue question-answering method based on VR scenario according to claim 1 is characterized in that: Input the original question and the current medical rescue task into the retrieval enhancement generation module to obtain initial context information, including: Performing natural language processing on the original question to extract core entities; Based on the nuclear emergency medical rescue knowledge graph, according to the current medical rescue task, through the entity triple relationship and preset rules, the corresponding related entities associated with the core entity are queried to form a multimodal entity set; The multimodal entity set is converted into a word vector in a unified vector space, and a semantic search is performed in the knowledge base to obtain context information.

5. The nuclear emergency medical rescue question-answering method based on VR scenario according to claim 4 is characterized in that: Based on the nuclear emergency medical rescue knowledge graph and the current medical rescue mission, the corresponding related entities associated with the core entity are queried through entity triple relationships and preset rules to form a multimodal entity set, including: Analyze the type of the current medical rescue mission and extract mission keywords; Based on the task keywords and the nuclear emergency medical rescue knowledge graph, the related entities corresponding to the core entity are extracted, and the dynamic weight of each related entity is calculated according to the association relationship in the nuclear emergency medical rescue knowledge graph; According to the dynamic weight, a multimodal entity set core entity and corresponding related entities are formed into a multimodal entity set.

6. The nuclear emergency medical rescue question-answering method based on VR scenario according to claim 2 is characterized in that: Inputting the original question and the search results into the large language model to obtain a standard answer corresponding to the original question includes: Unifying the formats of the original question and the search results to obtain a tokenized sequence, and adding scene labels; Analyze the semantic structure of the original question based on the intent decomposition model, identify the core requirement type, and generate the corresponding prompt word template; The tokenized original question, search results, scenario labels, task priority labels, and prompt word templates are input into the large language model, triggering the model's conditional generation mechanism and performing logical reasoning based on the entity relationships in the nuclear emergency medical rescue knowledge graph. The answers output by the large language model are verified for consistency. By comparing the matching degree between the key entities in the search results and the answer content, content screening is performed to generate standard answers.

7. The nuclear emergency medical rescue question-answering method based on VR scenario according to claim 1 is characterized in that: The semantic structure of the original question is parsed based on the intent decomposition model, the core requirement type is identified, and the corresponding prompt word template is generated, including: Analyze the semantic structure of the original problem through the intention decomposition model to identify the core requirement type; Search the predefined template library based on the core requirement type to obtain multiple prompt words; A template is constructed according to the multiple prompt words, and the entities, scene labels, and priorities corresponding to the prompt words are filled into the template placeholders to obtain a prompt word template.

8. The nuclear emergency medical rescue question-answering method based on VR scenario according to claim 1 is characterized in that: Input the original question and the current medical rescue task into the retrieval enhancement generation module to obtain initial context information, including: Convert the original question and the current medical rescue task text into question word vectors; A semantic search is performed in the knowledge base based on the question word vector to obtain context information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the nuclear emergency medical rescue question-and-answer method based on the VR scenario as described in any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the nuclear emergency medical rescue question-and-answer method based on a VR scenario as described in any one of claims 1 to 8 above are implemented.

Images