Multi-modal data fused mixed reality simulation medical system and method
By acquiring the set of occupants to generate free space, calculating clearance values and bottleneck points, identifying clearance events, and establishing clearance occupancy accounts, the problem of existing systems being unable to identify high-risk moments with narrow main instrument channels is solved. This enables precise risk management and collaborative guidance in mixed reality simulation medical systems, improving training efficiency and safety.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-06
- Publication Date
- 2026-04-10
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing mixed reality simulation medical systems cannot proactively identify high-risk moments where the passageway for main instruments is extremely narrow but no collision has occurred. They also lack quantitative assessment mechanisms, which leads to the omission of core risk scenarios in real surgery during training and makes it difficult to provide targeted collaborative guidance.
By acquiring the set of occupants, generating free space and calculating clearance values, marking bottlenecks, identifying clearance events, establishing clearance occupancy accounts, and generating collaborative guidance information, spatial risk quantification and responsibility entity association are achieved.
It accurately captures critical high-risk moments when the passage is extremely narrow without collision, clarifies the responsible party, provides visual collaborative guidance, transforms training experience into reusable data-driven processes, and improves the efficiency and safety of simulation training.
Smart Images
Figure CN121839063A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of mixed reality simulation medicine technology, and more specifically to a mixed reality simulation medicine system and method that integrates multimodal data. Background Technology
[0002] Mixed reality simulation medicine has become a key supporting technology in the surgical field for improving the standardization of surgical procedures and optimizing the efficiency of multi-role team collaboration. It is widely used in preoperative training, process simulation, and team coordination drills for complex minimally invasive surgeries. In real surgical scenarios, surgeons, assistants, and nurses need to work collaboratively within the limited operating space defined by the operating table and sterile area. Various surgical instruments and equipment need to be arranged in an orderly manner around the core operational objective, and the main instrument needs to accurately reach the surgical target point from its real-time tracked dynamic position. During this process, the natural movement of personnel's limbs, the adjustment of instrument arrangement, and the fine-tuning of equipment position may gradually compress the passage space of the main instrument, forming a critical state where the risk has significantly increased even without physical contact. This state is directly related to safety hazards such as instrument interference and tissue damage in actual surgery, and is a core issue that mixed reality simulation training needs to focus on reproducing and addressing. Current mixed reality simulation medical systems still suffer from key technological limitations, making it difficult to match the risk management needs of real surgical scenarios. Existing systems often only trigger alarms after a physical collision occurs, failing to proactively identify high-risk moments where the main instrument passageway is extremely narrow but no collision has occurred. This leads to the omission of core risk scenarios in real surgery during training. When spatial bottlenecks appear in the passageway, the system cannot accurately pinpoint the entities most directly impacting the bottleneck, making it difficult to determine responsibility for spatial risks. Furthermore, the lack of a mechanism to convert the degree of passageway narrowing into quantifiable assessment data prevents the system from providing targeted action guidance to the team based on risk status. This forces team collaboration to rely on experience-based judgment, making it difficult to distill spatial collaboration experience from training into reusable standardized procedures. This severely restricts the supporting role of simulation training in improving the safety and efficiency of actual surgery. Summary of the Invention
[0003] To address the shortcomings of existing technologies, this invention proposes a mixed reality simulation medicine method that integrates multimodal data, solving the problem that it cannot actively identify high-risk moments when the passageway for the main medical device is extremely narrow but no collision has occurred.
[0004] To achieve the above objectives, the present invention provides the following technical solution: Obtain the set of occupants, which is a set of three-dimensional geometric data of all obstacle objects in the mixed reality scene, and obtain the workspace, the position of the main instrument tip, and the mission target position; Based on the set of occupiers and the workspace, a free space for barrier-free passage within the mixed reality scene is generated. In the free space, the optimal path from the tip of the main instrument to the target location is determined by a path search algorithm, and the clearance value of the optimal path is obtained. The clearance value is the minimum distance between the optimal path and the set of occupiers. The spatial point corresponding to the net value of the optimal path is marked as the bottleneck point. Based on the set of occupied volumes and the bottleneck point, the participant closest to the bottleneck point is determined and marked as the closest participant. Calculate the change in net airspace value between adjacent time points, determine the time of the net airspace event based on the change in net airspace value, and form a list of net airspace events; A net space occupancy account is established for each participant. The net space occupancy account includes the participant identifier and the cumulative net space occupancy value. The space resource consumption value of each net space event is calculated based on the net space event list, and the space resource consumption value is added to the cumulative net space occupancy value of the corresponding participant. Collaboration guidance information and comprehensive settlement records are generated based on the comparison results between the cumulative value of net airspace occupancy and the preset risk threshold.
[0005] Furthermore, the set of occupants includes participant occupants, instrument occupants, and equipment occupants. The participant occupant is a three-dimensional geometric model of the part of the body of the person currently participating in the simulated surgery entering the operating area. The instrument occupant is a three-dimensional geometric model of the surgical instrument currently being used. The equipment occupant is a three-dimensional geometric model of the current surgical equipment. The workspace is the physically restricted area where surgical procedures can be performed; The position of the tip of the main instrument is the three-dimensional spatial coordinate of the tip of the core surgical instrument currently being operated on; The target location is the precise three-dimensional coordinate of the core operation that needs to be performed in the current surgical step.
[0006] Furthermore, free space is generated based on the set of occupied volumes and the workspace through the spatial set difference operation, specifically as follows: Remove the area covered by the occupier set from the area covered by the workspace, and mark the remaining area as free space.
[0007] Furthermore, a path search algorithm is used in free space to solve all continuous paths that start from the tip of the main instrument, end at the target point of the mission, and are completely within free space, forming a set of feasible paths; For each feasible path in the set of feasible paths, traverse all spatial points on the feasible path, calculate the Euclidean distance from each spatial point to the set of occupied volumes, and select the Euclidean distance with the smallest value as the path bottleneck clearance of the feasible path. The path bottleneck clearance is the width that the main instrument can pass through at the narrowest point of the feasible path. Compare the bottleneck clearance of all feasible paths in the feasible path set, select the maximum value, mark the maximum value as the clearance value, and mark the feasible path corresponding to the maximum value as the optimal path. The optimal path is the safest passage path for the main instrument in the current scenario. The net air value at each time point is arranged in chronological order to form a net air value time series, which includes the net air value and the corresponding time.
[0008] Furthermore, the spatial point corresponding to the net value of the optimal path is marked as the bottleneck point. The Euclidean distance from the participant's body to the bottleneck point is calculated for all participants in the simulated surgery. The participant whose body corresponds to the smallest Euclidean distance is marked as the closest participant.
[0009] Further, calculate the difference between the net air value at time k and the net air value at time k-1 in the net air value time series, and mark this difference as the net air change at time k, where k≥2; When the change in net air volume at time k is negative and the change in net air volume at time k+1 is positive, time k is determined to be the net air volume event moment. The net air volume event moment is the moment when the channel goes from continuously narrowing to starting to widen, which is the local trough moment. The clearance value, the nearest participant, and the bottleneck point corresponding to the clearance event time are combined into a clearance event. Each clearance event is assigned an event identifier. All clearance events are combined into a clearance event list according to the time order of the clearance event times. Each entry in the clearance event list includes the event identifier corresponding to a clearance event, the clearance event time, the clearance value, the nearest participant, and the bottleneck point.
[0010] Furthermore, an independent space occupancy account is established for each participant in the simulated surgery. The space occupancy account includes the participant's identifier and the cumulative space occupancy value. The initial value of the cumulative space occupancy value is set to 0, and the cumulative space occupancy value is the cumulative value of the participant's space resource consumption. For each clearance event in the clearance event list, calculate the corresponding space resource consumption value for that clearance event. The specific calculation method is as follows: Divide 1 by the clearance value corresponding to the clearance event to obtain the space resource consumption value. The space resource consumption value is the degree of space resource consumption corresponding to a single clearance event. Extract the nearest participant corresponding to the airspace event from the airspace event list, locate the airspace occupancy account corresponding to the nearest participant based on the participant identifier, and add the calculated space resource consumption value to the cumulative airspace occupancy value of the nearest participant. Whenever a new net air event occurs, the cumulative net air occupation value of the corresponding participant's net air occupation account is updated in real time.
[0011] Furthermore, obtain the cumulative net air occupancy value of all participants' net air occupancy accounts; A risk threshold is set, and collaborative guidance information is generated based on the cumulative net airspace occupancy value and the risk threshold. The specific generation process is as follows: When a participant's cumulative net airspace occupancy value is greater than or equal to the risk threshold, a red evacuation arrow is displayed in their field of vision, pointing away from the bottleneck point of the net airspace event. At the same time, a semi-transparent red overlay is superimposed on the area where the bottleneck point is located, indicating that entry into the high-risk area is prohibited. When a participant's cumulative net space occupancy value is less than the risk threshold, a green operation path is displayed in their field of vision, pointing from the current location to the task target location, and operation steps are superimposed with text prompts. The risk threshold is used to distinguish whether the cumulative net space occupied by participants has reached a critical risk state that requires the initiation of yielding guidance, based on historical surgical simulation scenarios and typical training settings.
[0012] Furthermore, the final result of the cumulative net space occupancy value of each participant and the change curve of the cumulative net space occupancy value are integrated into spatial behavior data; The completion status of the current surgical task is integrated into task completion data; The ratio of 1 to (1 + cumulative net space occupancy) will be used as the performance score for each participant. Spatial behavior data, task completion data, and performance scores are integrated into a comprehensive settlement record.
[0013] Furthermore, a mixed reality simulation medical system integrating multimodal data is proposed to realize the mixed reality simulation medical method integrating multimodal data as described above, including: The spatial risk quantification module is used to obtain the set of occupiers and the workspace, generate free space through spatial set difference operation, solve the set of feasible paths, calculate the net value, optimal path, bottleneck point and nearest participant, and complete the spatial risk quantification and association with the responsible party. The airspace event identification module is used to calculate the change in airspace between adjacent moments, determine the time of the airspace event, combine the airspace value, the nearest participant and the bottleneck point into an airspace event, and generate a list of airspace events in chronological order. The clearance occupancy accounting module creates clearance occupancy accounts for participants, calculates the space resource consumption value of a single clearance event, and adds it to the corresponding participant's cumulative clearance occupancy value, updating account data in real time. The collaborative guidance and settlement module allocates collaborative responsibilities based on the cumulative value of net space occupancy, guides participants' actions through visual prompts, and integrates spatial behavior data, task completion data, and performance scores to generate a comprehensive settlement record after the simulation ends. The motion capture module acquires the three-dimensional geometric data of the participant's body, the instrument's body, and the device's body, as well as the position of the tip of the main instrument, through optical trackers, instrument positioning sensors, etc. The visualization module displays red evacuation arrows, semi-transparent red overlays, green operation paths, and text prompts for operation steps in the MR scene, based on the collaborative guidance requirements, to provide risk warnings and visualize operation guidance.
[0014] Compared with existing technologies, it has the following advantages: This proposed mixed reality simulation medical system and method, which integrates multimodal data, obtains a set of occupants consisting of participant occupants, instrument occupants, and equipment occupants. It then generates free space by combining the workspace with spatial set difference operations. A path search algorithm is then used to solve for the set of feasible paths and determine the optimal path and clearance value. Subsequently, the clearance value changes at adjacent moments in the time series are calculated. The timing of clearance events is determined based on the positive and negative inflection of the clearance changes. This accurately captures the critical high-risk moment when the passageway for the main instrument has become extremely narrow without physical collision. This solves the problem that existing systems can only alarm after a collision and cannot reproduce the core risk scenarios of real surgery. It makes the mixed reality simulation environment more closely match the spatial risk characteristics of actual surgery and provides precise scenario support for team risk response training. The proposed mixed reality simulation medical system and method that integrates multimodal data marks the spatial points corresponding to the net space value of the optimal path as bottleneck points, calculates the Euclidean distance from the occupants of all participants to the bottleneck points to determine the closest participants, and realizes a direct link between spatial risk and the responsible party. At the same time, a net space occupancy account is established for each participant, and the spatial resource consumption value is obtained by dividing 1 by the net space value corresponding to the net space event. This value is accumulated into the cumulative net space occupancy value of the closest participants, transforming the narrowness of the passage into measurable spatial resource consumption data. This effectively solves the problems of ambiguous spatial risk responsibility and lack of quantitative assessment mechanism in existing systems, making the impact of participants' spatial occupancy objectively measurable and providing data basis for the division of responsibilities and behavior assessment in team collaboration. The proposed mixed reality simulation medical system and method, which integrates multimodal data, generates visual collaborative guidance information such as red evacuation arrows, semi-transparent red overlays, or green operation paths for participants in different states by comparing the cumulative value of space occupancy with preset risk thresholds. This clarifies the action direction and operation permissions of team members. After the simulation, the system integrates the spatial behavior data, surgical task completion data, and performance scores of each participant to form a comprehensive settlement record. This transforms the spatial collaboration experience during training into reusable data-driven process assets, solving the problems of existing systems lacking targeted collaborative guidance and difficulty in accumulating experience. It also reduces the cost of repeated team adjustments and improves the efficiency of simulation training and the safety of subsequent surgical collaboration. Attached Figure Description
[0015] Figure 1 This is a schematic diagram of the method flow of the present invention; Figure 2 This is a schematic diagram of the system framework of the present invention. Detailed Implementation
[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0017] Please see Figure 1 This application provides a mixed reality simulation medical method that integrates multimodal data; The method specifically includes the following steps: Step 1: Obtain the current set of occupants. The set of occupants is a collection of 3D geometric data of all obstacle objects in the MR (Mixed Reality) scene at the current moment, including: participant occupants, instrument occupants, and equipment occupants. The participant occupant is the 3D geometric model of the part of the body of the person currently participating in the surgery (e.g., surgeon, assistant) that enters the operating area (e.g., forearm, hands). The instrument occupant is the 3D geometric model of the surgical instrument currently being used (e.g., needle forceps, energy knife). The equipment occupant is the 3D geometric model of the current surgical equipment (e.g., microscope arm, robotic arm). The set of occupants can be obtained through the motion capture module (e.g., optical tracker), instrument positioning sensor, or depth camera of the MR system. The workspace is the physically restricted area where surgical operations can be performed (e.g., the three-dimensional range of the operating table surface). It consists of fixed spatial parameters and is obtained through the scene configuration information of the surgical task script. The free space is generated by the spatial set difference operation based on the occupant set and the workspace. Specifically, the area covered by the occupant set at the current moment is removed from the area covered by the workspace, and the remaining area is the free space at the current moment. The free space at the current moment is the barrier-free passage area in the MR scene at the current moment. It is the spatial range in which the main instrument can safely pass. It provides strict spatial boundary constraints for solving subsequent feasible paths, ensuring that the solved path conforms to the physical constraints of the actual surgical scene. The current position of the tip of the main instrument is obtained. The current position of the tip of the main instrument is the three-dimensional spatial coordinate of the tip of the core surgical instrument currently being operated, which is obtained through the motion capture module of the MR system. The target location of the task is obtained. The target location of the task is the precise three-dimensional coordinates of the core operation to be performed in the current surgical step (such as the suture target point or the blood vessel exposure point). It is obtained through the preset information of the surgical task script or the manual marking by the physician in the MR scene. In the current free space, a path search algorithm is used, such as voxel graph search or probabilistic landmark method, to solve for all continuous paths that start from the tip of the main instrument and end at the target site of the mission, and are completely in free space, forming a set of feasible paths. Specifically, all main instrument passage paths that meet the spatial constraints are obtained to provide sufficient path samples for subsequent net clearance value calculation, ensuring that the net clearance value can reflect the risk status of the optimal path. This is existing technology, so it will not be elaborated here. For each feasible path in the set of feasible paths at the current moment, traverse all spatial points on the feasible path, calculate the Euclidean distance from each spatial point to the set of occupied volumes at the current moment, and select the Euclidean distance with the smallest value as the path bottleneck clearance of the feasible path. The path bottleneck clearance is the width that the main instrument can pass through at the narrowest point of the feasible path. Compare the bottleneck clearance of all feasible paths in the current feasible path set, select the maximum value, mark the maximum value as the clearance value, and mark the feasible path corresponding to the maximum value as the optimal path. The optimal path is the safest passage path for the main instrument in the current scenario. The net clearance values at various times are arranged in chronological order to form a net clearance value time series. The net clearance value time series includes the net clearance value and the corresponding time. The net clearance value time series reflects the change of risk in the main equipment passage over time. Specifically, the feasible path set is the collection of all paths that the main vehicle can traverse without collision in the current MR scene. The passage capacity of each feasible path is determined by the space margin at its narrowest point. The path bottleneck clearance is the position on the feasible path closest to an obstacle, and its value represents the width allowed for the main vehicle to pass through at the narrowest point of that feasible path. The passage restriction of a feasible path is determined by the path bottleneck clearance, i.e., the space margin at its narrowest point. The path bottleneck clearance of all paths in the feasible path set is compared, and the maximum value is selected because the path corresponding to this maximum value has the widest passage width at its narrowest point among all feasible paths. The main vehicle has the largest space margin and the lowest collision risk when traveling along this path, therefore, this path is marked as such. The optimal path is the safest passage path for the main instrument in the current scenario. The maximum value is marked as the clearance value, which is the narrowest passage width of the safest path that the main instrument can choose in the current scenario. The clearance value can quantify the spatial risk status of the main instrument's passage. Its value directly reflects the space margin for the main instrument's passage. The smaller the value, the higher the collision risk. The optimal path can clearly define the safe passage choice for the main instrument in the current scenario. The optimal path and the clearance value together constitute the core foundation of the spatial risk quantitative management in the MR scenario of this solution. It provides the necessary quantitative indicators and spatial objects for subsequent identification of critical dangerous moments in the channel and location of the responsible parties corresponding to the risks. It is a prerequisite for realizing team collaboration spatial risk control and process orchestration. Mark the spatial point corresponding to the path bottleneck clearance of the optimal path at the current moment as the bottleneck point. Calculate the Euclidean distance from the participant's body to the bottleneck point for all participants in the simulated surgery. Mark the participant whose body corresponds to the smallest Euclidean distance as the closest participant at the current moment. Specifically, the bottleneck point is the spatial point corresponding to the clearance value of the optimal path at the current moment. The safety of passage of the optimal path is determined by the spatial margin at its narrowest point, which is the position with the smallest distance between the path and the set of occupants. Therefore, this position is marked as the bottleneck point. By locating the bottleneck point at the current moment, the physical location with the most concentrated risk is identified. The participants' occupants are the main entities that constitute spatial obstacles. Their physical distance from the bottleneck point directly reflects the degree of influence on the core risk location. By calculating the Euclidean distance from each participant's occupant to the bottleneck point, the entity that has the most direct impact on the bottleneck space occupation can be determined. This realizes the association between spatial risk and specific participating entities and completes the precise division of risk responsibility. The bottleneck point and the closest participants together provide specific risk objects and responsible entities for the spatial risk management of team collaboration in the MR scenario of this solution. This is a key basis for subsequent spatial resource scheduling and team action choreography.
[0018] Step 2: Calculate the difference between the net clearance value at time k and time (k-1) in the net clearance value time series. Mark this difference as the net clearance change at time k, where k ≥ 2. Specifically, the net clearance change is the difference between the net clearance values at adjacent times. It is used to quantify the contraction or expansion trend of the main instrument passage. When the net clearance change is negative, it means that the passage at the current time is in a contraction state compared to the previous time. When the net clearance change is positive, it means that the passage at the current time is in an expansion state compared to the previous time. The static value of the net clearance value is transformed into a dynamic trend through the net clearance change, providing a quantifiable trend basis for subsequent identification of the critical danger moment of the passage. When the change in net clearance at time k is negative and the change in net clearance at time k+1 is positive, time k is determined to be the net clearance event moment. The net clearance event moment is the local trough moment when the channel goes from continuously narrowing to starting to widen. It is the critical risk moment when the passage from the main instrument to the mission target site is the narrowest and the riskiest. Specifically, determining the moment of the clearance event can solve the problem of the difficulty in accurately capturing the risk of a passage being in a critically narrow state without physical collision in a mixed reality simulated medical scenario. This critically narrow state has a short duration and strong dynamic changes, making it impossible to reliably identify through manual observation. This state is precisely the core cause of the risk of interference with the passage of the main instrument. Therefore, it is necessary to determine the moment of the clearance event to transform this critical risk state into a recordable and manageable discrete event unit, which can then quantify the key time node of spatial risk. It should be noted that the sign of the change in clearance directly reflects the state of contraction or expansion of the passage. When the change in clearance at the k-th moment is negative, it means that the passage is in a state of continuous contraction and the clearance value is continuously decreasing. When the change in clearance at the (k+1)-th moment is positive, it means that the passage begins to expand and the clearance value is continuously increasing. The turning point between the two, that is, the k-th moment, corresponds exactly to the local minimum point where the clearance value changes from decreasing to increasing, and also corresponds to the narrowest point of the passage. Therefore, by combining the signs of the changing trends, the critical risk moment when the passage is narrowest can be accurately located. The clearance value, the nearest participant, and the bottleneck point corresponding to the clearance event are combined into a clearance event. Each clearance event has an event identifier. All clearance events are combined into a clearance event list according to the time sequence of the clearance event times. Each entry in the clearance event list includes the event identifier corresponding to a clearance event, the clearance event time, the clearance value, the nearest participant, and the bottleneck point. Specifically, by associating discrete clearance events with corresponding risk values, responsible parties, and spatial locations, a structured event record is formed, providing a complete basis for the subsequent division of responsibilities and accounting of spatial resources.
[0019] Step 3: Establish an independent net space occupancy account for each participant in the simulated surgery. The net space occupancy account includes the participant's identifier and the cumulative net space occupancy value. Set the initial value of the cumulative net space occupancy value to 0. The cumulative net space occupancy value is used to quantify the degree of space resource consumption of the participant in the dangerous scenario. It is the cumulative value of the participant's space resource consumption. The larger the value, the more significant the impact of the participant's space occupancy in the critical channel state. Specifically, establish an independent resource consumption measurement account for each participant to realize the correlation between the responsible party for space risk and the degree of consumption, and provide a basic carrier for subsequent quantitative accounting. For each clearance event in the clearance event list, calculate the space resource consumption value corresponding to that clearance event. The specific calculation method is as follows: divide 1 by the clearance value corresponding to the clearance event to obtain the space resource consumption value, that is, 1 ÷ clearance value = space resource consumption value. The space resource consumption value is the degree of space resource consumption corresponding to a single clearance event. Since the smaller the clearance value, the narrower the main equipment passage and the higher the risk of space occupation, the value of the degree of space resource consumption is negatively correlated with the clearance value. That is, the smaller the clearance value of the clearance event, the greater the space resource consumption value. Extract the nearest participant corresponding to the airspace event from the airspace event list, locate the airspace occupancy account corresponding to the nearest participant based on the participant identifier, and add the calculated space resource consumption value to the cumulative airspace occupancy value of the nearest participant to complete one accounting operation. During the MR simulation, whenever a new net air event occurs, the accounting operation is immediately repeated to update the cumulative net air occupation value of the corresponding participant's net air occupation account in real time. Specifically, spatial resource consumption value is a quantitative indicator of the degree of spatial risk in a clearance event. The thinner the space margin between the main instrument and surrounding occupiers, the higher the risk and impact of space occupation. Therefore, by converting the event clearance value into spatial resource consumption value, the degree of impact of the responsible party on the surgical operation space in a single clearance event is quantified. The cumulative clearance occupation value is a dedicated spatial resource consumption account established for each participant, used to calculate their cumulative space occupation impact in all clearance events. Since the same participant may act as the responsible party and have a space occupation impact in multiple clearance events, the spatial resource consumption value corresponding to each event is added to the participant's cumulative clearance occupation value to achieve continuous measurement of their space occupation impact, ensuring that the account data is consistent with the participant's actual spatial behavior. The core of the accounting operation is to transform geometric spatial risk into manageable and schedulable resource consumption data. Traditional surgical simulation only focuses on alarms after a collision occurs in the model, but in reality, no collision occurs. The core risk factor is the critically narrow passageway, a state that cannot be covered by simple collision detection. However, by using accounting, discrete critical space risks can be transformed into structured resource consumption records, providing a quantitative basis for team role scheduling and positioning adjustments in subsequent MR scenarios. At the same time, it enables an objective assessment of participants' spatial behavior. Since surgical operation space is a limited and scarce resource, main instruments, participant limbs, and equipment arms all need to occupy this space to complete the operation. The space occupation behavior of different entities will affect each other, causing the degree of passageway narrowing to change dynamically. The impact of space occupation on operational feasibility has the characteristics of resource scarcity and consumption. Therefore, it can be abstracted as a measurable spatial resource. The spatial resource consumption value corresponds to the cost of occupying this scarce resource by the responsible entity. The larger the value, the higher the degree of space resource crowding out by the occupation behavior. Through quantitative calculation, the mapping of spatial risk to resource consumption is realized. Finally, through accounting operations, the digital management and mapping of interactive behavior in simulated surgical space is realized.
[0020] Step 4: Obtain the cumulative net space occupancy value of all participants' accounts. If a participant's cumulative net space occupancy value is larger, it means that the frequency of its occupation in the dangerous passage scenario is higher and the degree of space resource consumption is heavier. It will need to take on the responsibility of actively giving way and withdrawing from the bottleneck area. If a participant's cumulative net space occupancy value is smaller, it means that its space resource consumption cost is lower. It will be given priority to obtain the collaborative permission to approach the passage and perform core operations. Set a risk threshold. When a participant’s cumulative net air space occupation value is greater than or equal to the risk threshold, a red evacuation arrow will be displayed in their field of vision, pointing away from the bottleneck point of the net air space event. At the same time, a semi-transparent red mask will be superimposed on the area where the bottleneck point is located to indicate that entry into the high-risk area is prohibited. When a participant's cumulative net space occupancy value is less than the risk threshold, a green operation path is displayed in their field of vision, pointing from the current position to the task target site, and operation step text prompts are superimposed, such as prompting the next step to perform suturing operation; Specifically, the risk threshold is used to distinguish whether the cumulative space occupancy value of participants has reached a continuous critical risk state requiring the initiation of yielding guidance. Based on the space resource consumption value corresponding to the risk level of the main instrument passage in the surgical simulation scenario, and combined with the frequency of space occupancy events during typical training, the risk threshold is set with the core criterion that when a participant's cumulative space resource consumption value reaches this threshold, their space occupancy behavior has created a continuous critical risk to the main instrument passage. For example, when the main instrument passage is in a high-risk state, assuming a clearance width of 3mm, the single space resource consumption value is approximately 0.33, indicating a congested state; assuming a clearance width of 8mm, the single consumption value is approximately 0.125, indicating a safe state; assuming a clearance width of 15mm, the single consumption value is approximately 0.067. In training, the frequency of space occupancy events is typically 5 to 12 times. The risk threshold range for calculating consumption value needs to cover the cumulative results of multiple high-risk events or numerous congestion events. In this example, the risk threshold is set to 1.5, which means that when the cumulative value of clearance occupancy reaches 1.5, the space occupancy behavior of the corresponding participant has met the judgment criteria for continuous critical risk. Specifically, this corresponds to about 4 to 5 high-risk clearance events (3mm clearance width, single consumption 0.33), or about 12 congestion clearance events (8mm clearance width, single consumption 0.125). At this time, their space occupancy has continuously affected the passage of the main instrument, and it is necessary to initiate yielding or evacuation guidance to effectively reduce the channel risk, which meets the collaborative safety requirements of surgical simulation training. The judgment result is then directly converted into intuitive MR visualization prompts to guide team members to adjust their actions and positions in real time, reduce the space risk of the main instrument passage, and improve the accuracy and efficiency of surgical collaboration. After this round of MR simulation training, the final result of the cumulative net space occupancy value of each participant and the change curve of the cumulative net space occupancy value are integrated into spatial behavior data. The completion degree of the current surgical task, such as whether all key operation steps are completed, and the number of occurrences and handling effects of net space events during the training process are integrated into task completion data. The ratio of 1 to (1 + cumulative net space occupancy value) is used as the performance score of each participant. The performance score ranges from 0 to 1. The smaller the cumulative net space occupancy value, the closer the performance score is to 1, which means that the participant's spatial behavior is more reasonable and the performance is better. Specifically, the performance score is a quantitative evaluation index generated based on the participant's spatial resource consumption degree. It is used to objectively evaluate the participant's spatial behavior performance in MR simulation training. By generating a comprehensive settlement record that includes spatial behavior data, task completion data, and performance score, a full-dimensional objective review of the training process is achieved, providing data basis for the optimization of subsequent training programs. In the next MR simulation of the same surgical scenario, the initial role assignment can be completed based on the obtained spatial behavior data, task completion data and performance scores, and a special training collaboration plan can be formulated and continuously iterated and optimized. This transforms the collaborative experience of MR simulation training into reusable digital process assets, avoids repeated adjustments, and improves the efficiency and relevance of subsequent MR simulation training.
[0021] Furthermore, refer to Figure 2 As shown, a mixed reality simulation medical system integrating multimodal data is proposed to implement the mixed reality simulation medical method integrating multimodal data as described above, including: The spatial risk quantification module is used to obtain the set of occupiers and the workspace, generate free space through spatial set difference operation, solve the set of feasible paths, calculate the net value, optimal path, bottleneck point and nearest participant, and complete the spatial risk quantification and association with the responsible party. The airspace event identification module is used to calculate the change in airspace between adjacent moments, determine the time of the airspace event, combine the airspace value, the nearest participant and the bottleneck point into an airspace event, and generate a list of airspace events in chronological order. The clearance occupancy accounting module creates clearance occupancy accounts for participants, calculates the space resource consumption value of a single clearance event, and adds it to the corresponding participant's cumulative clearance occupancy value, updating account data in real time. The collaborative guidance and settlement module allocates collaborative responsibilities based on the cumulative value of net space occupancy, guides participants' actions through visual prompts, and integrates spatial behavior data, task completion data, and performance scores to generate a comprehensive settlement record after the simulation ends. The motion capture module acquires the three-dimensional geometric data of the participant's body, the instrument's body, and the device's body, as well as the position of the tip of the main instrument, through optical trackers, instrument positioning sensors, etc. The visualization module displays red evacuation arrows, semi-transparent red overlays, green operation paths, and text prompts for operation steps in the MR scene, based on the collaborative guidance requirements, to provide risk warnings and visualize operation guidance.
[0022] The above embodiments are only used to illustrate the technical methods of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical methods of the present invention without departing from the spirit and scope of the technical methods of the present invention.
Claims
1. A mixed reality simulation medical method fusing multi-modal data, characterized in that, include: Obtain the set of occupants, which is a set of three-dimensional geometric data of all obstacle objects in the mixed reality scene, and obtain the workspace, the position of the main instrument tip, and the mission target position; Based on the set of occupiers and the workspace, a free space for barrier-free passage within the mixed reality scene is generated. In the free space, the optimal path from the tip of the main instrument to the target location is determined by a path search algorithm, and the clearance value of the optimal path is obtained. The clearance value is the minimum distance between the optimal path and the set of occupiers. The spatial point corresponding to the net value of the optimal path is marked as the bottleneck point. Based on the set of occupied volumes and the bottleneck point, the participant closest to the bottleneck point is determined and marked as the closest participant. Calculate the change in net airspace value between adjacent time points, determine the time of the net airspace event based on the change in net airspace value, and form a list of net airspace events; A net space occupancy account is established for each participant. The net space occupancy account includes the participant identifier and the cumulative net space occupancy value. The space resource consumption value of each net space event is calculated based on the net space event list, and the space resource consumption value is added to the cumulative net space occupancy value of the corresponding participant. Collaboration guidance information and comprehensive settlement records are generated based on the comparison results between the cumulative value of net airspace occupancy and the preset risk threshold.
2. The mixed reality simulated medical method fusing multi-modal data according to claim 1, wherein, include: The set of occupants includes participant occupants, instrument occupants, and equipment occupants. The participant occupant is the three-dimensional geometric model of the part of the body of the person currently participating in the simulated surgery entering the operating area. The instrument occupant is the three-dimensional geometric model of the surgical instrument currently being used. The equipment occupant is the three-dimensional geometric model of the surgical equipment currently being used. The workspace is the physically restricted area where surgical procedures can be performed; The position of the tip of the main instrument is the three-dimensional spatial coordinate of the tip of the core surgical instrument currently being operated on; The target location is the precise three-dimensional coordinate of the core operation that needs to be performed in the current surgical step.
3. The mixed reality simulation medical method for fusing multimodal data according to claim 2, characterized in that, include: Free space is generated based on the set of occupied volumes and the workspace through spatial set difference operations. The specific generation method is as follows: Remove the area covered by the occupier set from the area covered by the workspace, and mark the remaining area as free space.
4. The mixed reality simulation medicine method for fusing multimodal data according to claim 2, characterized in that, include: In free space, a path search algorithm is used to solve all continuous paths that start from the tip of the main instrument and end at the target point of the mission, and are completely within free space, forming a set of feasible paths. For each feasible path in the set of feasible paths, traverse all spatial points on the feasible path, calculate the Euclidean distance from each spatial point to the set of occupied volumes, and select the Euclidean distance with the smallest value as the path bottleneck clearance of the feasible path. The path bottleneck clearance is the width that the main instrument can pass through at the narrowest point of the feasible path. Compare the bottleneck clearance of all feasible paths in the feasible path set, select the maximum value, mark the maximum value as the clearance value, and mark the feasible path corresponding to the maximum value as the optimal path. The optimal path is the safest passage path for the main instrument in the current scenario. The net air value at each time point is arranged in chronological order to form a net air value time series, which includes the net air value and the corresponding time.
5. The mixed reality simulation medical method for fusing multimodal data according to claim 4, characterized in that, include: Mark the spatial point corresponding to the net value of the optimal path as the bottleneck point. Calculate the Euclidean distance from the participant's body to the bottleneck point for all participants in the simulated surgery. Mark the participant whose body corresponds to the smallest Euclidean distance as the closest participant.
6. The mixed reality simulation medicine method for fusing multimodal data according to claim 5, characterized in that, include: Calculate the difference between the net air value at time k and the net air value at time k-1 in the net air value time series, and mark this difference as the net air change at time k, where k≥2; When the change in net air volume at time k is negative and the change in net air volume at time k+1 is positive, time k is determined to be the net air volume event moment. The net air volume event moment is the moment when the channel goes from continuously narrowing to starting to widen, which is the local trough moment. The clearance value, the nearest participant, and the bottleneck point corresponding to the clearance event time are combined into a clearance event. Each clearance event is assigned an event identifier. All clearance events are combined into a clearance event list according to the time order of the clearance event times. Each entry in the clearance event list includes the event identifier corresponding to a clearance event, the clearance event time, the clearance value, the nearest participant, and the bottleneck point.
7. The mixed reality simulation medical method for fusing multimodal data according to claim 6, characterized in that, include: Create an independent space occupancy account for each participant in the simulated surgery. The space occupancy account includes the participant's identifier and the cumulative space occupancy value. Set the initial value of the cumulative space occupancy value to 0. The cumulative space occupancy value is the cumulative value of the participant's space resource consumption. For each clearance event in the clearance event list, calculate the corresponding space resource consumption value for that clearance event. The specific calculation method is as follows: Divide 1 by the clearance value corresponding to the clearance event to obtain the space resource consumption value. The space resource consumption value is the degree of space resource consumption corresponding to a single clearance event. Extract the nearest participant corresponding to the airspace event from the airspace event list, locate the airspace occupancy account corresponding to the nearest participant based on the participant identifier, and add the calculated space resource consumption value to the cumulative airspace occupancy value of the nearest participant. Whenever a new net air event occurs, the cumulative net air occupation value of the corresponding participant's net air occupation account is updated in real time.
8. The mixed reality simulation medical method for fusing multimodal data according to claim 7, characterized in that, include: Obtain the cumulative net air usage value of all participants' accounts; A risk threshold is set, and collaborative guidance information is generated based on the cumulative net airspace occupancy value and the risk threshold. The specific generation process is as follows: When a participant's cumulative net airspace occupancy value is greater than or equal to the risk threshold, a red evacuation arrow is displayed in their field of vision, pointing away from the bottleneck point of the net airspace event. At the same time, a semi-transparent red overlay is superimposed on the area where the bottleneck point is located, indicating that entry into the high-risk area is prohibited. When a participant's cumulative net space occupancy value is less than the risk threshold, a green operation path is displayed in their field of vision, pointing from the current location to the task target location, and operation steps are superimposed with text prompts. The risk threshold is used to distinguish whether the cumulative net space occupied by participants has reached a critical risk state that requires the initiation of yielding guidance, based on historical surgical simulation scenarios and typical training settings.
9. The mixed reality simulation medical method for fusing multimodal data according to claim 8, characterized in that, include: The final result of the cumulative net space occupancy value of each participant and the change curve of the cumulative net space occupancy value are integrated into spatial behavior data; The completion status of the current surgical task is integrated into task completion data; The ratio of 1 to (1 + cumulative net space occupancy) will be used as the performance score for each participant. Spatial behavior data, task completion data, and performance scores are integrated into a comprehensive settlement record.
10. A mixed reality simulation medical system that integrates multimodal data, used to implement the mixed reality simulation medical method that integrates multimodal data as described in any one of claims 1-9, characterized in that, include: The spatial risk quantification module is used to obtain the set of occupiers and the workspace, generate free space through spatial set difference operation, solve the set of feasible paths, calculate the net value, optimal path, bottleneck point and nearest participant, and complete the spatial risk quantification and association with the responsible party. The airspace event identification module is used to calculate the change in airspace between adjacent moments, determine the time of the airspace event, combine the airspace value, the nearest participant and the bottleneck point into an airspace event, and generate a list of airspace events in chronological order. The clearance occupancy accounting module creates clearance occupancy accounts for participants, calculates the space resource consumption value of a single clearance event, and adds it to the corresponding participant's cumulative clearance occupancy value, updating account data in real time. The collaborative guidance and settlement module allocates collaborative responsibilities based on the cumulative value of net space occupancy, guides participants' actions through visual prompts, and integrates spatial behavior data, task completion data, and performance scores to generate a comprehensive settlement record after the simulation ends. The motion capture module acquires the three-dimensional geometric data of the participant's body, the instrument's body, and the device's body, as well as the position of the tip of the main instrument, through optical trackers, instrument positioning sensors, etc. The visualization module displays red evacuation arrows, semi-transparent red overlays, green operation paths, and text prompts for operation steps in the MR scene, based on the collaborative guidance requirements, to provide risk warnings and visualize operation guidance.