Subarachnoid space puncture training system based on augmented reality

Abstract

The invention relates to the technical field of augmented reality medical training, in particular to a subarachnoid space puncture training system based on augmented reality, which comprises a scene loading module, a process control module, a visualization processing module, a synchronous explanation module and an interaction control module. The system loads a three-dimensional anatomical or panoramic video training scene according to the selection of a user, and starts a four-stage progressive training process. And carrying out transparency gradient processing on the three-dimensional human body model through material rendering control, and dynamically generating an anatomical structure visualization effect from the skin to the deep tissue. And the synchronous explanation module activates the dynamic annotation system, displays the corresponding anatomical annotation according to the training stage, and synchronously plays voice explanation and subtitles based on the time axis. The interaction control module responds to the user operation instruction and controls the playing state of the training process. According to the system, progressive visualization of an anatomical structure and accurate and synchronous explanation of training content are realized, and the immersion, coherence and teaching effect of training are improved.

Description

Technical Field

[0001] This invention relates to the field of extended reality medical training technology, and in particular to an extended reality-based subarachnoid puncture training system. Background Technology

[0002] In clinical medical education, subarachnoid puncture, a commonly used invasive procedure, demands extremely high levels of anatomical knowledge, spatial awareness, and operational skills from physicians. Traditional training primarily relies on theoretical lectures, two-dimensional atlases, and physical models, making it difficult for trainees to develop an intuitive understanding of the complex, three-dimensional anatomical layers within the spinal canal. With technological advancements, simulation training systems based on virtual reality or augmented reality have been introduced. These systems reproduce progressively layered anatomical structures using three-dimensional models and allow trainees to practice the procedures virtually.

[0003] Existing extended reality training systems have limitations in their core experience. Regarding 3D visualization, most systems can only present static anatomical models or display different layers by instantly switching between different models. This discrete and discontinuous visualization method fragments the complete path perception from the skin surface to the target cavity, failing to simulate the dynamic changes in tissue layers during the gradual insertion of the puncture needle. This results in a disconnect between the visual feedback received by trainees and the realistic tactile experience, leading to insufficient training in spatial layer perception. In terms of guiding training content, a common practice is to simply combine pre-recorded audio or video explanations with operable 3D scenes. Due to the lack of a sophisticated synchronization mechanism, the voice explanations, text prompts, and the scene content and operational steps currently seen by the trainee are often out of sync. The annotations of anatomical structures are also mostly static overlays, unable to be dynamically updated with the training phase, causing a separation between theoretical learning and practical operation, resulting in a rigid training process, weak interactivity, and hindering the efficiency of skill and knowledge integration and internalization. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and propose a training system for subarachnoid puncture based on extended reality.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: a training system for subarachnoid puncture based on extended reality, comprising: The scene loading module loads the corresponding 3D anatomy training scene or panoramic video training scene according to the user's selection command in the waiting scene. It adopts an asynchronous loading method, displays a loading indicator and disables user interaction during scene switching, and generates an initialized training scene. The process control module, based on the initialized training scenario, initiates a four-stage progressive training process, coordinating various functional modules through an event mechanism and asynchronous task chain to generate an ordered sequence of training content. The visualization processing module uses the ordered sequence of training content to control the material rendering control module to perform transparency gradient processing on the three-dimensional human body model, generating a visualization effect of the anatomical structure from the skin to the deep tissues. The synchronous explanation module, based on the visualization effect of the anatomical structure, activates the dynamic annotation system, displays the corresponding anatomical annotation groups in sequence according to the current training stage, and synchronously plays the audio explanation and subtitles through the multimedia synchronous playback mechanism based on the time axis, generating an interactive training screen with synchronous explanation; The interactive control module responds to user input commands via the extended reality interactive control module, controlling the playback status of the training process, enabling the training process to be paused, resumed, or returned to a waiting scenario, and generating a user-led training process.

[0006] Preferably, based on the initialized training scenario, a four-stage progressive training process is initiated. This process coordinates various functional modules through an event mechanism and asynchronous task chains to generate an ordered sequence of training content, including: During the overall anatomical structure recognition stage, the material rendering control module smoothly transitions the human body model from an opaque state to a semi-transparent state, generating an overall anatomical view that allows observation of the bones and tissue structures beneath the skin. In the detailed analysis of the skeletal structure, the scene animation module controls the spinal skeleton model to fly into the center of the scene with a smooth displacement animation with scaling effect and automatically rotate to the best viewing angle, generating a key display of the skeletal structure. During the puncture path and tissue layer display stage, the complete process of the puncture needle entering the body layer by layer is simulated. When the puncture needle penetrates each layer of tissue, the animation is automatically paused and the current tissue layer is marked, generating a puncture path demonstration of penetrating layer by layer. During the neural structure and injection demonstration phase, the neural structure inside the subdural space is displayed through a perspective effect, and particle effects are activated to simulate the injection process and fluid flow animation, generating a complete demonstration of drug diffusion and cerebrospinal fluid outflow. Based on the overall anatomical view, the key images of bony structures, the demonstration of the puncture path penetrating layer by layer, and the complete demonstration of drug diffusion and cerebrospinal fluid outflow, the training content is combined into a complete sequence according to a preset order.

[0007] Preferably, in the puncture path and tissue layer display stage, the complete process of the puncture needle entering the body layer by layer from the surface is simulated. When the puncture needle penetrates each layer of tissue, the animation is automatically paused and the current tissue layer is marked, generating a demonstration of the puncture path that penetrates layer by layer, including: The control module for puncture animation playback starts running, causing the virtual puncture needle to move from the starting point of the skin layer; The spatial relationship between the tip of the puncture needle and each tissue layer is determined in real time using a tissue layer detection algorithm. When a penetration event is detected, a pause command is sent to the animation control module. According to the pause command, the playback of the puncture animation is paused, and the dynamic annotation system is invoked to display the name of the currently penetrated tissue layer. Activate the voice explanation module to play the anatomical features and key operational points related to the current tissue level; After the audio explanation ends, a continuation command is received, the puncture animation resumes playback to the next tissue layer, and the process is repeated until all preset tissue layers are penetrated, generating a puncture path demonstration that includes multiple pauses in the explanation.

[0008] Preferably, based on the visualization effect of the anatomical structure, the dynamic annotation system is activated, and the corresponding anatomical annotation groups are displayed sequentially according to the current training stage. Audio explanations and subtitles are played synchronously through a time-axis-based multimedia synchronous playback mechanism, generating an interactive training screen with synchronized explanations, including: Based on the current training stage, the corresponding anatomical annotation group is called from the annotation group management module. The anatomical annotation group includes the overall structure group, the bony structure group, the puncture path group, and the neural structure group. By using a dynamic connection annotation mechanism based on anchor point pairing, a visual connection is established between the anatomical structure anchor points on the 3D model and the annotation text anchor points on the user interface. The display and hiding of the visual connection is controlled by gradient animation effects, including transparency gradients and line gradual visibility. The time-axis-based multimedia synchronous playback mechanism is launched simultaneously, matching the subtitle content corresponding to the current playback time according to the structured stored subtitle data, and ensuring that audio playback and subtitle display remain synchronized. The annotation display, line-connecting animation, voice explanation and subtitle content are integrated into the current scene rendering screen to generate a multimedia synchronized interactive training screen.

[0009] Preferably, a dynamic connection annotation mechanism based on anchor point pairing is used to establish visual connections between anatomical structure anchor points on the 3D model and annotation text anchor points on the user interface, including: Set anatomical structure anchor points on the preset anatomical landmarks of the 3D human body model, and set corresponding annotation text anchor points on the user interface canvas; During each frame rendering process, the projection coordinates of the anatomical structure anchor points in screen space are calculated; Based on the projection coordinates and the screen coordinates of the labeled text anchor points, dynamically generate a Bézier curve or straight line connecting the two points as a visual connection. The vertex data of the visualized connection is updated in real time by the graphics renderer to ensure that the connection is updated correctly as the model moves or the view changes. Based on the user's interaction with the interface, the display status of the visual connection is controlled, including always visible, hidden, or only visible during specific training phases.

[0010] Preferably, in response to user input commands via the extended reality interactive control module, the playback status of the training process is controlled, enabling pause, resumption, or return to a waiting scenario, generating a user-led training process, including: A unified controller input detection module is encapsulated to continuously monitor controller button events of head-mounted virtual reality display devices; When the B button on the controller is pressed, the playback status of the current training content is switched. If the current playback status is playing, the content is paused and the control interface is brought up. If the current playback status is paused, the content is resumed. When a button A on the controller is pressed, the current context is determined. If the context is during training, it is treated as a confirmation operation; if the context is on the control interface, it is treated as a command to return to the waiting scenario. In the panoramic video training mode, the viewing angle of the virtual camera is controlled by the input value of the right joystick, the position is offset along the local Z-axis of the virtual camera, and the offset is limited to a set range. Based on the results of the operation instructions, update the internal state machine of the training scenario and adjust the presentation of the scenario content accordingly.

[0011] Preferably, it also includes a panoramic video display module to provide an immersive panoramic medical operation video viewing experience, including: In the waiting scenario, in response to the user's instruction to enter the panoramic video training mode, the panoramic video resources are loaded and the playback controller is initialized. The video playback module is responsible for decoding and controlling the playback of panoramic videos, supporting play, pause, and jump operations, and driving subtitle matching and audio synchronization; Respond to application foreground / background switching events, automatically pause video playback when the application loses focus, and automatically resume playback from the pause point when the application regains focus; The playback state machine manages five states: not ready, waiting to play, playing, user paused, and system paused, ensuring the correctness of the state switching logic. After the video finishes playing, a scene switch is automatically triggered, returning to the waiting scene. Users can also manually return to the waiting scene during playback using the A button on the controller.

[0012] Preferably, the video playback module is responsible for decoding and controlling the playback of the panoramic video, supporting play, pause, and jump operations, and driving subtitle matching and audio synchronization, including: Initialize the video decoder and load the specified panoramic video file into the memory buffer; Create a separate rendering thread to render the decoded video frames onto a 360° spherical projection model; Provides a playback control interface, including start playback, pause playback, resume playback, and stop playback; During playback, the current playback timestamp is obtained in real time, and the subtitle database is queried to match the subtitle content that should be displayed; The playback progress of the audio player is adjusted synchronously to ensure that the audio, video, and subtitle content are synchronized in time. It processes user commands to jump to a specific point in time and continues playback.

[0013] Preferably, the system further includes a material rendering control module for achieving smooth control of the transparency of the 3D model, including: Maintain two states for the human body model: opaque material instances and transparent material instances. Switching between two material states is done asynchronously, updating the material's transparency attribute value frame by frame during the switching process; By using shader programming, the transparency value of each frame is calculated through linear interpolation to achieve a smooth transition effect from opaque to semi-transparent. Based on the needs of the current training phase, the transparency parameters of different anatomical structures are dynamically adjusted to highlight specific tissue layers; Receive instructions from the training process control module, trigger material switching operations at specific time points, and ensure that the transition animation is synchronized with the voice narration.

[0014] Preferably, the switching between the two material states is performed asynchronously, updating the material's transparency attribute value frame by frame during the switching process, including: Define an asynchronous task for material switching, wherein the asynchronous task receives a starting transparency value, a target transparency value, and a transition duration as parameters; In each frame rendering loop, calculate the ratio of the time elapsed since the start of the task to the total duration; Based on the ratio, the transparency value that should be set for the current frame is calculated using a linear interpolation algorithm; The calculated transparency value is assigned to the corresponding shader property of the 3D model material; After the transition duration ends, ensure that the material transparency accurately reaches the target transparency value and trigger the transition completion event; During the transition, ensure that other training content, such as animation playback and audio narration, is coordinated with this change in transparency.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are as follows: The material rendering control module performs transparency gradient processing on the 3D human body model. This technology achieves a smooth and gradual manifestation of anatomical structures from the epidermis to the subarachnoid space by adjusting the optical properties of the model's materials in real time and continuously. It simulates the visual changes in tissues as the puncture needle penetrates layer by layer, presenting structures that were originally hidden deep within in an intuitive and dynamic way. This allows the operator to clearly perceive the relative positions and spatial relationships of different tissues such as skin, subcutaneous tissue, ligaments, and dura mater, establishing a continuous spatial path cognition. The continuity of visual feedback enhances the immersion and realism of the training, providing a visual experience close to that of a real puncture in virtual operations, effectively training the trainee's depth perception and needle tip position judgment ability.

[0016] A timeline-based multimedia synchronous playback mechanism works in conjunction with a dynamic annotation system. This system precisely binds audio narration, subtitle information, and every step of the training process, ensuring that the narration corresponds in real-time to the current visual focus and operational procedure. The dynamic annotation system automatically activates and displays matching anatomical structure annotation groups based on preset training stage logic, avoiding information overload or omissions. This precise synchronization seamlessly integrates theoretical guidance with practical operation; narration progresses alongside operational steps, and annotations update as the visual focus changes. It creates a highly coherent and guided learning experience, eliminating cognitive interference and allowing trainees to fully concentrate on skills practice. This promotes the immediate application and consolidation of anatomical knowledge in operational scenarios, enhancing the overall fluency and teaching efficiency of the training. Attached Figure Description

[0017] Figure 1 This is a timing diagram of the extended reality-based subarachnoid puncture training system described in this invention. Figure 2 A flowchart for a four-stage progressive training program; Figure 3 This is a flowchart for explaining how the module works in sync. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] In the description of this invention, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, in the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0020] See Figure 1 After system startup, a waiting scenario is presented for the user to select a mode. The scene loading module loads the corresponding 3D anatomy training scene or panoramic video training scene based on the user's selection command within this scene. During scene loading, the system uses an asynchronous loading method, displaying a loading indicator and disabling user interaction during scene switching. Once resource loading and initialization are complete, an initialized training scene is generated. The process control module then initiates a preset four-stage progressive training process based on this initialized training scene. This module coordinates the calling order and timing of various functional modules in the system through a built-in event mechanism and asynchronous task chain, thereby generating an ordered sequence of training content. The visualization processing module uses this ordered sequence of training content to drive the material rendering control module to manipulate the material properties of the 3D human body model, achieving a gradual change in transparency from skin to deep tissues, thus generating a visualization effect of the anatomical structure. The synchronous explanation module is activated based on the visualization of the anatomical structure. Its internal dynamic annotation system calls and displays the corresponding anatomical annotation group according to the current training stage. Simultaneously, the timeline-based multimedia synchronous playback mechanism is triggered, playing pre-recorded audio explanations and corresponding subtitles in sync, generating an interactive training screen with synchronous explanations. Throughout the training process, the interactive control module continuously listens for operation commands input by the user through the extended reality interactive control module, responding to the user's control over the playback status of the training process, realizing the pause, resumption, or return to waiting scenarios of the training process, ultimately generating a user-led training process.

[0021] In one embodiment of the present invention, see [reference] Figure 2The process control module initiates a four-stage progressive training process based on the initialized training scenario. In the overall anatomical structure recognition stage, the process control module sends instructions to the material rendering control module, smoothly transitioning the transparency attribute of the human model's material from opaque to semi-transparent, thereby generating an overall anatomical view that allows observation of the bones and tissue structures beneath the skin. Next, in the detailed explanation of bony structures stage, the process control module, through the scene animation module, controls the spinal skeleton model to fly into the center of the scene with a smooth, scaled-down animation, and automatically rotates to the preset optimal viewing angle after reaching the predetermined position, generating a key display of the bony structures. Finally, in the puncture path and tissue layer display stage, the entire process of the puncture needle penetrating the body layer by layer is simulated. The puncture animation playback module starts running, moving the virtual puncture needle from the starting point in the skin layer. The tissue layer detection algorithm determines the spatial relationship between the needle tip and each tissue layer in real time. When a penetration event is detected, a pause command is sent to the animation control module, and the puncture animation playback pauses. Simultaneously, the dynamic annotation system is invoked to display the name of the currently penetrated tissue layer. The voice explanation module plays the anatomical features and key operational points related to that tissue layer. After the voice explanation ends, the system receives a continue command, resuming the puncture animation playback to the next tissue layer. This process is repeated until the puncture needle penetrates all preset tissue layers, generating a puncture path demonstration that includes multiple pauses for explanation. Finally, the neural structure and injection demonstration stage begins. The neural structures inside the subdural space are displayed using perspective effects, and particle effects are activated to simulate the injection process and fluid flow animation, generating a complete demonstration of drug diffusion and cerebrospinal fluid outflow. The flow control module combines the above overall anatomical view, key display of bony structures, puncture path demonstration of layer-by-layer penetration, and complete demonstration of drug diffusion and cerebrospinal fluid outflow into a complete training content sequence according to the preset four-stage order.

[0022] In practice, the process control module initiates a four-stage progressive training process based on the initialized training scenario. During the overall anatomical structure recognition stage, the process control module sends instructions to the material rendering control module to smoothly transition the transparency attribute of the human body model's material from an opaque state to a semi-transparent state. The transparency transition process follows linear interpolation calculation, and the calculation formula is as follows: ; in: This represents the transparency value of the current frame. This indicates the initial transparency value. This represents the target's transparency value. This indicates the time elapsed since the start of the transition. This represents the preset total transition time. By updating the transparency value frame by frame, a holistic anatomical view is generated that allows observation of the bones and tissue structures beneath the skin. In some embodiments, the flow control module controls the spinal skeleton model to fly into the center of the scene with a smooth, scaled-down translation animation via the scene animation module and automatically rotate to the optimal viewing angle. The translation path of the spinal skeleton model is defined by a Bézier curve in three-dimensional space. The scaling factor increases linearly from its initial value to the target value during the animation. The rotation angle is achieved through quaternion spherical linear interpolation, thereby generating a view that highlights the bony structure.

[0023] In implementation, the puncture path and tissue layer display stage simulates the complete process of the puncture needle entering the body layer by layer from the skin surface. The puncture animation playback module starts running, causing the virtual puncture needle to move from the starting point in the skin layer. The tissue layer detection algorithm determines the spatial positional relationship between the puncture needle tip and each tissue layer in real time. When a penetration event is detected, the tissue layer detection algorithm sends a pause command to the animation control module, and the puncture animation playback pauses immediately. Simultaneously, the dynamic annotation system is invoked to display the name of the currently penetrated tissue layer. The voice narration module synchronously plays anatomical features and operational points related to the current tissue layer. After the voice narration ends, the system receives a continue command and resumes the puncture animation playback to the next tissue layer. This process repeats until the puncture needle penetrates all preset tissue layers, generating a puncture path demonstration including multiple pauses and narrations. It can be understood that the tissue layer detection algorithm determines penetration events based on the intersection test between the puncture needle tip coordinates and the surface grid of each tissue layer, and the movement speed of the virtual puncture needle is controlled by the animation curve to ensure smooth movement. Optionally, the dynamic annotation system uses a fade-in / fade-out text animation effect when displaying the tissue layer name, and the audio playback of the voice narration module is strictly synchronized with the animation pause state.

[0024] In some embodiments, the neural structure and injection demonstration stage displays the neural structures inside the subdural space using a perspective effect and activates particle effects to simulate the injection process and fluid flow animation. The particle effects generate a complete demonstration of drug diffusion and cerebrospinal fluid outflow based on preset emitter parameters and physical simulation rules. The flow control module combines the overall anatomical view, the key bony structure display, the layer-by-layer puncture path demonstration, and the complete demonstration of drug diffusion and cerebrospinal fluid outflow into a complete training content sequence according to a preset order. It can be understood that each stage of the training content sequence is triggered by an event mechanism within the flow control module, and the switching between stages is managed by an asynchronous task chain to ensure timing correctness. Optionally, the automatic rotation speed of the vertebral skeleton model in the key bony structure display is configurable, and the penetration order of tissue layers in the puncture path demonstration strictly follows the definition of real anatomical structures.

[0025] In one embodiment of the present invention, see [reference] Figure 3The synchronous explanation module, based on the current training stage, retrieves the corresponding anatomical annotation groups from the annotation group management module. These anatomical annotation groups include overall structure groups, bony structure groups, puncture path groups, and neural structure groups. To clearly indicate the correspondence between the annotation text and the 3D model, the system employs a dynamic connection annotation mechanism based on anchor point pairing. Anatomical structure anchor points are set at preset anatomical landmarks on the 3D model, and corresponding annotation text anchor points are set on the user interface canvas. During each frame rendering process, the system calculates the projection coordinates of the anatomical structure anchor points in screen space, and then dynamically generates a Bézier curve or straight line connecting the two points as a visual connection based on these projection coordinates and the screen coordinates of the annotation text anchor points. The graphics renderer updates the vertex data of this visual connection in real time, ensuring that the connection updates correctly as the model moves or the user's perspective changes. Gradient animation effects control the display and hiding of the visual connection, including transparency gradients and line fading. The system simultaneously initiates a time-axis-based multimedia synchronous playback mechanism, matching the subtitle content corresponding to the current playback time based on structured stored subtitle data, and ensuring that the audio player's progress is synchronized with the subtitle display. The system integrates annotations, animated connections, audio narration, and subtitles into the current scene rendering, generating a multimedia-synchronized interactive training screen. The display status of the visual connections can be controlled based on user interaction with the interface, including always visible, hidden, or only displayed during specific training phases.

[0026] In practice, the synchronous explanation module is activated based on the visualization of anatomical structures. The module retrieves the corresponding anatomical annotation group from the annotation group management module according to the current training stage. These anatomical annotation groups include overall structure group, bony structure group, puncture path group, and neural structure group. In practice, the system uses a dynamic connection annotation mechanism based on anchor point pairing. Preset anatomical landmarks on the 3D model are used to set anatomical structure anchor points, and corresponding annotation text anchor points are set on the user interface canvas. During each frame rendering, the system calculates the projection coordinates of the anatomical structure anchor points in screen space. Based on the projection coordinates and the screen coordinates of the annotation text anchor points, a Bézier curve connecting the two points is dynamically generated as the visualization connection. The parametric equation of the quadratic Bézier curve used to generate the visualization connection is expressed as: ; in: This represents the coordinates of the point on the curve corresponding to the parameter u. This represents the screen projection coordinates of the starting point, i.e., the anchor point of the anatomical structure. This indicates the screen coordinates of the endpoint, i.e., the anchor point of the labeled text. The control point coordinates are derived by vertically offsetting a fixed number of pixels from the midpoint between the start and end points. The parameter u ranges from 0 to 1. The graphics renderer updates the vertex data of the visualized connections in real time to ensure that the connections update correctly as the model moves or the viewpoint changes. In some embodiments, the display and hiding of the visualized connections are controlled by gradient animation effects, including transparency gradients and line fading. The transparency gradient is achieved by changing the alpha channel value of the connection through linear interpolation, and the line fading is achieved by dynamically drawing the growth animation of the connection by controlling the initial and ending values ​​of parameter u in the parametric equation of the quadratic Bézier curve.

[0027] In implementation, the system synchronously initiates a time-axis-based multimedia synchronized playback mechanism. It matches the subtitle content corresponding to the current playback time based on structured subtitle data and ensures the audio player's progress is synchronized with the subtitle display. This integrates annotation display, connection animations, voice narration, and subtitle content into the current scene rendering, generating a multimedia synchronized interactive training screen. The structured subtitle data is stored in a database as key-value pairs of timestamps and text content, and the audio player provides a playback progress query interface accurate to milliseconds for real-time synchronization. Optionally, the display status of the visual connections can be controlled based on user interaction with the interface. Display status includes always visible, hidden, or only visible during specific training phases. Switching between display statuses is achieved by updating the rendering queue status of the corresponding connection in the graphics renderer.

[0028] In some embodiments, the positions of anatomical anchor points are bound to preset bone nodes or mesh vertex data when the 3D human model is imported, while the positions of annotation text anchor points are arranged according to a layout algorithm during user interface canvas initialization. It is understood that the annotation text anchor point arrangement algorithm needs to avoid overlap between anchor points and consider their approximate orientation relative to the anatomical anchor points in the 3D scene. Optionally, Bézier curve control points... The vertical offset direction can be determined based on the starting point coordinates. coordinates of the endpoint The relative positions are dynamically adjusted to maintain visual consistency in the curvature of the connecting lines. Optionally, the audio files for the narration and the subtitle data files have the same globally unique timeline identifier, and the timeline-based multimedia synchronous playback mechanism uses this identifier to associate and synchronously load the data.

[0029] In one embodiment of the present invention, the interactive control module continuously monitors button events on the controllers of the head-mounted virtual reality display device. When a button B is pressed, the module switches the playback state of the current training content. If the current state is playback, a pause operation is performed and the control interface is brought up; if the current state is paused, a resume playback operation is performed. When a button A is pressed, the module determines the current context. If the operation is during training, it is considered a confirmation operation; if it is in the control interface, it is interpreted as a command to return to the waiting scene. In the panoramic video training mode, the module controls the viewing angle of the virtual camera by reading the input value of the right joystick. Specifically, this is achieved by offsetting the position along the local Z-axis of the virtual camera and limiting the offset to a set minimum and maximum range. Based on the processing results of the above operation commands, the interactive control module updates the state of the internal state machine of the training scene and adjusts the presentation of the scene content accordingly, thereby achieving user-led control over the training process.

[0030] In specific implementation, the interaction control module encapsulates a unified controller input detection module. This module continuously listens for button events from the controllers of the head-mounted virtual reality display device. It polls the controller's button status at a fixed frame rate or registers message callbacks for specific button events. In some embodiments, when a button press event (B button) is detected, the controller input detection module switches the playback state of the current training content. If it's currently playing, it pauses and brings up the control interface; if it's currently paused, it resumes playback. The playback state is determined based on the "playing" or "paused" status flags defined in the training scene's internal state machine. When a button press event (A button) is detected, the controller input detection module determines the current context. The determination logic is based on the user's current interface level and mode. If in the 3D training scene, it triggers an interaction event for the currently focused object as a confirmation operation; if in the control interface, it triggers a scene switching process as a command to return to the waiting scene.

[0031] In practical implementation, in the panoramic video training mode, the viewing angle of the virtual camera is controlled by the input value of the right joystick. The control process is manifested as a position offset along the local Z-axis direction of the virtual camera. The position offset d is calculated based on the formula: ; Where: d represents the displacement of the virtual camera along its local Z-axis in the current frame, and k represents the preset sensitivity coefficient. This represents the time increment since the last frame was rendered. This represents the initial input value of the right joystick in the vertical direction, with a value range of [-1.0, 1.0]. Each calculated displacement d is accumulated in the virtual camera's current world coordinates, and the component of the virtual camera's position in the local Z-axis direction is limited to a set minimum distance. and maximum distance value Within the range. It can be understood that the local Z-axis direction of the virtual camera is determined by the camera's own orientation, and the joystick input value... The sign of the value determines the direction of displacement, whether forward or backward. Optionally, the sensitivity coefficient k can be adjusted according to user settings, with a minimum distance value. and maximum distance value The radius is set according to the spherical model of the panoramic video during scene initialization.

[0032] In some embodiments, the interactive control module updates the internal state machine of the training scenario based on the results of the operation instructions. The internal state machine includes multiple state nodes such as "Waiting Scenario," "3D Training in Progress," "3D Training Paused," "Panoramic Video Playing," and "Panoramic Video Paused." The transition between state nodes is triggered by pressing the A or B button on the controller. It can be understood that the state machine update is directly related to the interface calls of the scenario management module, used to load or unload corresponding scenario resources and adjust the display elements of the user interface. Optionally, after detecting a button event, the controller input detection module encapsulates the event type, controller identifier, and timestamp into a unified data structure and passes it to the event dispatcher. Optionally, the calling and hiding operations of the control interface are accompanied by fade-in and fade-out animations of user interface elements, with the animation duration synchronized with the state machine state transition events.

[0033] In one embodiment of the present invention, the panoramic video display module responds to the user's instruction to enter the panoramic video training mode in a waiting scenario, loads the specified panoramic video resources, and initializes the playback controller. The video playback module is responsible for decoding and controlling the playback of the panoramic video. It initializes the video decoder, loads the specified panoramic video file into a memory buffer, and creates an independent rendering thread to render the decoded video frames onto a 360° spherical projection model. This module provides a playback control interface, supporting start playback, pause playback, resume playback, and stop playback operations. During playback, the video playback module obtains the current playback timestamp in real time and queries the subtitle database to match the subtitle content to be displayed at that time. Simultaneously, it adjusts the playback progress of the audio player to ensure that the audio, video, and subtitle content are synchronized. This module can also handle user-issued jump commands through the interactive interface, locating a specified time point to continue playback. The panoramic video display module responds to application foreground / background switching events, automatically pausing video playback when the application loses focus and automatically resuming playback from the pause point when the application regains focus. The module internally manages the playback process through a playback state machine. The state machine defines five states: not ready, waiting to play, playing, user paused, and system paused, to ensure the correctness of the state switching logic. After the video playback is complete, the module automatically triggers the scene switching logic, causing the system to return to the waiting scene; the user can also manually trigger the command to return to the waiting scene during playback using the A button on the gamepad.

[0034] In practical implementation, the panoramic video display module of the system responds to the user's instruction to enter the panoramic video training mode in the waiting scenario. The panoramic video display module loads the specified panoramic video resources and initializes the playback controller. The loading process involves reading video file metadata from the storage medium and verifying format compatibility. The video playback module is responsible for the decoding and playback control of the panoramic video. The video playback module initializes the video decoder and loads the specified panoramic video file into the memory buffer. The size of the memory buffer is calculated based on the video bitrate and the preset pre-read duration. The video playback module creates an independent rendering thread to render the decoded video frames onto a 360° spherical projection model. The vertex coordinates and texture coordinates of the spherical projection model are pre-calculated and loaded into the graphics processor's video memory during initialization. See Table 1.

[0035] Table 1: Panoramic Video Playback State Machine Transition Rules Current status Triggering event Target state Not ready Video resource loading and decoder initialization completed To be played To be played Received start playback command Playing Playing The user manually pressed the pause button. User pause Playing Application loses focus (system event) System pause User pause The user actively pressed the continue button. Playing System pause Application restores focus (system event) Playing Playing Video plays to the end of file Not ready User pause The user manually pressed the stop button. Not ready System pause The user manually pressed the stop button. Not ready The video playback module provides a playback control interface, including functions for starting playback, pausing playback, resuming playback, and stopping playback. These interface functions are mapped to virtual buttons or gamepad button events in the user interface. During playback, the video playback module obtains the current playback timestamp in real time, queries the subtitle database to match the subtitle content to be displayed, and synchronously adjusts the playback progress of the audio player to ensure that the audio, video, and subtitle content are synchronized. The video playback module handles user-issued jump commands through the interactive interface, locating the specified time point to continue playback. The jump function is implemented through frame index lookup and keyframe positioning technology of the video decoder. In some embodiments, the panoramic video display module responds to application foreground / background switching events, automatically pausing video playback when the application loses focus and automatically resuming playback from the pause point when the application regains focus. Focus event listening is implemented through the application lifecycle callback interface provided by the operating system. It can be understood that the automatic pause and resume function requires access to the video playback module's internal timer to record and restore accurate playback time points.

[0036] The panoramic video display module manages the playback process through a playback state machine. The playback state machine defines five states: not ready, waiting to play, playing, user paused, and system paused. The state transition rules are shown in Table 1 to ensure the logical correctness of state switching. After video playback is complete, the panoramic video display module automatically triggers scene switching logic, causing the system to return to the waiting scene. The trigger condition is the "playback ended" event thrown by the video playback module. Users can understand that they can also manually trigger the return to the waiting scene during playback using the A button on the controller; manual triggering has higher priority than automatic switching logic. Optionally, the subtitle database is stored in a structured format, with each record containing a start timestamp, end timestamp, and subtitle text content. A binary search algorithm is used for query matching to improve efficiency. Optionally, the 360° spherical projection model is generated using latitude and longitude mapping, and its rendering shader supports bilinear filtering of video textures to improve visual quality.

[0037] In one embodiment of the invention, a material rendering control module is used to achieve smooth control of the transparency of a 3D model. The module maintains two material state instances for the human model: an opaque material instance and a transparent material instance. The module switches between the two material states asynchronously, updating the transparency attribute value of the material frame by frame during the switching process. Specifically, the module defines an asynchronous task for material switching, which receives a starting transparency value, a target transparency value, and a transition duration as parameters. In each frame rendering loop, it calculates the ratio of the elapsed time since the task started to the total duration. Based on this ratio, it uses a linear interpolation algorithm to calculate the transparency value to be set for the current frame. The calculated transparency value is assigned to the corresponding shader attribute of the 3D model material. After the transition duration ends, it ensures that the material transparency accurately reaches the target transparency value and triggers a switching completion event. By using shader programming, the transparency value for each frame is calculated through linear interpolation, achieving a smooth transition effect from opaque to semi-transparent or vice versa. The module dynamically adjusts the transparency parameters of different anatomical structure models according to the needs of the current training phase to highlight specific tissue layers. The material rendering control module receives instructions from the training process control module, triggers material switching operations at specific time points, and ensures that the transparency transition animation can be coordinated and synchronized with other training content such as voice explanations during the switching process.

[0038] In practical implementation, the material rendering control module is used to smoothly control the transparency of the 3D model. This module maintains two states for the human model: opaque material instances and transparent material instances. The opaque material instances have a transparency attribute value of "complete occlusion," while the transparent material instances have a preset semi-transparent value. The module switches between these two material states asynchronously, updating the transparency attribute value of the material frame by frame during the switching process. An asynchronous task for material switching is defined, receiving the initial transparency value, the target transparency value, and the transition duration as parameters. In each frame rendering loop, the ratio of the elapsed time since the start of the asynchronous material switching task to the total duration is calculated. Based on this ratio, a linear interpolation algorithm is used to calculate the transparency value to be set for the current frame. The calculation process follows the formula: ; in: This represents the transparency value of the current frame. Indicates the initial transparency value. Indicates the target transparency value. This indicates the cumulative time elapsed since the asynchronous task of material switching began. This indicates the total transition duration received by the asynchronous task of material switching. The calculated transparency value is assigned to the corresponding shader property of the 3D model's material. After the transition duration ends, the material transparency is ensured to precisely reach the target transparency value, and a switch completion event is triggered.

[0039] In some embodiments, shader programming is used to calculate the transparency value for each frame through linear interpolation, achieving a smooth transition effect from opaque to semi-transparent. The shader program runs on the graphics processor and receives the transparency value calculated and passed by the central processing unit each frame. It is understandable that the material rendering control module dynamically adjusts the transparency parameters of different anatomical structures according to the needs of the current training phase to highlight specific tissue layers. This dynamic adjustment process is achieved by independently calling asynchronous tasks for material switching for different anatomical structure models. Optionally, opaque and transparent material instances are loaded as material assets during system initialization. Both types of material instances share the same shader but have different initial parameter configurations.

[0040] In implementation, the material rendering control module receives instructions from the training process control module and triggers a material switching operation at a specific time point. These instructions include the target anatomical structure identifier, the target transparency value, and the transition duration. During the switching process, the transparency transition animation is coordinated and synchronized with other training content, such as audio explanations. This synchronization mechanism relies on a unified event timeline within the training process control module, aligning the start time of the asynchronous material switching task with the playback time of the audio explanation. In some embodiments, the transparency value... The value range is standardized to 0.0 to 1.0, where 0.0 represents complete transparency and 1.0 represents complete opacity. This is the initial transparency value. and target transparency value All are specified within this scope. It can be understood that updating the transparency attribute value of materials frame by frame is performed during the update phase of the graphics rendering loop, ensuring that visual changes are synchronized with the frame rate. Optionally, the asynchronous material switching task will send a switching completion event notification to the training flow control module after the transition duration ends. The training flow control module will then decide whether to trigger subsequent training content based on this notification. When it is necessary to switch the transparency of multiple anatomical structures simultaneously, the material rendering control module manages multiple independent asynchronous material switching task instances in parallel, with each asynchronous material switching task instance managing an independent 3D model material.

[0041] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments that can be applied to other fields. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A training system for subarachnoid puncture based on extended reality, characterized in that, include: The scene loading module loads the corresponding 3D anatomy training scene or panoramic video training scene according to the user's selection command in the waiting scene. It adopts an asynchronous loading method, displays a loading indicator and disables user interaction during scene switching, and generates an initialized training scene. The process control module, based on the initialized training scenario, initiates a four-stage progressive training process, coordinating various functional modules through an event mechanism and asynchronous task chain to generate an ordered sequence of training content. The visualization processing module uses the ordered sequence of training content to control the material rendering control module to perform transparency gradient processing on the three-dimensional human body model, generating a visualization effect of the anatomical structure from the skin to the deep tissues. The synchronous explanation module, based on the visualization effect of the anatomical structure, activates the dynamic annotation system, displays the corresponding anatomical annotation groups in sequence according to the current training stage, and synchronously plays the audio explanation and subtitles through the multimedia synchronous playback mechanism based on the time axis, generating an interactive training screen with synchronous explanation; The interactive control module responds to user input commands via the extended reality interactive control module, controlling the playback status of the training process, enabling the training process to be paused, resumed, or returned to a waiting scenario, and generating a user-led training process.

2. The extended reality-based subarachnoid puncture training system according to claim 1, characterized in that, Based on the initialized training scenario, a four-stage progressive training process is initiated. Through an event mechanism and asynchronous task chains, various functional modules are coordinated to generate an ordered sequence of training content, including: During the overall anatomical structure recognition stage, the material rendering control module smoothly transitions the human body model from an opaque state to a semi-transparent state, generating an overall anatomical view that allows observation of the bones and tissue structures beneath the skin. In the detailed analysis of the skeletal structure, the scene animation module controls the spinal skeleton model to fly into the center of the scene with a smooth displacement animation with scaling effect and automatically rotate to the best viewing angle, generating a key display of the skeletal structure. During the puncture path and tissue layer display stage, the complete process of the puncture needle entering the body layer by layer is simulated. When the puncture needle penetrates each layer of tissue, the animation is automatically paused and the current tissue layer is marked, generating a puncture path demonstration of penetrating layer by layer. During the neural structure and injection demonstration phase, the neural structure inside the subdural space is displayed through a perspective effect, and particle effects are activated to simulate the injection process and fluid flow animation, generating a complete demonstration of drug diffusion and cerebrospinal fluid outflow. Based on the overall anatomical view, the key images of bony structures, the demonstration of the puncture path penetrating layer by layer, and the complete demonstration of drug diffusion and cerebrospinal fluid outflow, the training content is combined into a complete sequence according to a preset order.

3. The extended reality-based subarachnoid puncture training system according to claim 2, characterized in that, During the puncture path and tissue layer demonstration phase, the entire process of the puncture needle entering the body layer by layer from the surface is simulated. The animation automatically pauses and marks the current tissue layer as the puncture needle penetrates each layer, generating a demonstration of the puncture path through each layer, including: The control module for puncture animation playback starts running, causing the virtual puncture needle to move from the starting point of the skin layer; The spatial relationship between the tip of the puncture needle and each tissue layer is determined in real time using a tissue layer detection algorithm. When a penetration event is detected, a pause command is sent to the animation control module. According to the pause command, the playback of the puncture animation is paused, and the dynamic annotation system is invoked to display the name of the currently penetrated tissue layer. Activate the voice explanation module to play the anatomical features and key operational points related to the current tissue level; After the audio explanation ends, a continuation command is received, the puncture animation resumes playback to the next tissue layer, and the process is repeated until all preset tissue layers are penetrated, generating a puncture path demonstration that includes multiple pauses in the explanation.

4. The extended reality-based subarachnoid puncture training system according to claim 1, characterized in that, Based on the visualization of the anatomical structures, the dynamic annotation system is activated, and the corresponding anatomical annotation groups are displayed sequentially according to the current training stage. Simultaneously, audio explanations and subtitles are played using a time-axis-based multimedia synchronous playback mechanism, generating an interactive training screen with synchronized explanations, including: Based on the current training stage, the corresponding anatomical annotation group is called from the annotation group management module. The anatomical annotation group includes the overall structure group, the bony structure group, the puncture path group, and the neural structure group. By using a dynamic connection annotation mechanism based on anchor point pairing, a visual connection is established between the anatomical structure anchor points on the 3D model and the annotation text anchor points on the user interface. The display and hiding of the visual connection is controlled by gradient animation effects, including transparency gradients and line gradual visibility. The time-axis-based multimedia synchronous playback mechanism is launched simultaneously, matching the subtitle content corresponding to the current playback time according to the structured stored subtitle data, and ensuring that audio playback and subtitle display remain synchronized. The annotation display, line-connecting animation, voice explanation and subtitle content are integrated into the current scene rendering screen to generate a multimedia synchronized interactive training screen.

5. The extended reality-based subarachnoid puncture training system according to claim 4, characterized in that, A dynamic connection annotation mechanism based on anchor point pairing is used to establish visual connections between anatomical structure anchor points on the 3D model and annotation text anchor points on the user interface, including: Set anatomical structure anchor points on the preset anatomical landmarks of the 3D human body model, and set corresponding annotation text anchor points on the user interface canvas; During each frame rendering process, the projection coordinates of the anatomical structure anchor points in screen space are calculated; Based on the projection coordinates and the screen coordinates of the labeled text anchor points, dynamically generate a Bézier curve or straight line connecting the two points as a visual connection. The vertex data of the visualized connection is updated in real time by the graphics renderer to ensure that the connection is updated correctly as the model moves or the view changes. Based on the user's interaction with the interface, the display status of the visual connection is controlled, including always visible, hidden, or only visible during specific training phases.

6. The extended reality-based subarachnoid puncture training system according to claim 1, characterized in that, Responding to user input commands via the extended reality interactive control module, the system controls the playback status of the training process, enabling pause, resumption, or return to waiting scenarios, and generating a user-driven training process, including: A unified controller input detection module is encapsulated to continuously monitor controller button events of head-mounted virtual reality display devices; When the B button on the controller is pressed, the playback status of the current training content is switched. If the current playback status is playing, the content is paused and the control interface is brought up. If the current playback status is paused, the content is resumed. When a button A on the controller is pressed, the current context is determined. If the context is during training, it is treated as a confirmation operation; if the context is on the control interface, it is treated as a command to return to the waiting scenario. In the panoramic video training mode, the viewing angle of the virtual camera is controlled by the input value of the right joystick, the position is offset along the local Z-axis of the virtual camera, and the offset is limited to a set range. Based on the results of the operation instructions, update the internal state machine of the training scenario and adjust the presentation of the scenario content accordingly.

7. The extended reality-based subarachnoid puncture training system according to claim 1, characterized in that, It also includes a panoramic video display module to provide an immersive panoramic medical procedure video viewing experience, including: In the waiting scenario, in response to the user's instruction to enter the panoramic video training mode, the panoramic video resources are loaded and the playback controller is initialized. The video playback module is responsible for decoding and controlling the playback of panoramic videos, supporting play, pause, and jump operations, and driving subtitle matching and audio synchronization; Respond to application foreground / background switching events, automatically pause video playback when the application loses focus, and automatically resume playback from the pause point when the application regains focus; The playback state machine manages five states: not ready, waiting to play, playing, user paused, and system paused, ensuring the correctness of the state switching logic. After the video finishes playing, a scene switch is automatically triggered, returning to the waiting scene. Users can also manually return to the waiting scene during playback using the A button on the controller.

8. The extended reality-based subarachnoid puncture training system according to claim 7, characterized in that, The video playback module is responsible for decoding and controlling the playback of panoramic videos, supporting play, pause, and jump operations, and driving subtitle matching and audio synchronization, including: Initialize the video decoder and load the specified panoramic video file into the memory buffer; Create a separate rendering thread to render the decoded video frames onto a 360° spherical projection model; Provides a playback control interface, including start playback, pause playback, resume playback, and stop playback; During playback, the current playback timestamp is obtained in real time, and the subtitle database is queried to match the subtitle content that should be displayed; The playback progress of the audio player is adjusted synchronously to ensure that the audio, video, and subtitle content are synchronized in time. It processes user commands to jump to a specific point in time and continues playback.

9. The extended reality-based subarachnoid puncture training system according to claim 1, characterized in that, The system also includes a material rendering control module for smooth control of the transparency of the 3D model, including: Maintain two states for the human body model: opaque material instances and transparent material instances. Switching between two material states is done asynchronously, updating the material's transparency attribute value frame by frame during the switching process; By using shader programming, the transparency value of each frame is calculated through linear interpolation to achieve a smooth transition effect from opaque to semi-transparent. Based on the needs of the current training phase, the transparency parameters of different anatomical structures are dynamically adjusted to highlight specific tissue layers; Receive instructions from the training process control module, trigger material switching operations at specific time points, and ensure that the transition animation is synchronized with the voice narration.

10. A training system for subarachnoid puncture based on extended reality according to claim 9, characterized in that, Switching between two material states asynchronously, updating the material's transparency attribute value frame by frame during the switching process, including: Define an asynchronous task for material switching, wherein the asynchronous task receives a starting transparency value, a target transparency value, and a transition duration as parameters; In each frame rendering loop, calculate the ratio of the time elapsed since the start of the task to the total duration; Based on the ratio, the transparency value that should be set for the current frame is calculated using a linear interpolation algorithm; The calculated transparency value is assigned to the corresponding shader property of the 3D model material; After the transition duration ends, ensure that the material transparency accurately reaches the target transparency value and trigger the transition completion event; During the transition, ensure that other training content, such as animation playback and audio narration, is coordinated with this change in transparency.

Images