A Puncture Training Method Based on Extended Reality

Abstract

This specification discloses an extended reality-based puncture training method, which includes: displaying a virtual human body model, a virtual puncture tool, and an image display control to a user through an extended reality device, wherein the virtual human body model is generated based on medical body data, and the image display control is configured to display a target cross-sectional image, the target cross-sectional image corresponding to the target cross-section of the target lesion to be punctured in the virtual human body model; and updating at least one of the target cross-sectional image and the virtual human body model in response to the user's control command on the virtual puncture tool, so as to display the real-time position of the virtual puncture tool in the virtual human body model.

Description

Technical Field

[0001] This invention relates to the field of medical simulation technology, and in particular to a puncture training method based on extended reality. Background Technology

[0002] Puncture is an indispensable key technique in modern medicine, and its precise operation directly affects the accuracy of diagnosis and the safety of treatment. However, traditional training models rely heavily on instructor experience, static phantoms, or limited simulations, and generally suffer from problems such as non-standardized procedures, difficulty in quantifying operational steps, and lengthy learning cycles, resulting in beginners lacking standardized and systematic skill development paths. In recent years, extended reality (AR) technology, especially virtual reality (VR) and mixed reality (MR), has provided innovative solutions for puncture training, such as 3D visualization, path planning, and immersive simulation, showing potential in improving operational accuracy and optimizing assessment. However, existing AR systems still have significant limitations: their guidance mechanisms are often mechanical and rigid, failing to provide intelligent, adaptive, real-time guidance based on the learner's cognitive stage and operational proficiency; the training process is disconnected from the learner's skill development curve, lacking dynamic difficulty feedback and progressive support, resulting in insufficient training efficiency and depth.

[0003] Therefore, there is an urgent need to develop a more intelligent extended reality puncture training system that can grow with learners' abilities and provide personalized guidance to fill the current technological gap. Summary of the Invention

[0004] One embodiment of this specification provides a puncture training method based on extended reality. The puncture training method includes: displaying a virtual human body model, a virtual puncture tool, and an image display control to a user via an extended reality device, wherein the virtual human body model is generated based on medical body data, and the image display control is configured to display a target cross-sectional image, the target cross-sectional image corresponding to the target cross-section of the target lesion to be punctured in the virtual human body model; and updating at least one of the target cross-sectional image and the virtual human body model in response to the user's control command of the virtual puncture tool, to display the real-time position of the virtual puncture tool in the virtual human body model.

[0005] In some embodiments, in response to a user's control command on a virtual puncture tool, the real-time coordinate information of the virtual puncture tool in the virtual world coordinate system corresponding to the virtual human body model is determined; based on the real-time coordinate information, image elements representing the virtual puncture tool are superimposed on the target cross-section image to display the real-time position of the virtual puncture tool in the target cross-section.

[0006] In some embodiments, in response to a user's control command on a virtual puncture tool, the real-time coordinate information of the virtual puncture tool in the virtual world coordinate system corresponding to the virtual human body model is determined; based on the real-time coordinate information, a display adjustment area centered on the virtual puncture tool is determined; and the surface tissue model located within the display adjustment area is visualized and adjusted to reveal the corresponding internal anatomical structure model, thereby displaying the real-time position of the virtual puncture tool relative to the internal anatomical structure model.

[0007] In some embodiments, the size of the display adjustment area is proportional to the puncture depth of the virtual puncture tool.

[0008] In some embodiments, in response to a user's control command on a virtual puncture tool, it is determined whether the virtual puncture tool collides with a key internal anatomical structure in the internal anatomical structure model; in response to determining that the virtual puncture tool collides with a key internal anatomical structure, a visual cue or virtual obstruction feedback is triggered through an augmented reality device.

[0009] In some embodiments, a movable cross-sectional positioning plane is superimposed and displayed on a virtual human body model using an extended reality device; in response to a user's instruction to adjust the position of the cross-sectional positioning plane, the current cross-sectional image corresponding to the current cross-sectional position is determined based on medical body data, and the current cross-sectional image is updated and displayed in real time in the image display control; in response to a path planning instruction input by the user, the cross-section located at the current cross-sectional position in the virtual human body model is taken as the target cross-section, and the current cross-sectional image is taken as the target cross-sectional image.

[0010] In some embodiments, the puncture path drawn by the user on the target cross-sectional image is obtained; the actual physical length corresponding to the puncture path is determined; and a label representing the actual physical length is added at the corresponding position of the puncture path, and the label is displayed through an augmented reality device.

[0011] In some embodiments, a first mark of the needle insertion point drawn by the user at a first position in a target cross-sectional image or another cross-sectional image is obtained; based on the transformation relationship between the medical data coordinate system corresponding to the medical body data and the virtual world coordinate system corresponding to the virtual human body model, a second position corresponding to the first position is determined in the virtual human body model; a second mark of the needle insertion point is added to the second position of the virtual human body model, and the second mark of the needle insertion point is displayed through an extended reality device.

[0012] In some embodiments, a target training mode is determined based on user characteristic information; and an auxiliary tool is determined based on the target training mode and made available to the user. Attached Figure Description

[0013] This specification will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0014] Figure 1 These are schematic diagrams illustrating application scenarios of the puncture training system according to some embodiments of this specification; Figure 2 This is a block diagram of a puncture training system according to some embodiments of this specification; Figure 3 This is an exemplary schematic diagram of a puncture training phase according to some embodiments of this specification; Figure 4 This is an exemplary flowchart illustrating the process of updating at least one of a target cross-sectional image and a virtual human body model according to some embodiments of this specification; Figure 5 This is an exemplary schematic diagram of the virtual content corresponding to the target recognition stage shown in some embodiments of this specification; Figure 6 This is an exemplary schematic diagram of the virtual content corresponding to the path planning stage shown in some embodiments of this specification; Figure 7 These are exemplary schematic diagrams of virtual puncture tools and their corresponding image elements as shown in some embodiments of this specification; Figure 8 This is an exemplary schematic diagram of a determination aid tool according to some embodiments of this specification. Detailed Implementation

[0015] To more clearly illustrate the technical solutions of the embodiments in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some examples or embodiments of this specification. For those skilled in the art, these drawings can be applied to other similar scenarios without creative effort. Unless obvious from the context or otherwise specified, the same reference numerals in the drawings represent the same structures or operations.

[0016] It should be understood that the terms “system,” “device,” “unit,” and / or “module” used herein are one way to distinguish different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they may be replaced by other expressions.

[0017] Unless the context clearly indicates an exception, words such as "a," "an," "a kind," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0018] Flowcharts are used in this specification to illustrate the operations performed by the system according to embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0019] Figure 1 This is a schematic diagram illustrating application scenarios of a puncture training system according to some embodiments of this specification. This puncture training system can be applied in scenarios such as simulation teaching and skills testing. In some embodiments, such as... Figure 1 As shown, the application scenario 100 of the puncture training system may include a user 160, an extended reality device 130, and a virtual environment 170.

[0020] User 160 is the subject of puncture training, such as students, nurses, and doctors.

[0021] Virtual environment 170 refers to an environment constructed using extended reality devices and presented to user 160 for puncture training. In some embodiments, virtual environment 170 may be a virtual examination room space, or other virtual environments for puncture training. Virtual environment 170 may present multiple virtual contents, such as virtual human body model 110, virtual user 140, interactive control interface 150, virtual puncture tools 180, and virtual imaging equipment 190, etc.

[0022] Virtual human body model 110 refers to a three-dimensional human body model simulating a real patient. In some embodiments, virtual human body model 110 can be generated based on medical body data. For more information on how to generate virtual human body model 110, please refer to [link to relevant documentation]. Figure 3 And its explanation.

[0023] Virtual user 140 may include a virtual avatar of user 160 in virtual environment 170. Virtual user 140 may also include a virtual character for providing training guidance, such as virtual user 140 being used to provide operation prompts and feedback information to user 160.

[0024] Interactive control interface 150 refers to an interactive interface that displays information (such as medical images, interactive controls, and prompts) and receives user operations. For example, interactive control interface 150 may display image display controls, cross-sectional position fine-tuning controls, body positioning tools, and window width and level setting tools. For more information on image display controls, cross-sectional position fine-tuning controls, body positioning tools, and window width and level setting tools, please refer to [link to relevant documentation]. Figure 5 And its explanation.

[0025] The virtual puncture tool 180 refers to a virtual instrument used to simulate puncture operations in a virtual environment 170. The user 160 can control the posture and position of the virtual puncture tool 180 through an extended reality device 130 to perform virtual puncture operations.

[0026] Virtual imaging device 190 refers to a virtual device set up in virtual environment 170 for performing medical image acquisition, reconstruction, processing, and / or display. Virtual imaging device 190 may include virtual scanning equipment and virtual components related to medical image processing or display. Virtual imaging device 190 may include virtual CT equipment, virtual MR equipment, etc. In some embodiments, virtual imaging device 190 may include a virtual hospital bed for supporting virtual human body model 110. In some embodiments, the virtual hospital bed may be omitted.

[0027] Extended reality device 130 may include a device that allows user 160 to obtain an extended reality experience. For example, extended reality device 130 may include virtual reality (VR) components, augmented reality (AR) components, mixed reality (MR) components, etc., or any combination thereof. In some embodiments, extended reality device 130 may include extended reality helmets, extended reality glasses, extended reality patches, stereo headphones, etc., or any combination thereof. For example, extended reality device 130 may include Google Glass™, Oculus Rift™, Gear VR™, etc.

[0028] In some embodiments, such as Figure 1 As shown, the extended reality device 130 may include a display component 131 and an input component 132. The display component 131 may be configured to display virtual content (such as a virtual human model and a virtual puncture tool) in the virtual environment 170 to the user 160. The input component 132 may be configured to receive user input for operating the virtual environment 170. In some embodiments, the user 160 may operate the virtual environment 170 directly through gesture control or other interactive methods, without relying on the input component 132. For example, in response to the user 160's upward hand gesture, the virtual puncture tool in the virtual environment 170 may move upward.

[0029] In some embodiments, user 160 can interact with virtual content through display component 131. For example, when user wears display component 131, the user's head movements and / or gaze direction can be tracked to render virtual content in response to changes in user position and / or orientation, thereby providing an immersive extended reality experience that reflects changes in the user's perspective.

[0030] In some embodiments, the input component 132 enables interaction between the user and virtual content (e.g., a virtual human model) displayed on the display component 131. For example, the input component 132 may include a touch sensor, microphone, etc., for receiving user input; the user input may be provided to the extended reality device 130 and used to control the virtual environment by updating the visual content rendered on the display component 131. In some embodiments, the user input received by the input component 132 may include touch, voice input, and / or gesture input, and may be sensed using any suitable sensing technology (e.g., capacitive, resistive, acoustic, optical). In some embodiments, the input component 132 may include a handle, glove, stylus, game console, etc.

[0031] In some embodiments, user 160 can interact with virtual content (such as interactive control interface 150, virtual human body model 110, and virtual puncture tool 180) in virtual environment 170 through extended reality device 130 to complete the target recognition stage, path planning stage, and puncture operation stage. For example, in response to user 160's control command on the virtual puncture tool, extended reality device 130 can update at least one of the target cross-sectional image and virtual human body model 110 to display the real-time position of virtual puncture tool 180 in virtual human body model 110. For more information on the target recognition stage, path planning stage, and puncture operation stage, please refer to [link to relevant documentation]. Figure 3 And its explanation.

[0032] In some embodiments, such as Figure 1 As shown, the application scenario 100 of the puncture training system may further include a processing device 120. The processing device 120 may be communicatively connected to the extended reality device 130 to process data and / or information from the extended reality device 130. For example, the processing device 120 may receive interactive instructions input by the user 160 through the input component 132, update the virtual content in the virtual environment 170 based on the interactive instructions, and present the updated virtual environment 170 via the display component 131. In some embodiments, the processing device 120 may be integrated into the extended reality device 130.

[0033] It should be understood that the above description of the application scenario 100 of the puncture training system is for illustrative purposes only and is not intended to limit the scope of the invention. In some embodiments, the virtual environment 170 may include one or more other virtual contents. Alternatively, one or more virtual contents (such as virtual user 140, virtual imaging device 190) in the virtual environment 170 may be omitted.

[0034] Figure 2 This is a block diagram of a puncture training system according to some embodiments of this specification. For example... Figure 2 As shown, the puncture training system 200 may include a display module 210, an update module 220, a determination module 230, and a display module 240. In some embodiments, Figure 2 The modules shown can be implemented using processing devices.

[0035] The display module 210 is configured to display a virtual human body model, a virtual puncture tool, and an image display control to a user via an extended reality device. The virtual human body model is generated based on medical body data, and the image display control is configured to display a target cross-sectional image that corresponds to the target cross-section of the target lesion to be punctured in the virtual human body model.

[0036] The update module 220 is configured to update at least one of the target cross-sectional image and the virtual human body model in response to a user's control command on the virtual puncture tool, so as to display the real-time position of the virtual puncture tool in the virtual human body model.

[0037] The determining module 230 is configured to determine the real-time coordinate information of the virtual puncture tool in the virtual world coordinate system corresponding to the virtual human body model in response to the user's control command on the virtual puncture tool. In some embodiments, the determining module 230 is further configured to determine a display adjustment area centered on the virtual puncture tool based on the real-time coordinate information.

[0038] In some embodiments, the determining module 230 is further configured to, in response to a user's control command on the virtual puncture tool, determine whether the virtual puncture tool collides with a key internal anatomical structure in the internal anatomical structure model; and, in response to determining that the virtual puncture tool collides with a key internal anatomical structure, trigger a visual cue or virtual obstruction feedback through an augmented reality device.

[0039] In some embodiments, the determining module 230 is further configured to, in response to a user's instruction to adjust the position of the cross-section positioning plane, determine the current cross-section image corresponding to the current cross-section position based on medical body data, and update and display the current cross-section image in the image display control in real time; and, in response to a user's input path planning instruction, take the cross-section located at the current cross-section position in the virtual human body model as the target cross-section, and take the current cross-section image as the target cross-section image.

[0040] In some embodiments, the determining module 230 is further configured to acquire the puncture path drawn by the user on the target cross-sectional image; determine the actual physical length corresponding to the puncture path; add a label representing the actual physical length at the corresponding position of the puncture path; and display the label through an extended reality device.

[0041] In some embodiments, the determining module 230 is further configured to acquire a first mark of the needle insertion point drawn by the user at a first position in a target cross-sectional image or another cross-sectional image; determine a second position in the virtual human body model corresponding to the first position based on the transformation relationship between the medical data coordinate system corresponding to the medical body data and the virtual world coordinate system corresponding to the virtual human body model; add a second mark of the needle insertion point at the second position in the virtual human body model; and display the second mark of the needle insertion point through an extended reality device.

[0042] In some embodiments, the determining module 230 is further configured to determine a target training mode based on the user's characteristic information; and based on the target training mode, determine auxiliary tools that can be made available to the user.

[0043] Display module 240 is configured to overlay image elements representing a virtual puncture tool onto a target cross-sectional image based on real-time coordinate information to display the real-time position of the virtual puncture tool in the target cross-section. In some embodiments, display module 240 is further configured to perform visualization adjustments on the surface tissue model located within the display adjustment area to expose the corresponding internal anatomical structure model, thereby displaying the real-time position of the virtual puncture tool relative to the internal anatomical structure model.

[0044] It should be understood that Figure 2 The modules shown can be implemented in various ways. It should be noted that the above description of the puncture training system 200 and its modules is for convenience only and should not limit this specification to the scope of the illustrated embodiments. It is understood that those skilled in the art, after understanding the principle of the device, may arbitrarily combine the various modules or construct sub-devices connected to other modules without departing from this principle. In some embodiments, Figure 2 The display module 210, update module 220, determination module 230, and display module 240 disclosed herein can be different modules within a single device, or a single module can implement the functions of two or more of the aforementioned modules. For example, the modules can share a single storage module, or each module can have its own separate storage module. Such variations are all within the scope of protection of this specification.

[0045] Figure 3 This is an exemplary schematic diagram illustrating a puncture training phase according to some embodiments of this specification. Figure 3 As shown, the puncture training process can be divided into a target recognition stage, a path planning stage, and a puncture operation stage. In each stage, the user needs to perform different operations. Some embodiments in this specification provide the user with various auxiliary tools (also referred to as scaffolding tools) in each stage to assist the user in completing the operation of that stage.

[0046] The target identification stage is used for lesion localization. The goal of this stage is to assist the user in selecting the target cross-section corresponding to the target lesion in the virtual human body model and generating a target cross-sectional image. For example... Figure 3 As shown, during the target recognition phase, the processing device can perform the following operations.

[0047] Step 311: The virtual human body model is generated based on medical body data.

[0048] Medical volume data refers to three-dimensional volumetric data characterizing human tissues, organs, or lesions. For example, medical volume data may include CT volume data and / or MR volume data. In some embodiments, medical volume data can be acquired by medical imaging equipment (such as CT scanners, MR scanners, etc.) during a medical scan of a patient. For example, medical volume data may be a three-dimensional CT image acquired from a patient's historical CT scan.

[0049] Virtual human body models are three-dimensional, visualized digital patient images generated based on medical data.

[0050] The virtual human body model may include a body surface tissue model, which is used to display the body surface tissue of the virtual human body model. In some embodiments, the processing device may segment and reconstruct the medical body data to obtain a 3D Mesh model suitable for rendering in a virtual environment; configure texture maps for the 3D Mesh model to present the appearance details of the human body structure in the virtual environment; and simultaneously, for areas not included in the medical body data (such as non-scanned areas such as limbs and face), construct supplementary models to complete the areas, and merge the supplementary models with the 3D Mesh model to form the body surface tissue of the virtual human body model.

[0051] In some embodiments, the virtual human body model may further include an internal anatomical structure model. The internal anatomical structure model may be generated based on medical body data and is used to display internal anatomical tissue structures. For example, the internal anatomical structure model may display at least one of the bones, muscles, and organs of the virtual human body model. In some embodiments, the virtual human body model may be a model pre-built and stored based on medical body data. The processing device may directly invoke the pre-built and stored virtual human body model to display to the user before or during puncture training.

[0052] The surface tissue model can encompass the internal anatomical tissue model. The surface tissue model can have a displayed state and a hidden state. When the surface tissue model is displayed, the user can only see the surface tissues of the virtual human model and cannot see the internal anatomical structures. When the surface tissue model is hidden, the user can see the internal anatomical structures of the virtual human model. In some embodiments, the user can adjust the display state of the surface tissue model and / or the internal anatomical structure model using assistive tools. For example, the user can use a surface tissue hiding tool to hide the surface tissue model to display the internal anatomical structure model. As another example, the user can further use an internal anatomical structure hiding tool to select specific internal anatomical structures to hide. More information about assistive tools can be found in [link to relevant documentation]. Figure 8 And its explanation.

[0053] In some embodiments, while the virtual human body model is generated based on medical body data, the processing device can also record the scaling, rotation, and displacement parameters from the medical data coordinate system corresponding to the medical body data to the virtual world coordinate system corresponding to the virtual human body model, and establish the transformation relationship between the medical data coordinate system corresponding to the medical body data and the virtual world coordinate system corresponding to the virtual human body model. Here, the medical data coordinate system refers to the spatial coordinate system corresponding to the medical body data. The virtual world coordinate system corresponding to the virtual human body model refers to the spatial coordinate system corresponding to the virtual environment in which the virtual human body model resides, used to represent the spatial position of the virtual content within the virtual environment.

[0054] Step 312: Present the virtual environment to the user using an extended reality device.

[0055] The processing device can display a virtual environment and virtual content corresponding to the target recognition stage to the user through extended reality devices. The virtual content corresponding to the target recognition stage can include virtual human models, cross-sectional positioning planes, and image display controls.

[0056] Step 313: Using the cross-section positioning plane and image display controls, assist the user in selecting the target cross-section and generate the target cross-section image.

[0057] Figure 5 These are exemplary schematic diagrams of virtual content corresponding to the target recognition stage as shown in some embodiments of this specification. For example... Figure 5 As shown, the virtual content in the virtual environment 170 includes a virtual human body model 110, a cross-sectional positioning plane 510, and an image display control 152.

[0058] An image display control is a display component that displays medical images in a virtual environment. Image display controls can display cross-sectional images. For example, an image display control can display transverse, sagittal, or coronal images of the lungs. In some embodiments, an image display control is configured to display a target cross-sectional image.

[0059] A target cross-section is a specific cross-section selected by the user in a virtual human body model for puncture path planning. For example, a target cross-section may include one or more of a target transverse section, a target sagittal section, and a target coronal section. In some embodiments, the target cross-section is a target transverse section.

[0060] The target cross-sectional image corresponds to the target cross-section of the lesion to be punctured in the virtual human body model. For example, the target cross-sectional image may include a target cross-sectional image, a target sagittal image, and a target coronal image corresponding to the target transverse section, the target sagittal plane, and the target coronal plane, respectively, or a portion of the aforementioned images. In some embodiments, the target cross-sectional image is a target transverse section image corresponding to the target transverse section. The target lesion to be punctured refers to the lesion in the virtual human body model that serves as the puncture target. For example, when the target lesion to be punctured is a pulmonary nodule, the target cross-sectional image may be a lung cross-sectional image containing the pulmonary nodule.

[0061] The cross-sectional positioning plane 510 refers to a movable virtual plane superimposed on the virtual human body model, used to select the spatial location of a cross-section. For example, the cross-sectional positioning plane 510 may include one or more of a transverse section positioning plane, a coronal section positioning plane, and a sagittal section positioning plane. The cross-sectional positioning plane 510 may be a semi-transparent plane parallel to the cross-section of the virtual human body model, and its position can be adjusted to select a specific cross-section location. The position of the cross-sectional positioning plane 510 and the cross-sectional image displayed in the image display control are linked. This linkage means that when the position of the cross-sectional positioning plane 510 is adjusted, the cross-sectional image displayed in the image display control can be updated synchronously to display the cross-sectional image corresponding to the location of the cross-section at the position of the cross-sectional positioning plane 510.

[0062] In some embodiments, the processing device can overlay a movable cross-sectional positioning plane onto a virtual human body model using an extended reality device; in response to a user's instruction to adjust the position of the cross-sectional positioning plane, it determines the current cross-sectional image corresponding to the current cross-sectional position based on medical body data, and updates and displays the current cross-sectional image in real time in the image display control; in response to a path planning instruction input by the user, it takes the cross-section located at the current cross-sectional position in the virtual human body model as the target cross-section, and uses the current cross-sectional image as the target cross-sectional image.

[0063] The position adjustment command refers to the command to translate and adjust the cross-section positioning plane. For example, the user can select the cross-section positioning plane by inputting component 132 or by using gesture control, and then drag the cross-section positioning plane to adjust its position.

[0064] In some embodiments, such as Figure 5As shown, the interactive control interface 150 can display a cross-section position fine-tuning control 151, which allows the user 160 to fine-tune the position of the cross-section positioning plane 510. For example, the position of the cross-section positioning plane 510 can be fine-tuned stepwise or continuously using the cross-section position fine-tuning control 151. By providing the cross-section position fine-tuning control, the adjustment accuracy of the current cross-section position can be improved, thereby reducing the cross-section selection deviation caused by insufficient positioning accuracy and improving the accuracy of subsequent target cross-section determination.

[0065] Taking the cross-sectional positioning plane as an example, whenever the user adjusts the position of the cross-sectional positioning plane using the cross-sectional position fine-tuning control 151, the processing device can determine the mapped cross-sectional position in the medical body data based on the transformation relationship between the medical data coordinate system corresponding to the medical body data and the virtual world coordinate system corresponding to the virtual human body model. Using multi-plane reconstruction technology, the device extracts the cross-sectional image corresponding to the mapped cross-sectional position from the medical body data, using it as the current cross-sectional image, and displays it in the image display control. The mapped cross-sectional position refers to the cross-sectional position corresponding to the current cross-sectional position in the medical body data. Multi-plane reconstruction technology can determine the two-dimensional image corresponding to a plane at any angle in the three-dimensional volume data.

[0066] Path planning commands are commands used to invoke path planning tools in auxiliary tools to perform path drawing operations.

[0067] Users can determine whether a cross-section at the current cross-section position is suitable as a target cross-section for puncture path planning based on the current cross-section image displayed in the image display control. If the user determines it is suitable as a target cross-section, they can input a path planning command. In response to the user's input path planning command, the path planning stage can begin. Simultaneously, the processing device can use the cross-section at the current cross-section position in the virtual human body model as the target cross-section and the currently displayed current cross-section image as the target cross-section image. For example, when a user performs a path drawing operation based on the currently displayed current cross-section image, the cross-section corresponding to the current cross-section position can be determined as the target cross-section, and the currently displayed current cross-section image can be determined as the target cross-section image.

[0068] For example, users can select a cross-section that meets the following clinical criteria as the target cross-section: the lesion area in the cross-section is as large as possible to enable visual observation and localization of the lesion; and the minimum distance between the puncture path planned based on the cross-section and the human structures (such as bones, blood vessels, and airways) around the target lesion is greater than a preset safe distance threshold, so as to reduce the risk of damage to the human structures around the target lesion during the puncture process.

[0069] In some embodiments of this specification, by overlaying a movable cross-sectional positioning plane onto a virtual human body model, users can intuitively understand the spatial relationship between the current cross-sectional position and the target lesion in the virtual environment. They can also quickly traverse different cross-sectional positions by moving the positioning plane, thereby improving the efficiency of target cross-section selection. By selecting the target cross-sectional image from the real-time updated current cross-sectional image, a more suitable imaging basis is provided for subsequent puncture path planning, helping to reduce path planning risks.

[0070] In some embodiments, positioning planes in other directions may be overlaid on the virtual human body model, such as sagittal and / or coronal positioning planes. The image display control can further display sagittal and coronal images. The sagittal positioning plane and the sagittal image displayed in the image display control are linked. The coronal positioning plane and the coronal image displayed in the image display control are also linked. The sagittal and coronal positioning planes are used to assist the user in observing the spatial location of the target lesion in the virtual human body model from different directions.

[0071] In some embodiments, users can also fine-tune the positions of the coronal and sagittal positioning planes using the cross-sectional position fine-tuning controls. For example... Figure 5 As shown, the cross-sectional position fine-tuning control 151 may also include adjustment controls along the coronal plane and along the sagittal plane to perform step-by-step or continuous fine-tuning of the coronal and sagittal positioning planes respectively, and update and display the corresponding sagittal and coronal images in real time in the image display control to assist the user in observing the position of the target lesion from different directions.

[0072] Step 314: Assist the user in setting the window width, window position, and / or body position.

[0073] In some embodiments, such as Figure 5 As shown, the interactive control interface 150 can display a window width and window level setting tool 154 to assist the user in setting the window width and window level of medical images in the image display control. In some embodiments, the window width and window level setting tool includes multiple preset options for window parameters corresponding to multiple body parts, such as preset options for lung windows, mediastinal windows, and bone windows. After the user selects any preset option, the processing device can apply the window width and window level parameters corresponding to the preset option to the currently displayed medical image and update the display results in the image display control in real time. For example, in response to the user selecting the mediastinal window preset option, the processing device can apply the window width and window level parameters corresponding to the mediastinal window preset option to the medical image to adjust the display effect of the target lesion and related soft tissue structures.

[0074] In some embodiments, such as Figure 5As shown, the interactive control interface 150 may also include a body position setting tool 153 to assist users in setting the body position of the virtual human body model.

[0075] Body position refers to the posture of a virtual human body model. Users can use the body position setting tool 153 to set the virtual human body model to supine, prone, left lateral, and right lateral positions, etc., so as to observe the location of the target lesion in different positions.

[0076] In some embodiments, step 314 may be omitted.

[0077] The path planning phase is used to plan the puncture path. The goal of this phase is to assist the user in determining the puncture path and needle insertion point corresponding to the target lesion. Once in the path planning phase, the virtual content in the virtual environment can be updated to reflect the content required for this phase. This virtual content may include a virtual human model, path planning tools, and image display controls.

[0078] like Figure 3 As shown, during the path planning phase, the processing device can perform the following operations.

[0079] Step 321: Assist the user in drawing the puncture path in the target cross-sectional image.

[0080] The puncture path refers to the planned route that guides the virtual puncture tool to the target lesion. After the user selects the target cross-section during the target identification phase, the image display control will fix the corresponding target cross-section image. The user can use the path planning tool in the auxiliary tools to perform path drawing operations on the target cross-section image to draw the puncture path. For more information on auxiliary tools and path planning tools, please refer to [link to relevant documentation]. Figure 8 And its explanation.

[0081] In some embodiments, the puncture path can be a complete path drawn by the user. For example, a complete path can be a path from the body surface tissue to the target lesion. The puncture path can also be a segmented path drawn by the user, consisting of at least two path segments. For example, the puncture path can include a first puncture path and a second puncture path. Taking lung puncture training as an example, since the target lesion is located inside the lung, and the pleura is located between the chest wall and the lung, the virtual puncture tool needs to reach and pass through the pleura before it can continue to advance towards the target lesion when entering the lung from the body surface tissue. The user can draw a first puncture path from the body surface tissue to the pleura on the target cross-sectional image using a path planning tool, and perform a virtual puncture operation based on the first puncture path; in response to the user confirming that the virtual puncture tool has reached the pleura, the processing device can execute... Figure 4Steps 420 and 431 update the target cross-sectional image. Based on the updated target cross-sectional image, the user then draws a second puncture path from the pleura to the target lesion, thus forming a puncture path composed of two path segments. By using a segmented path as the puncture path, the user can determine the needle insertion distance and direction of the first and second puncture paths separately. After the user completes the virtual puncture operation based on the first puncture path, the needle insertion distance and direction of the second puncture path are further determined and adjusted in conjunction with the updated target cross-sectional image, which helps to improve the accuracy of path planning.

[0082] In some embodiments, the processing device may predetermine a standard three-dimensional puncture path as a reference puncture path based on medical body data. When the user draws the puncture path on the target cross-sectional image, the processing device can project the reference puncture path onto the target cross-sectional image to overlay the reference path projection, thereby assisting the user in drawing the puncture path on the target cross-sectional image.

[0083] In some embodiments, when a user performs a path drawing operation on a target cross-sectional image using a path planning tool, the processing device can determine the path deviation between the user-drawn puncture path and the projection of a reference path; in response to the path deviation exceeding a preset deviation threshold, the processing device can control the extended reality device to output a warning message to prompt the user to adjust the puncture path. The preset deviation threshold can be pre-set based on prior knowledge.

[0084] In some embodiments, the processing device may acquire the puncture path drawn by the user on the target cross-sectional image; determine the actual physical length corresponding to the puncture path; add a label representing the actual physical length at the corresponding position of the puncture path; and display the label through an augmented reality device.

[0085] After the user draws the puncture path on the target cross-sectional image using a path planning tool, the processing device can obtain the positional information of the puncture path drawn by the user on the target cross-sectional image, and determine the actual physical length corresponding to the puncture path based on the pixel spacing and slice thickness information in the target cross-sectional image. Here, the pixel spacing in the target cross-sectional image refers to the actual length corresponding to adjacent pixels in the target cross-sectional image; the slice thickness information refers to the thickness parameter of the medical body data in the slice direction, used to characterize the thickness range of the target cross-section. Specifically, the processing device can determine the pixel distance of the puncture path in the target cross-sectional image; and based on the pixel spacing and slice thickness information in the target cross-sectional image, convert the pixel distance of the puncture path in the target cross-sectional image into a physical distance, which is taken as the actual physical length corresponding to the puncture path.

[0086] After determining the actual physical length corresponding to the puncture path, the processing device can automatically add a label representing the actual physical length at the corresponding position of the puncture path in the target cross-sectional image, and display it through an extended reality device, allowing the user to view the puncture path and its corresponding actual physical length in the target cross-sectional image. The label representing the actual physical length can include the length value and the length unit.

[0087] Figure 6 These are exemplary schematic diagrams illustrating the virtual content corresponding to the path planning phase as shown in some embodiments of this specification. For example... Figure 6 As shown, the user can draw the puncture path 630 on the target cross-section image using the path planning tool. The processing device can add a label "5.2cm" representing the actual physical length at the corresponding position of the puncture path, indicating that the actual physical length of the puncture path is 5.2cm.

[0088] In some embodiments, when the puncture path is a segmented path, the processing device can generate segment length labels at the corresponding positions of each path segment, and generate a total length label representing the total physical length.

[0089] In some embodiments of this specification, the extended reality device typically displays a label representing the actual physical length, enabling users to intuitively obtain the length information of the puncture path, thereby improving the accuracy and standardization of path planning and providing a more reliable length reference for subsequent puncture operations.

[0090] Step 322 assists the user in drawing the first mark of the needle entry point in the target cross-sectional image or another cross-sectional image.

[0091] In some embodiments, such as Figure 6 As shown, the processing device can acquire a first mark 610 of the needle insertion point drawn by the user at a first position in a target cross-sectional image or another cross-sectional image; based on the transformation relationship between the medical data coordinate system corresponding to the medical body data and the virtual world coordinate system corresponding to the virtual human body model, determine a second position in the virtual human body model corresponding to the first position; add a second mark 620 of the needle insertion point at the second position in the virtual human body model, and display the second mark 620 of the needle insertion point through an extended reality device.

[0092] The needle insertion point is the location on the body surface tissue where the needle is inserted. The user can select a first location on the target cross-sectional image and draw a first mark 610 at that location.

[0093] Another cross-sectional image refers to a cross-sectional image selected by the user relative to the target cross-sectional image. In some embodiments, the user may also select a first position on the other cross-sectional image and draw a first mark 610 at that position. For example, when the puncture path drawn by the user on the target cross-sectional image does not pass through the surface tissue of the virtual human body model (e.g., the puncture path only includes the path segment from the pleura to the target lesion); or when, during the puncture operation, the user uses an inclined needle insertion to avoid human structures (such as ribs), resulting in the needle insertion point not coinciding with the starting position of the puncture path drawn on the target cross-sectional image, the user may select another cross-sectional image as the other cross-sectional image and select a first position and draw the first mark 610 of the needle insertion point on the other cross-sectional image.

[0094] The first position refers to the location selected by the user on the target cross-sectional image or another cross-sectional image to characterize the needle insertion point. In some embodiments, the user can use the starting position of the puncture path as the first position.

[0095] The first mark refers to an image mark added at a first location to indicate the needle insertion point location on the target cross-sectional image or another cross-sectional image. In some embodiments, the user can draw the first mark on the target cross-sectional or another cross-sectional image using a needle insertion point marking tool.

[0096] The second position refers to the coordinate position in the virtual world corresponding to the first position within the virtual human body model. In some embodiments, the processing device can determine the two-dimensional coordinates of the first position in the medical data coordinate system corresponding to the medical body data; based on the coordinates of the cross-section corresponding to the first position and the two-dimensional coordinates, determine the three-dimensional coordinates of the first position in the medical data coordinate system; based on the transformation relationship between the medical data coordinate system corresponding to the medical body data and the virtual world coordinate system corresponding to the virtual human body model, map the three-dimensional coordinates to the virtual world coordinate system to determine the mapped position. The mapped position can be directly used as the second position. Alternatively, the processing device can use the transformed position as a base point, extend a ray of a preset length (e.g., 1 meter) in a preset reverse direction, determine the intersection point of the ray with the surface tissue of the virtual human body model, and determine the intersection point as the needle insertion point (i.e., the second position) of the virtual human body model.

[0097] The second marker refers to an image marker on the virtual human body model indicating the location of the needle insertion point. In some embodiments, in response to drawing the first marker, the processing device can automatically add the second marker to the needle insertion point of the virtual human body model and display it through an extended reality device. In some embodiments, the first marker and the second marker of the needle insertion point may correspond to the same needle insertion point. The first marker of the needle insertion point is used to characterize the position of the needle insertion point in a target cross-sectional image or another cross-sectional image, and the second marker of the needle insertion point is used to characterize the position of the needle insertion point in the virtual human body model.

[0098] In some embodiments of this specification, by drawing a first mark of the needle insertion point on the target cross-sectional image or another cross-sectional image, and determining a second position corresponding to the first position in the virtual human body model and displaying a second mark of the needle insertion point, the user can intuitively see the spatial correspondence of the needle insertion point in the two-dimensional image and the three-dimensional model, reducing the cost of repeated positioning and adjustment, improving the user's spatial position perception ability, and facilitating subsequent puncture operations to be performed according to the needle insertion point.

[0099] The puncture operation phase refers to the stage where the virtual puncture procedure is performed. The goal of this phase is to assist the user in performing the virtual puncture, understanding the real-time position of the virtual puncture tool during the procedure, and fine-tuning the puncture path based on the real-time position. Once the puncture operation phase begins, the virtual content in the virtual environment can be updated to correspond to the virtual content of this phase. The virtual content corresponding to the puncture operation phase may include a virtual human body model, a virtual puncture tool, and image display controls. For example... Figure 3 As shown, during the puncture procedure, the treatment device can perform the following operations.

[0100] Step 331: Display the virtual puncture tool to the user using an augmented reality device and obtain the user's control commands for the virtual puncture tool. Step 332: Update at least one of the target cross-sectional image and the virtual human body model to assist the user in understanding the real-time position of the virtual puncture tool within the virtual human body model.

[0101] In some embodiments, after the path planning phase is completed, the processing device may control the extended reality device to automatically display the puncture tool configuration control; or, in response to a user's triggering operation on the puncture tool configuration control, the puncture tool configuration control may be displayed. The user can select the virtual puncture tool to be used in the puncture operation phase through the puncture tool configuration control.

[0102] In response to user commands to operate the virtual puncture tool, the processing device can update at least one of the target cross-sectional image and the virtual human body model to display the real-time position of the virtual puncture tool within the virtual human body model. More information can be found at [link to relevant documentation]. Figure 4 And its explanation.

[0103] Step 333: Perform collision detection on the virtual puncture tool to help the user understand whether it has collided with critical internal anatomical structures.

[0104] In some embodiments, collision detection of the virtual puncture tool to assist the user in understanding whether a collision has occurred with a critical internal anatomical structure includes: in response to the user's control command of the virtual puncture tool, determining whether the virtual puncture tool has collided with a critical internal anatomical structure in the internal anatomical structure model; and in response to determining that the virtual puncture tool has collided with a critical internal anatomical structure, triggering a visual cue or virtual obstruction feedback through an augmented reality device.

[0105] Key internal anatomical structures may include bones and / or other internal tissue structures that need to be avoided during puncture. In some embodiments, the internal anatomical model may include pre-annotated key internal anatomical structures.

[0106] In some embodiments, in response to a user's control command on the virtual puncture tool, the processing device can acquire real-time coordinate information of the virtual puncture tool in the virtual world coordinate system; based on the real-time coordinate information, a collision detection line segment is constructed. The collision detection line segment is positioned along the axial direction of the virtual puncture tool, and its length corresponds to the actual length of the virtual puncture tool. The processing device can perform intersection detection between the collision detection line segment and key internal anatomical structures in the internal anatomical structure model to determine whether a collision has occurred between the virtual puncture tool and a key internal anatomical structure.

[0107] In response to determining that the virtual puncture tool has collided with a critical internal anatomical structure, the processing device can trigger visual cues and / or virtual obstruction feedback via an augmented reality device, and restrict the movement of the virtual puncture tool to reduce the risk of continued misoperation; in response to determining that the virtual puncture tool has not collided with the critical internal anatomical structure, the virtual puncture tool remains maneuverable. Visual cues may include highlighting the collision area and / or displaying warning colors (e.g., red warning).

[0108] Virtual obstruction feedback is a form of feedback used to make users perceive the effect of a virtual puncture tool encountering resistance. In some embodiments, this feedback can be presented through an augmented reality device, showing the effect of the virtual puncture tool encountering an obstacle, thereby creating a pseudo-tactile sensory experience in a specific scenario. Implementation methods of virtual obstruction feedback include, but are not limited to, controlling the virtual puncture tool to stop moving forward, reverse its movement, rebound, or change its trajectory to simulate obstruction. For example, when the virtual puncture tool collides with key internal anatomical structures such as bones in a virtual scene, the processing device can control it to stop moving forward and rebound, thus giving the user a sensory experience similar to real obstruction, enhancing immersion and realism during the operation.

[0109] In some embodiments of this specification, by detecting whether the virtual puncture tool collides with key internal anatomical structures in the internal anatomical structure model, users can avoid the risk of accidentally touching key internal anatomical structures (such as bones, blood vessels, or nerves) during training. Through visual cues or virtual obstruction feedback, users can quickly adjust the operation path or posture of the virtual puncture tool based on the cues, reducing accidental contact with key internal anatomical structures and thus lowering the risk of accidental contact with key internal anatomical structures.

[0110] In some embodiments of this specification, a progressive puncture training process, from target identification and localization to path planning and then to the puncture operation stage, ensures both the structure and repeatability of the puncture training process and takes into account the degree of operational freedom at different training levels. This is beneficial for improving the learning efficiency, operational consistency, and comprehensive evaluation effectiveness of puncture training.

[0111] Figure 4 This is an exemplary flowchart illustrating a process for updating at least one of a target cross-sectional image and a virtual human body model, according to some embodiments of this specification.

[0112] In some embodiments, such as Figure 4 As shown, updating the target cross-sectional image includes steps 410, 420, and 431.

[0113] Step 410: Obtain the user's control instructions for the virtual puncture tool.

[0114] The control commands for the virtual puncture tool include commands to control the advancement, retraction, angle adjustment, and / or posture adjustment of the virtual puncture tool.

[0115] Step 420: Determine the real-time coordinate information of the virtual puncture tool in the virtual world coordinate system corresponding to the virtual human body model.

[0116] Real-time coordinate information can characterize the real-time position of the virtual puncture tool in the virtual world coordinate system corresponding to the virtual human body model. In some embodiments, in response to the user's control command on the virtual puncture tool, the processing device can update the position of the virtual puncture tool in the virtual world coordinate system corresponding to the virtual human body model in real time, and determine the updated position as the real-time coordinate information of the virtual puncture tool.

[0117] Step 431: Based on real-time coordinate information, overlay image elements representing the virtual puncture tool onto the target cross-section image to display the real-time position of the virtual puncture tool in the target cross-section.

[0118] The image elements of a virtual puncture tool refer to graphic identifiers that represent the real-time coordinate information of the virtual puncture tool in the target cross-section. For example, the image elements of a virtual puncture tool can be line segments.

[0119] In some embodiments, the processing device can represent the virtual puncture tool as a needle segment in a virtual world coordinate system based on real-time coordinate information, and obtain the cross-sectional parameters corresponding to the target cross-section; construct the cross-sectional volume region corresponding to the target cross-section based on the cross-sectional parameters, and determine whether the needle segment intersects with the cross-sectional volume region. The cross-sectional parameters may include the cross-sectional position, cross-sectional direction, and cross-sectional thickness.

[0120] In response to the intersection between the needle body segment and the cross-sectional area corresponding to the virtual puncture tool, the processing device can determine the intersecting line segment between the needle body segment and the cross-sectional area, and superimpose the line segment corresponding to the intersecting line segment in the target cross-sectional image as an image element of the virtual puncture tool; in response to the absence of intersection between the needle body segment and the cross-sectional area, the processing device can choose not to superimpose the image element representing the virtual puncture tool in the target cross-sectional image.

[0121] Figure 7 These are exemplary schematic diagrams of virtual puncture tools and their corresponding image elements, as shown in some embodiments of this specification. Figure 7 As shown, the image element 710 representing the virtual puncture tool is overlaid on the target cross-section image in the image display control. The user can use this element to understand the real-time position of the virtual puncture tool in the target cross-section. Since the target cross-section has a preset cross-sectional thickness, the image element 710 representing the virtual puncture tool can correspond to the virtual puncture tool 180 being located in a part of the target cross-section.

[0122] In some embodiments of this specification, by determining the real-time coordinates of the virtual puncture tool in the virtual world coordinate system corresponding to the virtual human body model and overlaying image elements representing the virtual puncture tool onto the target cross-section image, the user can intuitively see the real-time position of the virtual puncture tool in the target cross-section. The user can promptly determine the current posture of the virtual puncture tool based on the image elements and make fine adjustments, thereby reducing repeated trials and ineffective adjustments and improving the efficiency of puncture training.

[0123] In some embodiments, such as Figure 4 As shown, updating the virtual human body model includes steps 410, 420, 432, and 442. Detailed descriptions of steps 410-420 are provided above and will not be repeated here.

[0124] Step 432: Based on real-time coordinate information, determine the display adjustment area centered on the virtual puncture tool.

[0125] The display adjustment area refers to the spatial area in a virtual human body model where the display of the body surface tissue model is adjusted. The display adjustment area can be a cylindrical area, a conical area, or other preset shaped areas.

[0126] In some embodiments, users can use the tissue visualization tool within the assistive tools to visualize the surface tissues of a virtual human body model, determine the display adjustment area centered on the virtual puncture tool, observe the internal anatomical structures near the puncture tool, and assist the user in adjusting the puncture path. For more information on assistive tools and tissue visualization tools, please refer to [link to relevant documentation]. Figure 8 And its explanation.

[0127] In some embodiments, in response to a user's triggering operation on the tissue imaging tool, the processing device can use the needle tip position of the virtual puncture tool as the center position of the bottom surface of the display adjustment area, and construct the display adjustment area along the direction of the virtual puncture tool.

[0128] In some embodiments, the size of the adjustment area (e.g., the area of ​​the base) is proportional to the puncture depth of the virtual puncture tool. For example, the greater the puncture depth of the virtual puncture tool, the larger the area of ​​the adjustment area.

[0129] Step 442: Visualize the surface tissue model located within the display adjustment area to reveal the corresponding internal anatomical structure model, thereby displaying the real-time position of the virtual puncture tool relative to the internal anatomical structure model.

[0130] Visualization adjustments refer to modifying the way a surface tissue model is displayed to reveal the corresponding internal anatomical structures. For example, visualization adjustments may include adjusting transparency, hiding, partially hiding, weakening display, highlighting outlines, or other methods that facilitate the exposure of internal anatomical structures.

[0131] For example, the processing device can set the transparency parameter of the surface tissue model located within the display adjustment area to a preset transparency, so that the surface tissue model is displayed transparently or semi-transparently within the display adjustment area. The preset transparency can be any suitable value such as 50%, 60%, or 100%. The transparency of the surface tissue model outside the display adjustment area will not be adjusted, i.e., it will be 0. In this case, while keeping the surface tissue model outside the display adjustment area visible, the corresponding internal anatomical structure model within the display adjustment area can be revealed.

[0132] In some embodiments, in response to changes in the position of the virtual puncture tool, the processing device can synchronously update the position and size of the display adjustment area based on the real-time coordinate information of the virtual puncture tool in the virtual world coordinate system corresponding to the virtual human body model, so as to continuously display the real-time position of the virtual puncture tool relative to the internal anatomical structure model.

[0133] In some embodiments of this specification, by defining a display adjustment area centered on the virtual puncture tool and visually adjusting the surface tissue model located within the display adjustment area, local visualization of anatomical structures near the virtual puncture tool is achieved, allowing users to intuitively see the relative positional relationship between the virtual puncture tool and internal anatomical structures. Furthermore, the size of the display adjustment area is proportional to the puncture depth, and the display range of the display adjustment area can adaptively adjust with the puncture depth of the virtual puncture tool, enhancing the visualization capability of the internal anatomical structure model and thereby improving the operational accuracy of puncture training.

[0134] Figure 8 These are exemplary schematic diagrams illustrating the determination of auxiliary tools according to some embodiments of this specification. Figure 8 As shown, the processing device can determine the target training mode 830 from the forced guidance mode 821, the flexible jump mode 822 and the open and free mode 823 based on the user's characteristic information 810; and determine the auxiliary tools 840 that are open to the user based on the target training mode 830.

[0135] User characteristics can indicate the extent to which a user has received puncture training. For example, user characteristics can indicate whether a user is a new user who has not received puncture training, and whether the user has completed all operations in the target recognition, path planning, and puncture operation phases. In some embodiments, user characteristics may include training records corresponding to the user account.

[0136] Forced guidance mode refers to a training mode that guides users step-by-step during the initial learning phase. In forced guidance mode, the processing device can guide the user to complete the target recognition phase, path planning phase, and puncture operation phase in sequence according to the order of the puncture training phase, and does not allow the user to skip phases or certain operations.

[0137] Flexible jump mode refers to an optional training mode for advanced learning stages. In flexible jump mode, the processing device allows users to freely choose or repeat the training between the target recognition stage, the path planning stage, and the puncture operation stage, so as to conduct targeted training for the weak puncture training stage.

[0138] The open-free mode refers to a highly realistic training mode geared towards the proficiency stage. In this mode, the processing device can remove the sequential restrictions and default guidance prompts of the puncture training phase and open all auxiliary tools, providing a puncture training environment that closely resembles the actual operation process. In this mode, the processing device can also record and evaluate the user's operation process and output the evaluation results after training.

[0139] In some embodiments, in response to the user's feature information indicating that the user is a new user, the processing device can use the forced guidance mode as the target training mode; in response to the user not being a new user but not having completed all three stages of the puncture training phase, the flexible jump mode can be determined as the target training mode; in response to the user not being a new user and having completed all three stages of the puncture training phase, the open and free mode can be determined as the target training mode. The processing device can determine whether the user has completed all three stages of the puncture training phase based on the training records in the user's feature information; if no training records exist, the user is determined to be a new user.

[0140] Auxiliary tools refer to functional components that provide users with enhanced visualization, comparative evaluation, operational prompts, and display control during puncture training. These tools may include: cross-section positioning planes, image display controls, cross-section position fine-tuning controls, window width and level setting tools, path planning tools, body position setting tools, needle insertion point marking tools, reference needle insertion point display tools, real-time rendering tools, body surface tissue hiding tools, bone hiding tools, and tissue perspective tools, etc.

[0141] For more information on cross-section positioning plane, image display controls, cross-section position fine-tuning controls, window width and level setting tools, path planning tools, and body position setting tools, please refer to... Figure 3 And its explanation.

[0142] The path planning tool is used to draw and display the puncture path on the target cross-sectional image, and can overlay a reference puncture path. Users can compare whether the puncture path drawn on the target cross-sectional image deviates from the reference puncture path. For more information on reference puncture paths, please refer to [link to relevant documentation]. Figure 3 And its explanation.

[0143] In some embodiments, the path planning tool can also simultaneously display the corresponding 3D puncture path in a virtual human body model for users to compare the 3D paths. The 3D puncture path can be determined based on the needle insertion point and the puncture path itself. For more information on needle insertion points and puncture paths, please refer to [link to relevant documentation]. Figure 3 And its explanation.

[0144] The reference needle insertion point display tool is used to display the standard needle insertion point determined in advance based on medical body data, so that users can compare whether the needle insertion point deviates from the standard needle insertion point.

[0145] The real-time rendering tool is used to trigger virtual scans and update cross-sectional images. Users can trigger a single virtual scan by selecting the "Start Scan" button on the real-time rendering tool. Optionally, users can enable real-time rendering through the real-time rendering tool to continuously perform virtual scans and update the cross-sectional images in the image display controls in real time.

[0146] The surface tissue hiding tool is used to hide the surface tissues of a virtual human body model to reveal the internal anatomical structure; the skeleton hiding tool is used to hide the skeletal tissues in a virtual human body model to meet the observation needs of different puncture training.

[0147] The tissue fluoroscopy tool is used to expose the internal anatomical structure model corresponding to the body surface tissue model, thereby showing the real-time position of the virtual puncture tool relative to the internal anatomical structure model.

[0148] Different auxiliary tools are used for different puncture stages. For example, auxiliary tools for the target identification stage may include cross-section positioning plane, image display control, cross-section position fine-tuning control, window width and window level setting tool, and body position setting tool; auxiliary tools for the path planning stage may include path planning tool, needle insertion point marking tool, and reference needle insertion point display tool; auxiliary tools for the puncture operation stage may include real-time rendering tool, tissue perspective tool, body surface tissue hiding tool, and bone hiding tool, etc.

[0149] In some embodiments, in response to the target training mode being a forced-guided mode, the processing device may only provide the user with the auxiliary tools corresponding to the target recognition stage, and then provide the corresponding auxiliary tools to the user after they enter the path planning stage and the puncture operation stage. In response to the target training mode being a flexible-jump mode, the processing device may provide the auxiliary tools corresponding to the puncture stage selected by the user. In response to the target training mode being an open-free mode, the processing device may provide all auxiliary tools corresponding to the three stages for the user to choose freely.

[0150] In some embodiments of this specification, by defining the target training model and providing users with auxiliary tools, tiered management of the puncture training process can be achieved. For example, new users can be provided with mandatory guidance and limited access to certain tools to reduce the risk of misoperation and accelerate the establishment of basic skills; advanced users can access tools on demand to support targeted reinforcement of weak areas and improve training efficiency; and skilled users can access the complete set of tools to conduct highly realistic comprehensive drills and competency assessments. This allows for a progressive teaching loop from beginner to expert, improving the safety and personalization of puncture training.

[0151] The basic concepts have been described above. Obviously, for those skilled in the art, the detailed disclosure above is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are suggested in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

[0152] Furthermore, this specification uses specific terms to describe embodiments thereof. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Moreover, certain features, structures, or characteristics in one or more embodiments of this specification can be appropriately combined.

[0153] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this specification are not intended to limit the order of the processes and methods described herein. Although various examples have been discussed in the foregoing disclosure of some embodiments of the invention that are currently considered useful, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the spirit and scope of the embodiments described herein. For example, while the system components described above can be implemented using hardware devices, they can also be implemented solely using software solutions, such as installing the described system on existing servers or mobile devices.

[0154] Similarly, it should be noted that, in order to simplify the description disclosed herein and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of embodiments in this specification may sometimes combine multiple features into a single embodiment, drawing, or description thereof. However, this method of disclosure does not imply that the subject matter of this specification requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of a single embodiment disclosed above.

[0155] In some embodiments, numbers describing the quantity of components and attributes are used. It should be understood that such numbers used in the description of embodiments are modified in some examples with the terms "approximately," "approximately," or "generally." Unless otherwise stated, "approximately," "approximately," or "generally" indicates that the numbers are allowed to vary by ±20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximate values, which may be changed depending on the characteristics required by individual embodiments. In some embodiments, numerical parameters should take into account specified significant digits and employ a general method of digit reservation. Although the numerical ranges and parameters used to confirm their breadth of range in some embodiments of this specification are approximate values, in specific embodiments, such values ​​are set as precisely as feasible.

[0156] For each patent, patent application, patent application publication, and other material such as articles, books, specifications, publications, and documents referenced in this specification, the entire contents of which are incorporated herein by reference. This excludes historical application documents that are inconsistent with or conflict with the content of this specification, as well as documents that limit the broadest scope of the claims in this specification (currently or subsequently appended to this specification). It should be noted that in the event of any inconsistency or conflict between the descriptions, definitions, and / or terminology used in the supplementary materials to this specification and the content of this specification, the descriptions, definitions, and / or terminology used in this specification shall prevail.

[0157] Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments described herein. Other variations may also fall within the scope of this specification. Therefore, alternative configurations of the embodiments described herein are intended to be illustrative rather than limiting, and should be considered consistent with the teachings of this specification. Accordingly, the embodiments described herein are not limited to those explicitly introduced and described herein.

Claims

1. A puncture training method based on extended reality, characterized in that, include: The system uses an augmented reality device to display a virtual human body model, a virtual puncture tool, and an image display control to the user. The virtual human body model is generated based on medical body data, and the image display control is configured to display a target cross-sectional image that corresponds to the target cross-section of the target lesion to be punctured in the virtual human body model. In response to the user's control command on the virtual puncture tool, at least one of the target cross-sectional image and the virtual human body model is updated to display the real-time position of the virtual puncture tool in the virtual human body model.

2. The method as described in claim 1, characterized in that, Updating at least one of the target cross-sectional image and the virtual human body model to display the real-time position of the virtual puncture tool in the virtual human body model includes: In response to the user's control command on the virtual puncture tool, determine the real-time coordinate information of the virtual puncture tool in the virtual world coordinate system corresponding to the virtual human body model; and Based on the real-time coordinate information, image elements representing the virtual puncture tool are superimposed on the target cross-section image to display the real-time position of the virtual puncture tool in the target cross-section.

3. The method as described in claim 1, characterized in that, The virtual human body model includes an internal anatomical structure model and a body surface tissue model that wraps around the internal anatomical structure model. Updating at least one of the target cross-sectional image and the virtual human body model to display the real-time position of the virtual puncture tool in the virtual human body model includes: In response to the user's control command on the virtual puncture tool, the real-time coordinate information of the virtual puncture tool in the virtual world coordinate system corresponding to the virtual human body model is determined; Based on the real-time coordinate information, a display adjustment area centered on the virtual puncture tool is determined; and The surface tissue model located within the display adjustment area is visualized and adjusted to reveal the corresponding internal anatomical structure model, thereby displaying the real-time position of the virtual puncture tool relative to the internal anatomical structure model.

4. The method as described in claim 3, characterized in that, The size of the display adjustment area is proportional to the puncture depth of the virtual puncture tool.

5. The method as described in claim 1, characterized in that, The virtual human body model includes an internal anatomical structure model, and the method further includes: In response to the user's control command on the virtual puncture tool, determine whether the virtual puncture tool collides with key internal anatomical structures in the internal anatomical structure model; and In response to determining that the virtual puncture tool has collided with the critical internal anatomical structure, a visual cue or virtual obstruction feedback is triggered via the extended reality device.

6. The method as described in claim 1, characterized in that, The target cross-sectional image is determined in the following way: The extended reality device overlays and displays a movable cross-sectional positioning plane on the virtual human body model; In response to the user's instruction to adjust the position of the cross-section positioning plane, the current cross-section image corresponding to the current cross-section position is determined based on the medical body data, and the current cross-section image is updated and displayed in the image display control in real time; In response to the path planning command input by the user, the cross section located at the current cross section position in the virtual human body model is taken as the target cross section, and the current cross section image is taken as the target cross section image.

7. The method as described in claim 1, characterized in that, Before obtaining the control command, the method further includes: Obtain the puncture path drawn by the user on the target cross-sectional image; Determine the actual physical length corresponding to the puncture path; and A label representing the actual physical length is added at the corresponding position of the puncture path, and the label is displayed through the extended reality device.

8. The method as described in claim 1, characterized in that, Before obtaining the control command, the method further includes: Obtain the first mark of the needle insertion point drawn by the user at the first position in the target cross-sectional image or another cross-sectional image; Based on the transformation relationship between the medical data coordinate system corresponding to the medical body data and the virtual world coordinate system corresponding to the virtual human body model, a second position corresponding to the first position is determined in the virtual human body model. A second mark is added to the second position of the needle insertion point on the virtual human body model, and the second mark of the needle insertion point is displayed through the extended reality device.

9. The method as described in claim 1, characterized in that, The method further includes: Based on the user's characteristic information, determine the target training mode; and Based on the target training model, auxiliary tools are determined to be made available to the users.

10. A puncture training system based on extended reality, characterized in that, include: The display module is configured to display a virtual human body model, a virtual puncture tool, and an image display control to the user via an extended reality device. The virtual human body model is generated based on medical body data, and the image display control is configured to display a target cross-sectional image, which corresponds to the target cross-section of the target lesion to be punctured in the virtual human body model. The update module is configured to update at least one of the target cross-sectional image and the virtual human body model in response to the user's control command on the virtual puncture tool, so as to display the real-time position of the virtual puncture tool in the virtual human body model. The determination module is configured to determine the real-time coordinate information of the virtual puncture tool in the virtual world coordinate system corresponding to the virtual human body model in response to the user's control command on the virtual puncture tool; The display module is configured to overlay image elements representing the virtual puncture tool onto the target cross-section image based on the real-time coordinate information, so as to display the real-time position of the virtual puncture tool in the target cross-section.

Images