Synchronous display system and method for different-place remote ultrasonic equipment based on mixed reality

Abstract

The invention discloses a mixed reality-based remote ultrasonic equipment synchronous display system, which comprises expert end equipment, a cloud server platform and remote end equipment, and is characterized in that the expert end equipment comprises a dummy model, an ultrasonic probe model and an action tracker, the action tracker is installed on the ultrasonic probe model, and the dummy model is connected with the cloud server platform; the motion tracker is configured to be capable of capturing the movement track and angle change of the ultrasonic probe model on the dummy model, converting the movement track and angle change into digital signals and transmitting the digital signals to the cloud server platform, the remote end equipment comprises an MR helmet and a handle, the handle is configured to be capable of calibrating markers on the body surface of a testee, and the motion tracker is configured to be capable of capturing the movement track and angle change of the ultrasonic probe model on the dummy model. The MR helmet is arranged on the cloud server platform to generate a virtual dynamic three-dimensional human body model, the MR helmet is provided with a display screen and a space sensing module, and the MR helmet is configured to be capable of receiving data from the cloud server platform and superposing actions of a virtual probe on the real body of the testee in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of remote ultrasound examination, and particularly to a synchronous display system and method for remote ultrasound devices based on mixed reality across different locations. Background Art

[0002] Traditional remote ultrasound examination systems usually rely on video transmission or simple virtual reality (VR) technology, and have the following deficiencies:

[0003] 1. Experts need to wear helmets to operate, which reduces the operation freedom and is prone to fatigue;

[0004] 2. The alignment accuracy between the virtual environment and the actual physical space is insufficient, resulting in operation errors;

[0005] 3. There is a lack of a dynamic body shape adaptation mechanism, making it difficult to match the body shape differences between patients and dummies;

[0006] 4. The privacy protection measures are not perfect, and there is a risk of sensitive area data leakage.

[0007] Therefore, the technicians in this field are committed to developing a synchronous display system and method for remote ultrasound devices based on mixed reality across different locations. By using three-dimensional (3D) printed probe models, mixed reality (MR) spatial coordinate anchoring, and dynamic equal-proportion scaling algorithms, the above problems are solved, and the accuracy and operation efficiency of remote ultrasound examination are significantly improved. Summary of the Invention

[0008] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to provide a synchronous display system and method for remote ultrasound devices based on mixed reality across different locations to overcome the above-mentioned deficiencies of the prior art.

[0009] To achieve the above object, the present invention provides a synchronous display system for remote ultrasound devices based on mixed reality across different locations, including an expert-side device, a cloud server platform, and a remote-side device. The expert-side device includes a dummy model, an ultrasound probe model, and an action tracker. The action tracker is installed on the ultrasound probe model and is configured to be able to capture the movement trajectory and angle change of the ultrasound probe model on the dummy model and convert them into digital signals for transmission to the cloud server platform. The remote-side device includes an MR helmet and a handle. The handle is configured to be able to calibrate the markers on the subject's body surface to generate a virtual dynamic three-dimensional human body model. The MR helmet is equipped with a display screen and a spatial perception module, and the MR helmet is configured to be able to receive data from the cloud server platform and superimpose the actions of the virtual probe on the real body of the subject in real time.

[0010] Further, the dummy model is made of silicone and is configured to simulate the body structure of the subject.

[0011] Further, the ultrasonic probe model is made by 3D printing and has a similar shape and size to the actual ultrasonic probe.

[0012] Further, the cloud server platform includes a data transmission module, and the data transmission module is communicatively connected to the motion tracker and the MR helmet.

[0013] Further, the cloud server platform further includes a user management module, and the user management module is configured to allow users to access the system according to their permissions.

[0014] The present invention also provides a method for using the above-mentioned mixed reality-based remote ultrasonic device synchronous display system, including the following steps:

[0015] Step 1: An expert operates on the dummy model using the ultrasonic probe model to simulate a real ultrasonic examination process;

[0016] Step 2: The motion tracker captures the movement trajectory and angle change of the ultrasonic probe model in real time and converts them into digital signals for transmission to the cloud server platform;

[0017] Step 3: Collect the body information of the subject;

[0018] Step 4: Use a physical positioning clamp and manual calibration to construct a virtual coordinate system to achieve dynamic alignment between the virtual and real spaces;

[0019] Step 5: The examiner wears the MR helmet, and the MR helmet receives data from the cloud server platform and superimposes the actions of the virtual probe on the real body of the subject in real time.

[0020] Further, the specific steps of step 3 include the following steps:

[0021] Step 3.1: The examiner wears the MR helmet and uses the handle to calibrate the markers on the surface of the subject;

[0022] Step 3.2: The system generates a virtual dynamic three-dimensional human body model of the subject according to the markers and adjusts the size of the virtual dynamic three-dimensional human body model through an equal ratio scaling algorithm to adapt it to the actual body shape of the subject.

[0023] Further, the specific steps of step 4 include the following steps:

[0024] Step 4.1: Set the physical positioning clamp and combine it with manual calibration of the markers on the surface of the subject to construct a virtual coordinate system;

[0025] Step 4.2: The system realizes the precise mapping between the virtual space and the real physical space through physical positioning clamps and manual calibration points.

[0026] Furthermore, in Step 5, the display screen of the MR helmet only shows the actions of the virtual probe to avoid visual interference caused by the complete virtual dynamic three-dimensional human model.

[0027] Furthermore, the following steps are also included:

[0028] Step 6: The system real-time monitors the position of the virtual probe through the boundary detection algorithm;

[0029] Step 7: If it is detected that the virtual probe is approaching the boundary, the system automatically adjusts the position of the probe.

[0030] The beneficial effects of the present invention are as follows: improving the freedom and comfort of experts' operations, reducing fatigue; reducing visual interference for remote examiners, improving operation accuracy and efficiency, and enhancing collaboration reliability; improving the alignment accuracy between the virtual and real spaces, reducing operation errors; solving the problem of body shape differences, improving applicability; ensuring operation accuracy and consistency, protecting patients' privacy and safety, and conforming to medical norms; and improving system security, enhancing operation accuracy and safety.

[0031] The following will further illustrate the concept, specific structure and technical effects of the present invention with reference to the drawings to fully understand the purpose, features and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a schematic diagram of the system architecture and data flow of a preferred embodiment of the present invention;

[0033] Figure 2 is a flow chart of information collection and virtual-real coordinate alignment of a preferred embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of the anti-penetration detection mechanism of a preferred embodiment of the present invention;

[0035] Figure 4 is a schematic diagram of an expert operation of a preferred embodiment of the present invention;

[0036] Figure 5 is a schematic diagram of a remote examiner operation of a preferred embodiment of the present invention.

[0037] Among them, 10 - expert-side device, 11 - dummy model, 12 - ultrasonic probe model, 13 - motion tracker, 20 - cloud server platform, 21 - data transmission module, 22 - user management module, 30 - remote-side device, 31 - MR helmet, 32 - handle, 40 - subject, 50 - expert, 60 - examiner, 70 - actual ultrasonic probe. Detailed implementation manners

[0038] The following introduces multiple preferred embodiments of the present invention with reference to the accompanying drawings of the specification, making its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0039] In the drawings, components with the same structure are denoted by the same numerical labels, and components with similar structures or functions everywhere are denoted by similar numerical labels. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. To make the drawings clearer, the thickness of some components is appropriately exaggerated in the drawings.

[0040] Embodiment 1

[0041] As Figure 1 、 4 、and shown in FIG. 5, this embodiment provides a system for synchronously displaying remote ultrasonic devices based on mixed reality, including an expert-side device 10, a cloud server platform 20, and a remote-side device 30. The expert-side device 10 includes a dummy model 11, an ultrasonic probe model 12, and a motion tracker 13. The motion tracker 13 is installed on the ultrasonic probe model 12. The motion tracker 13 can capture the movement trajectory and angular changes of the ultrasonic probe model 12 on the dummy model 11 and convert them into digital signals and transmit them to the cloud server platform 20. The remote-side device 30 includes an MR helmet 31 and a handle 32. The handle 32 can calibrate the markers on the body surface of the subject 40 to generate a virtual dynamic three-dimensional human body model. The MR helmet 31 is equipped with a display screen and a spatial perception module. The MR helmet 31 can receive data from the cloud server platform 20 and superimpose the actions of the virtual probe on the real body of the subject 40 in real time.

[0042] The dummy model 11 can be made of silicone and can simulate the body structure of the subject 40.

[0043] The ultrasonic probe model 12 can be made by 3D printing and has a similar shape and size to the actual ultrasonic probe 50.

[0044] The cloud server platform 20 includes a data transmission module 21 and a user management module 22. The data transmission module 21 is communicatively connected to the motion tracker 13 and the MR helmet 31. The user management module 22 enables users to access the system according to their permissions. The users include an expert 60 and a remote examiner 70.

[0045] Embodiment 2

[0046] As Figures 1 - 5 shown, this embodiment provides a method for using a remote ultrasonic device synchronous display system based on mixed reality, including the following steps:

[0047] Step 1: The expert 60 operates on the dummy model 11 using the ultrasonic probe model 12 to simulate a real ultrasonic examination process;

[0048] Step 2: The motion tracker 13 captures the movement trajectory and angle change of the ultrasonic probe model 12 in real time and converts them into digital signals for transmission to the cloud server platform 20;

[0049] Step 3: The examiner 70 collects the physical information of the subject 40;

[0050] Step 4: Use a physical positioning clip and manual calibration to construct a virtual coordinate system to achieve dynamic alignment between the virtual and real spaces;

[0051] Step 5: The examiner 70 wears the MR helmet 31, and the MR helmet 31 receives data from the cloud server platform 20 and superimposes the actions of the virtual probe on the real body of the subject 40 in real time;

[0052] Step 3 specifically includes:

[0053] Step 3.1: The examiner 70 wears the MR helmet 31 and uses the handle 32 to calibrate the markers on the surface of the subject 40;

[0054] Step 3.2: The system generates a virtual dynamic three-dimensional human model of the subject 40 based on the markers and adjusts the size of the virtual dynamic three-dimensional human model through an equal ratio scaling algorithm to adapt to the actual body shape of the subject 40;

[0055] Step 4 specifically includes:

[0056] Step 4.1: Set the physical positioning clip, combine with manual calibration of the markers on the surface of the subject 40 to construct a virtual coordinate system;

[0057] Step 4.2: The system realizes the precise mapping between the virtual space and the real physical space through the physical positioning clip and the manually calibrated points;

[0058] In step 5, the display screen of the MR helmet 31 only shows the actions of the virtual probe to avoid visual interference caused by the complete virtual dynamic three-dimensional human model;

[0059] The usage method of the remote ultrasonic device synchronous display system may further include:

[0060] Step 6: The system monitors the position of the virtual probe in real time through a boundary detection algorithm;

[0061] Step 7: If it is detected that the virtual probe is approaching the boundary, the system automatically adjusts the position of the probe.

[0062] The following details the system architecture, hardware configuration, and implementation steps:

[0063] I. System Architecture and Hardware Configuration

[0064] The expert-side device 10 is configured with:

[0065] Silicone dummy model 11: Simulates the body structure of the subject 40, especially the abdominal area, to facilitate the expert 60 to perform ultrasonic operations.

[0066] 3D printed ultrasonic probe model 12: Has a similar shape and size to the actual ultrasonic probe 50, facilitating the expert 60 to perform real operation exercises.

[0067] Motion tracker 13: Installed on the ultrasonic probe model 12, used to capture the operation actions of the expert 60 and convert them into digital signals for transmission to the cloud server platform 20.

[0068] The remote-side device 30 is configured with:

[0069] MR helmet 31: Equipped with a high-definition display screen and a spatial perception module, capable of real-time displaying the actions of the virtual probe and superimposing the virtual image on the actual situation of the subject 40.

[0070] Handle 32: Used to calibrate the non-sensitive markers on the body surface of the subject 40 (such as the navel, lower edge of the rib), helping to generate a virtual dynamic three-dimensional human model.

[0071] The cloud server platform 20 is configured with:

[0072] Low-latency data transmission module 21: Ensures the data transmission speed and stability between the expert side and the remote side, supporting local area network or user self-deployment.

[0073] User management module 22: Allows the examiner 70 and the expert 60 to access the system according to their permissions, ensuring the security and privacy of the data.

[0074] II. Implementation Steps

[0075] Step 1: Expert-side operation

[0076] Expert 60 operates on the silicone dummy model 11 using the 3D printed ultrasound probe model 12 to simulate the real ultrasound examination process.

[0077] The motion tracker 13 captures the movement trajectory and angle changes of the ultrasound probe model 12 in real time and uploads this data to the cloud server platform 20.

[0078] Step Two: Collection of the subject's 40 body information

[0079] The examiner 70 wears the MR helmet 31 and uses the handle 32 to calibrate the non-sensitive surface markers (such as the navel and the lower edge of the rib cage) of the subject 40.

[0080] The system generates a dynamic three-dimensional abdominal model of the subject 40 based on these markers and adjusts the model size through an equal ratio scaling algorithm to adapt to the actual body shape of the subject 40.

[0081] Step Three: Alignment of virtual and real coordinates

[0082] A physical positioning clamp is set near the ultrasound examination bed, and combined with manually calibrating the fixed markers (such as the navel) on the surface of the subject 40, a virtual coordinate system is constructed.

[0083] The system realizes the precise mapping between the virtual space and the real physical space through the physical positioning clamp and the manually calibrated points, with the error controlled within 2 mm.

[0084] Step Four: Remote display

[0085] The MR helmet 31 worn by the examiner 70 receives the data from the cloud server platform 20, and the actions of the virtual probe are superimposed on the real body of the subject 40 in real time.

[0086] To simplify the interface, only the actions of the virtual probe are displayed on the remote end to avoid visual interference caused by the complete virtual human model.

[0087] Step Five: Anti-penetration detection

[0088] The system monitors the position of the virtual probe in real time through a boundary detection algorithm to prevent it from penetrating the virtual human model.

[0089] If it is detected that the virtual probe is approaching the boundary, the system will automatically adjust the position of the probe to ensure the safety and accuracy of the operation.

[0090] III. Technical effects

[0091] Improve operation accuracy and consistency: Through the dynamic coordinate anchoring mechanism, ensure the high-precision alignment between the virtual space and the real physical space, reduce operation errors, and improve the accuracy of the examination.

[0092] Reduce the operation fatigue of experts: Experts do not need to wear helmets and can operate the silicone dummy and probe model with naked eyes, increasing the freedom and comfort of operation.

[0093] Ensure the privacy and security of the subjects: When collecting the body information of the subjects, the system avoids sensitive areas and further ensures privacy protection through an anti-penetration mechanism.

[0094] Support multi-position operation: In the future, it can be extended to multiple positions such as side lying and sitting position to adapt to more clinical application scenarios.

[0095] AI-assisted function: To further improve the intelligence level of the system, the AI-assisted inspection function can be integrated in the future to help the inspector complete the inspection more efficiently.

[0096] Through the above embodiments, the actual application process of the remote ultrasound device synchronous display system based on mixed reality is demonstrated, verifying its effectiveness in improving the accuracy and operation efficiency of remote ultrasound inspection.

[0097] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations based on the concept of the present invention without creative efforts. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field of the present invention through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the protection scope determined by the claims.

Claims

1. A synchronous display system for remote ultrasound devices based on mixed reality, characterized in that, It includes an expert-side device, a cloud server platform, and a remote-side device. The expert-side device includes a dummy model, an ultrasound probe model, and an action tracker. The action tracker is installed on the ultrasound probe model and is configured to capture the movement trajectory and angle change of the ultrasound probe model on the dummy model and convert them into digital signals for transmission to the cloud server platform. The remote-side device includes an MR helmet and a handle. The handle is configured to calibrate the markers on the subject's body surface to generate a virtual dynamic three-dimensional human body model. The MR helmet is equipped with a display screen and a spatial perception module and is configured to receive data from the cloud server platform and superimpose the actions of the virtual probe on the subject's real body in real time.

2. The synchronous display system for remote ultrasound devices based on mixed reality according to claim 1, characterized in that The dummy model is made of silicone and is configured to simulate the body structure of the subject.

3. The synchronous display system for remote ultrasound devices based on mixed reality according to claim 1, wherein The ultrasound probe model is made by 3D printing and has a similar shape and size to an actual ultrasound probe.

4. The remote ultrasonic device synchronous display system based on mixed reality according to claim 1, wherein The cloud server platform includes a data transmission module, and the data transmission module is communicatively connected to the action tracker and the MR helmet.

5. The synchronous display system of the remote ultrasonic device based on mixed reality according to claim 4, characterized in that, The cloud server platform further includes a user management module, and the user management module is configured to allow users to access the system according to their permissions.

6. A method for using a synchronous display system of a remote ultrasonic device based on mixed reality according to any one of claims 1 to 5, characterized in that, It includes the following steps: Step 1: The expert operates on the dummy model using the ultrasound probe model to simulate a real ultrasound examination process. Step 2: The action tracker captures the movement trajectory and angle change of the ultrasound probe model in real time and converts them into digital signals for transmission to the cloud server platform. Step 3: Collect the body information of the subject. Step 4: Use physical positioning clips and manual calibration to construct a virtual coordinate system to achieve dynamic alignment between the virtual and real spaces. Step 5: The examiner wears the MR helmet, and the MR helmet receives data from the cloud server platform and superimposes the actions of the virtual probe on the subject's real body in real time.

7. The method of using the synchronous display system for remote ultrasonic devices based on mixed reality according to claim 6, characterized in that, The specific steps of Step 3 include the following steps: Step 3.1: The examiner wears the MR helmet and uses the handle to calibrate the markers on the subject's body surface. Step 3.2: The system generates a virtual dynamic three-dimensional human body model of the subject based on the markers and adjusts the size of the virtual dynamic three-dimensional human body model through an equal ratio scaling algorithm to adapt to the actual body shape of the subject.

8. The method of using the synchronous display system of the remote ultrasound device based on mixed reality according to claim 7, characterized in that, The specific steps of Step 4 include the following steps: Step 4.1: Set physical positioning clips and combine with manual calibration of the markers on the subject's body surface to construct a virtual coordinate system. Step 4.2: The system realizes the precise mapping between the virtual space and the real physical space through the physical positioning clips and the manually calibrated points.

9. The method of using the synchronous display system of the remote ultrasound device based on mixed reality according to claim 8, characterized in that, In Step 5, the display screen of the MR helmet only shows the actions of the virtual probe to avoid visual interference caused by the complete virtual dynamic three-dimensional human body model.

10. The method of using the synchronous display system of the remote ultrasound device based on mixed reality according to claim 8, characterized in that, It further includes the following steps: Step 6: The system real-time monitors the position of the virtual probe through a boundary detection algorithm. Step 7: If it is detected that the virtual probe is approaching the boundary, the system automatically adjusts the position of the probe.

Images