Wireless ultrasonic system of mixed reality technology

Abstract

The invention discloses a wireless ultrasonic system of a mixed reality technology, which relates to the technical field of medical instruments and comprises a wireless ultrasonic probe module, a mixed reality glasses module and a mobile terminal module. The mixed reality glasses module comprises a wireless transceiving module, a display module, a sensor module, an interactive input module, a data processing module, a storage module and a power supply module; the wireless transceiving module is used for receiving image information sent by the wireless ultrasonic probe module and sending processed data to the mobile terminal module; the display module comprises an optical component and a micro display, and the optical component comprises a lens, a reflector and an optical waveguide and is used for projecting a digital image into the visual field of a user. According to the mixed reality technology, the ultrasonic image can be overlaid into the sight line of a doctor in real time, so that the doctor can see the ultrasonic image more visually, and the diagnosis accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a wireless ultrasound system using mixed reality technology. Background Art

[0002] Ultrasonic waves have good directivity. When propagating in the human body, they encounter tissues and organs of different densities, which results in reflection, refraction and absorption. The distance, strength and number of echoes displayed on the oscilloscope screen, as well as whether the attenuation is obvious, can show the activity functions of certain organs in the body and accurately identify whether the tissues and organs contain liquid or gas, or are substantial tissues.

[0003] Traditional ultrasound examinations usually require doctors to manually operate a wired ultrasound probe and observe images on a dedicated display. In this mode, doctors often need to switch their line of sight between viewing the patient and viewing the display screen, which may result in information loss or inconvenience in operation. In recent years, with the development of wireless communication technology and mixed reality technology, wireless ultrasound equipment and MR-assisted diagnosis systems have emerged, but the existing technology has shortcomings in image real-time, positioning accuracy and interactive experience, so there is room for improvement. Summary of the invention

[0004] The purpose of the present invention is to solve the shortcomings of the prior art and propose a wireless ultrasound system with mixed reality technology. Its advantage is that mixed reality technology can superimpose ultrasound images on the doctor's line of sight in real time, allowing the doctor to see more intuitively and improve the accuracy of diagnosis.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A wireless ultrasound system using mixed reality technology, comprising a wireless ultrasound probe module, a mixed reality glasses module and a mobile terminal module;

[0007] The mixed reality glasses module includes a wireless transceiver module, a display module, a sensor module, an interactive input module, a data processing module, a storage module and a power module. The wireless transceiver module is used to receive image information sent by the wireless ultrasound probe module and send the processed data to the mobile terminal module; the display module includes optical components and a micro display, and the optical components include lenses, reflectors and optical waveguides for projecting digital images into the user's field of view; the micro display adopts LCoS reflective liquid crystal display technology to display high-resolution images;

[0008] The LCoS reflective liquid crystal display can fill liquid crystal molecules between the upper glass substrate and the lower metal reflective layer. The voltage between the metal reflective layer and the top ITO common electrode jointly determines the light flux performance of the liquid crystal molecules and displays different pixel grayscales. The display driving circuit is directly prepared on the silicon substrate.

[0009] The optical components use near-eye 3D diffraction display technology, combined with two light-guiding transparent holographic lenses to achieve mixed reality; virtual objects use digital light processing projection technology, and are projected from the micro-projector on the device through the holographic lens into the human eye, while the light source information in the real environment can also be received by the human eye; the light source irradiates the required image information onto the DMD chip through the optical components, and forms the required image through the high-speed switching function of the micro-mirror array on the DMD chip, and then transmits the light to the user's eyes through the optical components to form an image on the retina of the human eye.

[0010] The present invention is further configured such that the wireless ultrasound probe module includes a WI-FI module, an ultrasound acquisition module and a micro positioning sensor, wherein the WI-FI module is used for high-speed and stable transmission of image data, and the micro positioning sensor is used for real-time tracking of the probe position and posture.

[0011] The present invention is further configured such that the core of the digital light processing projection technology is a digital micromirror component, which is a micro-electromechanical device of a spatial light modulator. The surface of the chip contains hundreds of thousands to millions of highly reflective micromirror arrays, which can be switched at an extremely high speed to mix the three RGB primary colors into the desired projected image.

[0012] The present invention is further configured such that the interactive input module includes a gaze ray unit, a gesture control unit and a voice control unit.

[0013] The present invention is further configured such that the gaze ray unit is similar to a mouse in computer operation, except that in the mixed reality glasses, the movement of the cursor is controlled by rotating the head, and the position of the cursor represents the holographic target object annotated by the current user; by selecting the virtual object with the cursor, in conjunction with the other two interaction methods, the control and operation of the holographic object is achieved.

[0014] The present invention is further configured such that the gesture control unit is similar to a mouse click event in a computer operation, and the mixed reality glasses support three gesture controls, namely discrete gestures, grasping and opening. Through these three gesture controls, the placement of virtual objects, dragging of interfaces and switching of main interfaces are realized.

[0015] The present invention is further configured such that the voice control unit recognizes the operator's voice through a four-microphone array integrated on the mixed reality glasses, and also realizes object operation corresponding to the voice through script programming control.

[0016] The present invention is further configured such that the data processing module adopts a convolutional neural network as a mixed reality glasses module algorithm; the forward calculation process of implementing the convolution layer on the GPU mainly includes three calculation processes: expanding the convolution into a matrix, calculating the input weights and the input image, and adding a bias.

[0017] The present invention is further configured such that the mobile terminal module serves as a user interface, allowing doctors or patients to view ultrasound images, diagnostic reports and operation guides through a smart phone or tablet computer, while supporting remote consultation and data sharing.

[0018] The beneficial effects of the present invention are:

[0019] 1. The wireless design allows doctors to move freely during examination without being restricted by cables, which improves the convenience and efficiency of operation;

[0020] 2. Mixed reality technology can superimpose ultrasound images into the doctor's field of vision in real time, allowing the doctor to see more intuitively and improve the accuracy of diagnosis;

[0021] 3. Telemedicine capabilities are enhanced, and doctors can remotely control ultrasound probes and receive real-time images through wireless connections. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the structure of the components of a wireless ultrasound system using mixed reality technology proposed by the present invention;

[0023] Figure 2 This is a schematic diagram of the overall structure of a wireless ultrasound system using mixed reality technology proposed by the present invention;

[0024] Figure 3 A schematic diagram of a mixed reality glasses module of a wireless ultrasound system of mixed reality technology proposed in the present invention. DETAILED DESCRIPTION

[0025] The technical solution of this patent is further described in detail below in conjunction with specific implementation methods.

[0026] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0027] Reference Figure 1-3 , a wireless ultrasound system of mixed reality technology, including a wireless ultrasound probe module, a mixed reality glasses module and a mobile terminal module;

[0028] The wireless ultrasound probe module includes a WI-FI module, an ultrasound acquisition module and a micro positioning sensor. The WI-FI module is used for high-speed and stable transmission of image data, and the micro positioning sensor is used for real-time tracking of the probe position and posture.

[0029] The mixed reality glasses module includes a wireless transceiver module, a display module, a sensor module, an interactive input module, a data processing module, a storage module and a power module. The wireless transceiver module is used to receive image information sent by the wireless ultrasound probe module and send the processed data to the mobile terminal module; the display module includes optical components and a microdisplay. The optical components include lenses, reflectors and optical waveguides, which are used to project digital images into the user's field of view; the microdisplay adopts LCoS reflective liquid crystal display technology to display high-resolution images.

[0030] LCoS reflective liquid crystal display can fill liquid crystal molecules between the upper glass substrate and the lower metal reflective layer. The voltage between the metal reflective layer and the top ITO common electrode jointly determines the light transmission performance of the liquid crystal molecules and shows different pixel grayscales. The display drive circuit is directly prepared on the silicon substrate. In order to prevent the incident light from irradiating the transistor inside the silicon substrate to form photogenerated carriers and affect the performance of the drive circuit, a metal light shielding layer is usually added between the circuit routing layer and the metal reflective layer to achieve the purpose of shielding the incident light. The guide layer can determine the orderly arrangement of the liquid crystal molecules. The support pad is polished by chemical mechanical grinding to ensure its high consistency, and its height is determined by the thickness of the selected liquid crystal box.

[0031] The display principle of LCoS is as follows: when the external voltage of the pixel in the liquid crystal layer is zero, the incident S polarized light passes through the liquid crystal layer, and its polarization direction does not twist. When it reaches the bottom metal reflective layer and reflects back, it is still S polarized light and passes through the liquid crystal layer. It is then reflected back to the original light path through the PBS prism. In this case, the light does not enter the projection light path and there is no light output, that is, this pixel is in a "dark state". On the contrary, when there is an external voltage applied to the pixel, the polarization direction of the incident S polarized light will be polarized when passing through the liquid crystal layer. When it is reflected by the metal reflective layer and then passes through the liquid crystal layer, it will become P polarized light. This beam of P polarized light will enter the projection light path after passing through the PBS prism and display an image on the screen, that is, it will appear in a "bright state". The magnitude of the voltage applied to the two ends of the pixel will affect the light flux performance of the liquid crystal molecules, and thus determine the display grayscale of the pixel.

[0032] Optical components use near-eye 3D diffraction display technology, combined with two light-guiding transparent holographic lenses to achieve mixed reality; virtual objects use digital light processing projection technology, and are projected from the micro-projector on the device through the holographic lens into the human eye. At the same time, the light source information in the real environment can also be received by the human eye; the light source irradiates the required image information onto the DMD chip through optical components, and forms the required image through the high-speed switching function of the micromirror array on the DMD chip. The light is then transmitted to the user's eyes through optical components to form an image on the retina of the human eye; the core of digital light processing projection technology is digital micromirror components, which are micro-electromechanical devices of spatial light modulators. The surface of the chip contains hundreds of thousands to millions of high-reflection micromirror arrays, which can switch at extremely high speeds to mix the three RGB primary colors into the desired projected image.

[0033] In this embodiment, the interactive input module includes a gaze ray unit, a gesture control unit and a voice control unit.

[0034] The gaze ray unit is similar to the mouse in computer operation. The difference is that in the mixed reality glasses, the cursor movement is controlled by turning the head. The position of the cursor represents the holographic target object annotated by the current user; the virtual object is selected by the cursor, and the other two interaction methods are coordinated to realize the control and operation of the holographic object; the gesture control unit is similar to the mouse click event in computer operation. The mixed reality glasses support three types of gesture control: discrete gestures, grasping and opening. Through these three gesture controls, the placement of virtual objects, dragging of the interface and switching of the main interface are realized; the voice control unit recognizes the operator's voice through the 4 microphone arrays integrated on the mixed reality glasses, and realizes the object operation of the corresponding voice through script programming control; the application of gestures to realize the operation of virtual objects replaces the existence of the mouse in traditional computers, making the operation of the device more convenient.

[0035] The data processing module uses a convolutional neural network as the mixed reality glasses module algorithm; the forward calculation process of implementing the convolution layer on the GPU mainly includes three calculation processes: expanding the convolution into a matrix, calculating the input weights and the input image, and adding a bias.

[0036] The parallelization on GPU is implemented as follows:

[0037] Step 1: Use GPU to expand the input feature map and convolution kernel. Assuming that the size of the input feature map matrix input_data is n×n, the size of the convolution kernel is k×k, and there are m convolution kernels, the size of the convolution kernel matrix weight_data is k×k×m. According to the convolution expansion method, the number of rows of the input feature map expansion matrix input_col is the size of the convolution result feature map (n-k+1)×(n-k+1) (assuming the convolution step is 1), and the number of columns is the size of the convolution kernel k×k, then the size of input_col is (n-k+1)×(n-k+1)×k×k. The input feature map matrix input_data and its expansion matrix input_col are both stored on the GPU in the form of one-dimensional arrays. Therefore, to expand the input feature map on the GPU, it is only necessary to copy and construct the matrix elements through threads. The total number of threads num is set to (n-k+1)×(n-k+1)×k×k, where each thread is responsible for generating an element in input_col. In order to balance the memory and thread usage, the present invention opens 512 threads in each thread block, so num / 512 blocks need to be set.

[0038] The number of rows of the expanded convolution kernel matrix weight_col is k×k, and the number of columns is m, so the size of weight_col is k×k×m. The convolution kernel matrix weight_data and its expanded matrix weight_col are also stored on the GPU in the form of a one-dimensional array. Therefore, to expand the convolution kernel on the GPU, it is only necessary to copy and construct the matrix elements through threads. For the expansion of the convolution kernel, the total number of threads num is set to k×k×m, and each thread is responsible for generating an element in weight_col. The number of blocks is set to num / 512, and 512 threads are enabled in each block.

[0039] Step 2: Calculate the input weights and add bias on the GPU. After expanding the input feature map and convolution kernel to obtain the input_col and weight_col matrices respectively, calculate the input weights and directly perform matrix multiplication operations on input_col and weight_col. In order to improve the efficiency of matrix operations, CUDA specially designed the cuBLAS library. The cuBLAS library used in the present invention is used to process matrix multiplication using the cublasSgemm function in the cuBLAS library. The operation implemented in the cublasSgemm function is C=α*A*B+β*C. In the calculation of the input weights and, the A matrix is ​​input_col, the B matrix is ​​weight_col, α and β are set to 1 and 0 respectively, and the C matrix is ​​used to save the input weights and, and its size is (n-k+1)×(n-k+1)×m.

[0040] The cublasSgemm function is also used in the calculation of input weights and added bias on the GPU. Both α and β are set to 1. The A matrix is ​​a row vector consisting of (n-k+1)×(n-k+1) bias coefficients, and the B matrix is ​​a column vector consisting of m Bias values, thereby realizing the operation of adding bias to the C matrix.

[0041] Step 3: Calculate the input layer gradient input_diff. The input layer gradient input_diff can be calculated from the weight parameter weight_data and the output layer gradient output_diff. Assuming that the size of the input feature map, the size of the convolution kernel, and the number of convolution kernels are the same as the forward calculation process, the size of the unfolded matrix weight_col corresponding to weight_data is m×k×k, and the size of the output layer gradient output_diff is (n-k+1)×(n-k+1)×m. Since each input is involved in the convolution of m convolution kernels, the calculation of each input gradient also involves the weights corresponding to m convolution kernels. The matrix weight_col and the matrix output_diff are both known. Therefore, by calling the cublasSgemm function in the cuBLAS library, the matrix multiplication between weight_col and output_diff is implemented to obtain the input layer gradient input_diff, and the size of the input layer gradient input_diff is (n-k+1)×(n-k+1)×k×k. Where (n-k+1)×(n-k+1) represents the number of times a convolution kernel of size k×k performs sliding convolution on an input feature map of size n×n.

[0042] It is worth mentioning that the mobile terminal module serves as a user interface, allowing doctors or patients to view ultrasound images, diagnostic reports and operation guides through smartphones or tablets, while supporting remote consultation and data sharing.

[0043] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A wireless ultrasound system using mixed reality technology, characterized in that: It includes a wireless ultrasound probe module, a mixed reality glasses module and a mobile terminal module; The mixed reality glasses module includes a wireless transceiver module, a display module, a sensor module, an interactive input module, a data processing module, a storage module and a power module. The wireless transceiver module is used to receive image information sent by the wireless ultrasound probe module and send the processed data to the mobile terminal module; the display module includes optical components and a micro display, and the optical components include lenses, reflectors and optical waveguides for projecting digital images into the user's field of view; The micro display uses LCoS reflective liquid crystal display technology to display high-resolution images; The LCoS reflective liquid crystal display can fill liquid crystal molecules between the upper glass substrate and the lower metal reflective layer. The voltage between the metal reflective layer and the top ITO common electrode jointly determines the light flux performance of the liquid crystal molecules and displays different pixel grayscales. The display driving circuit is directly prepared on the silicon substrate. The optical components use near-eye 3D diffraction display technology, combined with two light-guiding transparent holographic lenses to achieve mixed reality; virtual objects use digital light processing projection technology, and are projected from the micro-projector on the device through the holographic lens into the human eye, while the light source information in the real environment can also be received by the human eye; the light source irradiates the required image information onto the DMD chip through the optical components, and forms the required image through the high-speed switching function of the micro-mirror array on the DMD chip, and then transmits the light to the user's eyes through the optical components to form an image on the retina of the human eye.

2. The wireless ultrasound system of mixed reality technology according to claim 1, characterized in that: The wireless ultrasound probe module includes a WI-FI module, an ultrasound acquisition module and a micro positioning sensor. The WI-FI module is used for high-speed and stable transmission of image data, and the micro positioning sensor is used for real-time tracking of the probe position and posture.

3. The wireless ultrasound system of mixed reality technology according to claim 2, characterized in that: The core of the digital light processing projection technology is the digital micromirror component, which is a micro-electromechanical device of a spatial light modulator. The surface of the chip contains hundreds of thousands to millions of highly reflective micromirror arrays. These micromirror arrays can be switched at extremely high speeds to mix the three RGB primary colors into the desired projected image.

4. The wireless ultrasound system of mixed reality technology according to claim 1, characterized in that: The interactive input module includes a gaze ray unit, a gesture control unit and a voice control unit.

5. The wireless ultrasound system of mixed reality technology according to claim 4, characterized in that: The gaze ray unit is similar to a mouse in computer operation, except that the cursor movement is controlled by rotating the head in the mixed reality glasses, and the position of the cursor represents the holographic target object annotated by the current user; by selecting the virtual object with the cursor and cooperating with the other two interaction methods, the control and operation of the holographic object can be achieved.

6. The wireless ultrasound system of mixed reality technology according to claim 5, characterized in that: The gesture control unit is similar to the mouse click event in computer operation. The mixed reality glasses support three types of gesture control: discrete gestures, grasping and opening. Through these three gesture controls, the placement of virtual objects, dragging of interfaces and switching of main interfaces are realized.

7. The wireless ultrasound system of mixed reality technology according to claim 6, characterized in that: The voice control unit recognizes the operator's voice through the four microphone arrays integrated on the mixed reality glasses, and also realizes object operation corresponding to the voice through script programming control.

8. The wireless ultrasound system of mixed reality technology according to claim 1, characterized in that: The data processing module adopts a convolutional neural network as the mixed reality glasses module algorithm; the forward calculation process of implementing the convolution layer on the GPU mainly includes three calculation processes: expanding the convolution into a matrix, calculating the input weights and the input image, and adding a bias.

9. A wireless ultrasound system using mixed reality technology according to any one of claims 1 to 8, characterized in that: The mobile terminal module serves as a user interface, allowing doctors or patients to view ultrasound images, diagnostic reports and operation guides through a smart phone or tablet computer, while supporting remote consultation and data sharing.

Images