Medical image three-dimensional visual imaging system and voice control system

Abstract

The invention discloses a medical image three-dimensional visual imaging system and a voice control system, and relates to the technical field of medical image.The medical image three-dimensional visual imaging system comprises a data acquisition module used for acquiring two-dimensional slice image data including a coronal plane, a sagittal plane and a cross section through computed tomography or magnetic resonance imaging; the segmentation identification module is used for setting a segmentation algorithm to segment each tissue structure of organs, tumors, blood vessels and nerves in the focus area; the three-dimensional reconstruction module is used for carrying out voxel filling on each slice and stacking the two-dimensional slices into a three-dimensional volume; and the holographic image module is used for calculating phase information of the surface of the object by simulating interference of light waves. According to the system, the visualization effect and the operation convenience of the medical image are improved through the efficient three-dimensional reconstruction and voice interaction technology, so that the diagnosis capability and the working efficiency of a doctor are improved, and the development and the application of the system bring a new breakthrough to the field of medical images and promote the progress of clinical practice.

Description

Technical Field

[0001] The present invention relates to the field of medical imaging technology, and in particular to a medical imaging three-dimensional visualization imaging system and a voice control system. Background Art

[0002] With the rapid development of medical imaging technology, medical imaging plays an increasingly important role in disease diagnosis, treatment planning and postoperative evaluation. Traditional medical imaging mainly includes X-rays, CT (computed tomography), MRI (magnetic resonance imaging), etc. These images can provide important information about the internal structure of the human body. However, with the increase in data volume and the increase in image complexity, doctors face huge challenges in interpreting and analyzing these images.

[0003] Currently, many medical imaging analyses rely on two-dimensional images, and doctors usually need to view multiple slices to obtain information about three-dimensional structures. This method is not only time-consuming, but also prone to information omissions and misjudgments;

[0004] In addition, in modern medical environments, medical staff often need to manually operate computers to adjust and analyze images when viewing images. This traditional operation method not only reduces work efficiency, but also may distract doctors from the image content and affect their judgment. Summary of the invention

[0005] The purpose of the present invention is to provide a medical image three-dimensional visualization imaging system and a voice control system to solve the problems raised in the above background technology.

[0006] To solve the above problems, the technical solutions adopted by the present invention are as follows:

[0007] Medical imaging 3D visualization system, including

[0008] A data acquisition module, which is used to acquire two-dimensional slice image data including coronal plane, sagittal plane and cross-section through computer tomography or magnetic resonance imaging;

[0009] The segmentation and identification module is used to set the segmentation algorithm to segment the organs, tumors, blood vessels, nerves and other tissue structures in the lesion area, and further identify the tissue structures;

[0010] The 3D reconstruction module is used to fill each slice with voxels, stack the 2D slices into a 3D volume, extract isosurfaces from the 3D volume data, further convert it into a 3D mesh, and generate triangles based on the values ​​of eight adjacent voxels. After that, the volume data is directly rendered into a 3D image based on volume rendering and post-processed, and the generated model is displayed by a 3D visualization tool;

[0011] The holographic imaging module is used to calculate the phase information of the object surface by simulating the interference of light waves, and further generate a hologram using the interference principle.

[0012] Preferably, the segmentation algorithm calculation steps are as follows:

[0013] S1, calculate the histogram of the image;

[0014] S2, calculate the probability of total pixels and each gray level of the image;

[0015] S3. For each possible threshold t, calculate the inter-class variance Find the between-class variance The maximum threshold t, after determining the optimal threshold t, the image is segmented;

[0016]

[0017] Among them, ω1(t) and ω0(t) are the pixel probabilities greater than and less than the threshold t, respectively, and μ0(t) and μ1(t) are the means less than and greater than the threshold t, respectively.

[0018] Preferably, the identification of the tissue structure is performed by assigning a unique color to each different tissue structure.

[0019] Preferably, the voxel calculation formula is as follows:

[0020] V(x,y,z)=I(i,j)

[0021] Among them, I(i,j) is the pixel value of the slice, x, y are the position of the slice, and z is the index of the slice.

[0022] Preferably, the volume rendering adopts a ray projection method, and the ray projection formula is:

[0023]

[0024] Among them, S(t) is the color of the light in the volume data, T(t) is the transmission function, t0 and t1 are the starting and ending positions of the light;

[0025] The post-processing adopts surface reconstruction to smooth and optimize the generated three-dimensional model. The reconstruction formula is:

[0026]

[0027] Where f is the input normal field and φ is the potential function to be solved.

[0028] Preferably, the phase information φ(x, y) of the surface of the object can be calculated by the height function h(x, y) of the object:

[0029] φ(x,y)=kh(x,y)

[0030] Where h(x,y) is the height of the object on the (x,y) plane.

[0031] Preferably, the steps of generating the hologram are as follows:

[0032] S11. The interference pattern is calculated through the phase information of the hologram. The composite form of the reference wave and the object wave is:

[0033] H(x,y)=E ref (x,y)+E obj (x,y)

[0034] Among them, E ref is the reference wave, E obj It is the object wave;

[0035] S12. Calculate the phase of the hologram. The phase of the hologram H(x, y) can be expressed as:

[0036] H(x,y)=|E ref (x,y)| 2 +|E obj (x,y)| 2 +E ref (x,y)E obj (x,y);

[0037] S13. The final hologram can be encoded using phase information and amplitude information. The phase information is represented by a discrete sine function:

[0038]

[0039] S14, storing the hologram on a camera sensor and an optical element, and reproducing it through a corresponding optical device.

[0040] Voice control system, including

[0041] A speech recognition module is used to receive the doctor's voice instructions and establish a speech recognition library, analyze the speech signal by converting it into text and further recognize the instructions;

[0042] The speech recognition module uses a speech recognition API to perform real-time speech recognition, parses commands using NLP tools, and uses a vtk medical 3D rendering engine to process the display and interaction of 3D models;

[0043] A model control module, which is used to control the three-dimensional model according to the parsed instructions, and to perform rotation, movement, scaling, display, hiding, transparency adjustment and restoration operations on the three-dimensional model;

[0044] The feedback module is used to implement the command feedback mechanism, confirm whether the operation is successful, and provide corresponding visual feedback.

[0045] Preferably, the voice recognition module includes a microphone, and the voice recognition library instructions include but are not limited to "turn up", "turn down", "turn left", "turn right", "move up", "move down", "move forward", "move backward", "zoom in", "zoom out", "show tissue", "hide tissue", "transparency N percent", "close tissue" and "restore".

[0046] Preferably, the model control module implements the transformation of the model by means of matrix transformation through the instructions parsed;

[0047] Among them, the mobile transformation matrix is:

[0048] d is the distance moved;

[0049] The rotation transformation matrix is:

[0050] θ is the rotation angle;

[0051] The scaling transformation matrix is:

[0052] s x ,s y ,s z is the scaling factor.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The present invention improves the visualization effect and operational convenience of medical images through efficient three-dimensional reconstruction and voice interaction technology, thereby enhancing doctors' diagnostic capabilities and work efficiency. The development and application of this system will bring new breakthroughs in the field of medical imaging and promote the progress of clinical practice.

[0055] This invention not only improves the diagnostic accuracy and operational convenience of medical images, but also brings significant progress to the medical industry by improving doctors' work efficiency and promoting multidisciplinary collaboration. This innovation will help improve patients' treatment experience and medical outcomes and promote the development of medical imaging technology.

[0056] The present invention converts two-dimensional image data into a three-dimensional model and performs holographic projection, allowing doctors to have a deeper understanding of the spatial relationship of lesions and reduce misjudgments. In addition, doctors can observe images from different angles to help identify complex lesions. Three-dimensional visualization can display anatomical structures more clearly, especially when dealing with tumors or other lesions, helping to better assess their size, shape and position.

[0057] Doctors of the present invention can directly control the imaging system through voice commands, reducing dependence on the mouse and keyboard, which is particularly important in sterile environments such as operating rooms and can avoid the risk of cross infection. The voice control system can provide instant response, and doctors do not need to interrupt operations when viewing and analyzing images, thereby improving work efficiency.

[0058] Through integrated voice control, the present invention allows doctors to quickly call up required imaging data, reduce tedious manual operations, and improve overall diagnosis and treatment efficiency. The system design takes user experience into consideration, integrates voice library design, and provides concise and clear voice instructions, which reduces the difficulty for doctors to adapt to new technologies and shortens training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is an overall schematic block diagram of the system of the present invention;

[0060] Figure 2 This is the system block diagram of the medical image 3D visualization imaging system;

[0061] Figure 3 This is the system block diagram of the voice control system. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] Embodiment 1:

[0064] See also Figure 1 and Figure 2 As shown, the present invention is a medical image three-dimensional visualization imaging system, comprising

[0065] A data acquisition module, which is used to acquire two-dimensional slice image data including coronal plane, sagittal plane and cross-section through computer tomography or magnetic resonance imaging;

[0066] The segmentation and identification module is used to set the segmentation algorithm to segment the organs, tumors, blood vessels, nerves and other tissue structures in the lesion area, and further identify the tissue structures;

[0067] The 3D reconstruction module is used to fill each slice with voxels, stack the 2D slices into a 3D volume, extract isosurfaces from the 3D volume data, further convert it into a 3D mesh, and generate triangles based on the values ​​of eight adjacent voxels. After that, the volume data is directly rendered into a 3D image based on volume rendering and post-processed, and the generated model is displayed by a 3D visualization tool;

[0068] The holographic imaging module is used to calculate the phase information of the object surface by simulating the interference of light waves, and further generate a hologram using the interference principle.

[0069] The segmentation algorithm calculation steps are as follows:

[0070] S1, calculate the histogram of the image;

[0071] S2, calculate the probability of total pixels and each gray level of the image;

[0072] S3. For each possible threshold t, calculate the inter-class variance Find the between-class variance The maximum threshold t, after determining the optimal threshold t, the image is segmented;

[0073]

[0074] Among them, ω1(t) and ω0(t) are the pixel probabilities greater than and less than the threshold t, respectively, and μ0(t) and μ1(t) are the means less than and greater than the threshold t, respectively.

[0075] The organizational structure is identified by assigning a unique color to each different organizational structure.

[0076] The voxel calculation formula is as follows:

[0077] V(x,y,z)=I(i,j)

[0078] Among them, I(i,j) is the pixel value of the slice, x, y are the position of the slice, and z is the index of the slice.

[0079] The volume rendering adopts a ray projection method, and the ray projection formula is:

[0080]

[0081] Among them, S(t) is the color of the light in the volume data, T(t) is the transmission function, t0 and t1 are the starting and ending positions of the light;

[0082] The post-processing adopts surface reconstruction to smooth and optimize the generated three-dimensional model. The reconstruction formula is:

[0083]

[0084] Where f is the input normal field and φ is the potential function to be solved.

[0085] The phase information φ(x, y) of the object surface can be calculated by the height function h(x, y) of the object:

[0086] φ(x,y)=kh(x,y)

[0087] Where h(x,y) is the height of the object on the (x,y) plane.

[0088] The steps of generating the hologram are as follows:

[0089] S11. The interference pattern is calculated through the phase information of the hologram. The composite form of the reference wave and the object wave is:

[0090] H(x,y)=E ref (x,y)+E obj (x,y)

[0091] Among them, E ref is the reference wave, E obj It is the object wave;

[0092] S12. Calculate the phase of the hologram. The phase of the hologram H(x, y) can be expressed as:

[0093] H(x,y)=|E ref (x,y)| 2 +|E obj (x,y)| 2 +E ref (x,y)E obj (x,y);

[0094] S13. The final hologram can be encoded using phase information and amplitude information. The phase information is represented by a discrete sine function:

[0095]

[0096] S14, storing the hologram on a camera sensor and an optical element, and reproducing it through a corresponding optical device.

[0097] As can be seen from the above, after the above system processes and segments and marks the organs, tumors, blood vessels, nerves and other tissue structures in the lesion area, the two-dimensional slices are regenerated to facilitate the generation of a clearer and more intuitive three-dimensional model, which can combine the coronal, sagittal and cross-sectional images to form a holographic image for patient examination;

[0098] Three-dimensional visualization models can provide a clearer representation of anatomical structures, especially when dealing with tumors or other lesions, helping to better assess their size, shape, and location.

[0099] Embodiment 2:

[0100] See also Figure 1 and Figure 3 As shown, the voice control system includes

[0101] A speech recognition module is used to receive the doctor's voice instructions and establish a speech recognition library, analyze the speech signal by converting it into text and further recognize the instructions;

[0102] The speech recognition module uses a speech recognition API to perform real-time speech recognition, parses commands using NLP tools, and uses a vtk medical 3D rendering engine to process the display and interaction of 3D models;

[0103] Speech recognition APIs include at least one of Google Cloud Speech-to-Text, Microsoft Azure SpeechService, IBM Watson Speech to Text, Amazon Transcribe, and Mozilla DeepSpeech. The speech recognition API supports multiple languages ​​for real-time speech recognition, has powerful background noise processing, and integrates speech recognition, synthesis, and translation functions. It supports custom models and multiple file formats, and can handle special terminology in the legal and medical industries.

[0104] The speech recognition algorithm includes an acoustic model algorithm, a language model algorithm and a decoding algorithm. The acoustic model algorithm is used to convert audio signals into phonemes, which includes a hidden Markov model (HMM) and a deep neural network (DNN). The language model algorithm is used to understand the probability relationship between words in a word sequence, using an n-gram model or a model based on a recurrent neural network (RNN). The decoding algorithm is used to find the optimal path, combining the outputs of the acoustic model and the language model.

[0105] A model control module, which is used to control the three-dimensional model according to the parsed instructions, rotate, move, scale, display, hide, adjust transparency and restore the three-dimensional model, and associate it with the holographic image model;

[0106] The feedback module is used to implement the command feedback mechanism, confirm whether the operation is successful, and provide corresponding visual feedback.

[0107] The voice recognition module includes a microphone, and the voice recognition library instructions include but are not limited to "turn up", "turn down", "turn left", "turn right", "move up", "move down", "move forward", "move backward", "zoom in", "zoom out", "show tissue", "hide tissue", "transparency N percent", "close tissue" and "restore".

[0108] The model control module implements the transformation of the model by means of matrix transformation through the instructions parsed;

[0109] Among them, the mobile transformation matrix is:

[0110] d is the distance moved;

[0111] The rotation transformation matrix is:

[0112] θ is the rotation angle;

[0113] The scaling transformation matrix is:

[0114] s x ,s y ,s z is the scaling factor.

[0115] Among them, the speech recognition library supports hot word customization, allowing users to preset specific vocabulary, including terms commonly used in surgery and specific commands that doctors may use, to ensure that the doctor's voice instructions can be accurately recognized during surgery.

[0116] From the above, it can be seen that in actual use, after the three-dimensional model is displayed in stereoscopic form through the platform, during the operation, the doctor can say commands, such as "move up", and the system will further convert the voice into text "move up" and recognize the command type "move up". The system then uses the translation transformation matrix to update the position of the three-dimensional model. After the system confirms that the transformed model has been successfully uploaded, the operation results can be fed back through voice or interface, such as: "Successfully enlarged", "Tissue has been hidden", etc., so that the doctor can confirm whether the operation is successful.

[0117] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0118] In the drawings of the embodiments disclosed in the present invention, only the structures related to the embodiments disclosed in the present invention are involved, and other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of the present invention can be combined with each other.

[0119] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A medical image three-dimensional visualization imaging system, characterized in that: include A data acquisition module, which is used to acquire two-dimensional slice image data including coronal plane, sagittal plane and cross-section through computer tomography or magnetic resonance imaging; The segmentation and identification module is used to set the segmentation algorithm to segment the organs, tumors, blood vessels, nerves and other tissue structures in the lesion area, and further identify the tissue structures; The 3D reconstruction module is used to fill each slice with voxels, stack the 2D slices into a 3D volume, extract isosurfaces from the 3D volume data, further convert it into a 3D mesh, and generate triangles based on the values ​​of eight adjacent voxels. After that, the volume data is directly rendered into a 3D image based on volume rendering and post-processed, and the generated model is displayed by a 3D visualization tool; The holographic imaging module is used to calculate the phase information of the object surface by simulating the interference of light waves, and further generate a hologram using the interference principle.

2. The medical image three-dimensional visualization imaging system according to claim 1, characterized in that: The segmentation algorithm calculation steps are as follows: S1, calculate the histogram of the image; S2, calculate the probability of total pixels and each gray level of the image; S3. For each possible threshold t, calculate the inter-class variance Find the between-class variance The maximum threshold t, after determining the optimal threshold t, the image is segmented; Among them, ω1(t) and ω0(t) are the pixel probabilities greater than and less than the threshold t, respectively, and μ0(t) and μ1(t) are the means less than and greater than the threshold t, respectively.

3. The medical image three-dimensional visualization imaging system according to claim 2, characterized in that: The organizational structure is identified by assigning a unique color to each different organizational structure.

4. The medical image three-dimensional visualization imaging system according to claim 1, characterized in that: The voxel calculation formula is as follows: V(x,y,z)=I(i,j) Among them, I(i,j) is the pixel value of the slice, x, y are the position of the slice, and z is the index of the slice.

5. The medical image three-dimensional visualization imaging system according to claim 1, characterized in that: The volume rendering adopts a ray projection method, and the ray projection formula is: Among them, S(t) is the color of the light in the volume data, T(t) is the transmission function, t0 and t1 are the starting and ending positions of the light; The post-processing adopts surface reconstruction to smooth and optimize the generated three-dimensional model. The reconstruction formula is: Where f is the input normal field and φ is the potential function to be solved.

6. The medical image three-dimensional visualization imaging system according to claim 1, characterized in that: The phase information φ(x, y) of the object surface can be calculated by the height function h(x, y) of the object: φ(x,y)=kh(x,y) Where h(x,y) is the height of the object on the (x,y) plane.

7. The medical image three-dimensional visualization imaging system according to claim 1, characterized in that: The steps of generating the hologram are as follows: S11. The interference pattern is calculated through the phase information of the hologram. The composite form of the reference wave and the object wave is: H(x,y)=E ref (x,y)+E obj (x,y) Among them, E ref is the reference wave, E obj It is an object wave; S12. Calculate the phase of the hologram. The phase of the hologram H(x, y) can be expressed as: H(x,y)=|E ref (x,y)| 2 +|E obj (x,y)| 2 +E ref (x,y)E obj (x,y); S13. The final hologram can be encoded using phase information and amplitude information. The phase information is represented by a discrete sine function: S14, storing the hologram on a camera sensor and an optical element, and reproducing it through a corresponding optical device.

8. Voice control system, characterized in that: include A speech recognition module is used to receive the doctor's voice instructions and establish a speech recognition library, analyze the speech signal by converting it into text and further recognize the instructions; The speech recognition module uses a speech recognition API to perform real-time speech recognition, parses commands using NLP tools, and uses a vtk medical 3D rendering engine to process the display and interaction of 3D models; A model control module, which is used to control the three-dimensional model according to the parsed instructions, and to perform rotation, movement, scaling, display, hiding, transparency adjustment and restoration operations on the three-dimensional model; The feedback module is used to implement the command feedback mechanism, confirm whether the operation is successful, and provide corresponding visual feedback.

9. The voice control system according to claim 8, characterized in that: The speech recognition module includes a microphone, and the speech recognition library instructions include but are not limited to "turn up", "turn down", "turn left", "turn right", "move up", "move down", "move forward", "move backward", "zoom in", "zoom out", "show tissue", "hide tissue", "transparency N percent", "close tissue" and "restore".

10. The voice control system according to claim 8, characterized in that: The model control module implements the transformation of the model by means of matrix transformation through the instructions parsed; Among them, the mobile transformation matrix is: d is the distance moved; The rotation transformation matrix is: θ is the rotation angle; The scaling transformation matrix is: s x ,s y ,s z is the scaling factor.

Images