Medical model visual interaction control method and system based on gesture recognition

Abstract

The invention discloses a medical model visual interaction control method and system based on gesture recognition, and relates to the technical field of medical image interaction, and the method comprises the steps: S1, achieving the three-dimensional imaging and processing of CT and MRI image data through VTK; s2, carrying out image acquisition on an operator, and capturing a gesture action image of the operator; s3, performing gesture segmentation on the captured gesture action image by adopting a gesture segmentation algorithm; s4, analyzing the gesture change of the operator based on gesture segmentation; and S5, performing real-time control on the medical three-dimensional model according to gesture change analysis. According to the medical model visual interaction control method based on gesture recognition, a doctor can operate through natural gestures, and dependence on traditional input equipment (such as a mouse and a keyboard) is reduced. The visual interaction mode can better conform to the operation habits of doctors, the working efficiency is improved, and the visual interaction mode is particularly important in medical scenes where quick response and complex information processing are needed.

Description

Technical Field

[0001] The present invention relates to the field of medical image interaction technology, and in particular to a medical model visualization interaction control method and system based on gesture recognition. Background Art

[0002] As one of the most important scientific and technological fields today, human-computer interaction technology has received much attention. Mechanical input devices such as keyboards and mice are difficult to perform 3D and highly free representations, and this interaction method is inconvenient to some extent. As research work progresses, more and more people are beginning to devote themselves to studying human-computer interaction technologies that conform to human habits, such as face recognition, body tracking and recognition, and gesture tracking.

[0003] The 3D visualization technology of medical images is a hot topic with a wide range of applications. Through 3D model visualization technology, surface rendering technology, combined with medical images and the latest applications in clinical diagnosis and treatment, a large number of applications of 3D visualization technology of medical images have been seen. In recent years, with the continuous emergence of image visualization technology, the endless emergence of image processing technology, and the rapid development of various acceleration algorithms, 3D visualization technology and its applications will continue to develop in the medical field. Medical image stereoscopic display technology reconstructs 3D models from 2D medical images, providing more intuitive, comprehensive and accurate information on lesions and normal tissues, which brings great convenience to doctors' clinical diagnosis and treatment.

[0004] With the continuous advancement of medical technology and the development of informatization, medical image processing and visualization technology play an increasingly important role in clinical diagnosis and surgical navigation. The visualization of medical models can not only provide clear anatomical structure and lesion information, but also provide doctors with a real-time interactive experience, helping them to better understand the patient's condition and develop more effective treatment plans. However, traditional medical model visualization technology mostly relies on input devices such as mouse and keyboard, which may lead to unnatural interaction in some cases, affecting the doctor's work efficiency and experience. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for visualizing and interactively controlling a medical model based on gesture recognition, so as to solve the problems raised in the above-mentioned background technology.

[0006] To solve the above problems, the technical solutions adopted by the present invention are as follows:

[0007] A visual interactive control method for medical models based on gesture recognition, including

[0008] S1. Realize 3D imaging and processing of CT and MRI image data through VTK, including data loading, preprocessing, segmentation, 3D visualization and holographic image generation;

[0009] S2, collecting images of the operator and capturing images of the operator's gestures;

[0010] S3, using a gesture segmentation algorithm to perform gesture segmentation on the captured gesture action image;

[0011] S4, analyzing the operator's gesture changes based on gesture segmentation;

[0012] S5. Based on the analysis of gesture changes, the medical three-dimensional model is controlled in real time.

[0013] Preferably, the data is loaded using VTK and stored in a DI COM format;

[0014] The data preprocessing includes smoothing and noise reduction processing;

[0015] The image segmentation uses threshold segmentation or an algorithm to extract the target structure;

[0016] The three-dimensional visualization converts the processed data into a three-dimensional model by creating a VTK rendering window, a renderer and an interactor, and combining the coronal, sagittal and cross-sectional views to display the original image data;

[0017] The holographic image generation calculates the phase information of the object surface by simulating the interference of light waves, and further generates a hologram using the interference principle.

[0018] Preferably, the data preprocessing adopts Gaussian smoothing technology, and uses Gaussian function to perform convolution operation on the image to reduce noise in the image and smooth the image. The Gaussian function expression is:

[0019]

[0020] Among them, G(x,y) is the value of the Gaussian function, σ is the standard deviation, which controls the width of the Gaussian kernel. The larger the standard deviation, the more obvious the smoothing effect, and (x,y) is the coordinate of the pixel in two-dimensional space.

[0021] Preferably, the threshold segmentation distinguishes the foreground and the background by a set threshold. If the gray value of the original image is I(x, y), segmentation is performed by a set threshold T. The segmentation formula is as follows:

[0022]

[0023] Among them, S(x,y) is the result image after segmentation, I(x,y) is the grayscale value of the original image, and T is the set threshold.

[0024] Preferably, the gesture segmentation algorithm adopts a deep learning method to perform gesture segmentation by training a convolutional neural network. The network can learn the features of the hand and achieve accurate segmentation. The steps for establishing the gesture segmentation algorithm are as follows:

[0025] S11. Collect and annotate a gesture database, use tools to annotate gestures, and generate segmentation masks for each image;

[0026] S12. Build a convolutional neural network model, and select a convolutional neural network architecture including U-Net, SegNet, or DeepLab for gesture segmentation;

[0027] S13: Perform model training and prepare images and corresponding segmentation masks.

[0028] Preferably, the gesture database is used to identify key points of the hand and distinguish different single-hand gestures according to program presets.

[0029] Preferably, the model training adopts the following formula:

[0030] Loss function:

[0031] Among them, y i is the true label, is the predicted output of the model;

[0032] The optimizer updates the weight formula:

[0033]

[0034] Among them, m t is the average value of the gradient, v t is the average of the squared gradients, α is the learning rate, and e is a small constant to prevent division by zero errors.

[0035] Medical model visualization interactive control system based on gesture recognition, including

[0036] A gesture recognition module that captures the operator’s hand movements using a camera or depth sensor;

[0037] The data processing module uses VTK to load medical imaging data and process the medical imaging data into a three-dimensional model;

[0038] The control module converts the gesture recognition results into control instructions for the model;

[0039] The visualization module is used to update the display of the three-dimensional model according to the instructions sent by the control module.

[0040] Preferably, the camera captures dynamic images of the hand at a high frame rate to provide continuous visual information, and analyzes the image using a computer vision algorithm to extract key points, contours and motion trajectories of the hand, thereby recognizing gestures;

[0041] The depth sensor is used to provide distance data between the hand and the sensor, helping the algorithm to obtain the three-dimensional position and posture of the hand, and accurately identify the shape and posture of the hand through the depth information.

[0042] Preferably, the gesture recognition module is used to remotely control the model through a single-hand gesture, and to perform initial recognition by raising one hand to show different initial movements, thereby completing the position conversion of the model.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention uses a medical model visualization interactive control method based on gesture recognition to enable doctors to operate through natural gestures, reducing dependence on traditional input devices (such as mouse and keyboard). This intuitive interaction method can better meet the doctor's operating habits and improve work efficiency, which is particularly important in medical scenarios that require rapid response and processing of complex information.

[0045] By implementing gesture recognition interaction, the present invention allows doctors to perform surgery or medical operations without touching any equipment, keeping their hands clean and hygienic. This contactless interaction method can significantly improve user experience, reduce the risk of cross infection, and improve patient safety.

[0046] The gesture recognition technology of the present invention can achieve real-time recognition and response. Doctors can quickly perform operations such as zooming, rotating and translating when viewing medical models, thereby more effectively analyzing the patient's condition. This efficient operation method helps doctors obtain the required information more quickly and improves the accuracy of diagnosis and treatment.

[0047] The present invention can adaptively adjust the parameters of gesture recognition and the model interaction mode according to different user needs and operating habits to meet the personalized needs of different medical scenarios and doctors. This flexibility enhances the application scope and practicality of the system, and promotes the innovation and development of medical technology by applying gesture recognition technology to the medical field. The system demonstrates how modern technology can effectively support traditional medical practice and promote the development of the medical industry towards digitalization and intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of the medical model visualization interactive control method based on gesture recognition of the present invention;

[0049] Figure 2This is a block diagram of the medical model visualization interactive control system based on gesture recognition of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] Embodiment 1:

[0052] See also Figure 1 As shown, the present invention is a medical model visualization interactive control method based on gesture recognition, comprising

[0053] S1. Realize 3D imaging and processing of CT and MRI image data through VTK, including data loading, preprocessing, segmentation, 3D visualization and holographic image generation;

[0054] S2, collecting images of the operator and capturing images of the operator's gestures;

[0055] S3, using a gesture segmentation algorithm to perform gesture segmentation on the captured gesture action image;

[0056] S4, analyzing the operator's gesture changes based on gesture segmentation;

[0057] S5. Based on the analysis of gesture changes, the medical three-dimensional model is controlled in real time.

[0058] Data is loaded using VTK and stored in DICOM format;

[0059] Data preprocessing includes smoothing and noise reduction;

[0060] Image segmentation uses threshold segmentation or algorithm to extract target structure;

[0061] 3D visualization creates VTK rendering windows, renderers, and interactors, and displays raw image data in coronal, sagittal, and cross-sectional planes, converting processed data into 3D models;

[0062] Holographic image generation calculates the phase information of the object surface by simulating the interference of light waves, and further uses the interference principle to generate a hologram.

[0063] The data preprocessing uses Gaussian smoothing technology, which uses Gaussian function to perform convolution operation on the image to reduce the noise in the image and smooth the image. The Gaussian function expression is:

[0064]

[0065] Among them, G(x,y) is the value of the Gaussian function, σ is the standard deviation, which controls the width of the Gaussian kernel. The larger the standard deviation, the more obvious the smoothing effect, and (x,y) is the coordinate of the pixel in two-dimensional space.

[0066] Threshold segmentation distinguishes the foreground and background by setting the threshold. If the gray value of the original image is I(x, y), the segmentation is performed by setting the threshold T. The segmentation formula is as follows:

[0067]

[0068] Among them, S(x,y) is the result image after segmentation, I(x,y) is the grayscale value of the original image, and T is the set threshold.

[0069] The gesture segmentation algorithm uses deep learning to perform gesture segmentation by training a convolutional neural network. The network can learn the features of the hand and achieve accurate segmentation. The steps to establish the gesture segmentation algorithm are as follows:

[0070] S11. Collect and annotate a gesture database, use tools to annotate gestures, and generate segmentation masks for each image;

[0071] S12. Build a convolutional neural network model, and select a convolutional neural network architecture including U-Net, SegNet, or DeepLab for gesture segmentation;

[0072] S13: Perform model training and prepare images and corresponding segmentation masks.

[0073] The gesture database is used to identify the key points of the hand and distinguish different single-hand gestures according to program presets.

[0074] The model training adopts the following formula:

[0075] Loss function:

[0076] Among them, y i is the true label, is the predicted output of the model;

[0077] The optimizer updates the weight formula:

[0078]

[0079] Among them, m t is the average value of the gradient, v t is the average of the squared gradients, α is the learning rate, and e is a small constant to prevent division by zero errors.

[0080] Specifically, for different sections (such as coronal, sagittal, and cross-sectional planes), image data can be captured using different coordinate systems;

[0081] Coronal plane: fix z coordinate, change x and y;

[0082] Sagittal plane: fix x coordinate, change y and z;

[0083] Cross section: fix y coordinate, vary x and z.

[0084] As can be seen from the above, the above scheme realizes the three-dimensional imaging of CT, MRI and other image data. After processing, the organs, tumors, blood vessels, nerves and other tissue structures in the lesion area are segmented and marked to generate a clearer and more intuitive three-dimensional model, which can combine the coronal, sagittal and cross-sectional images to form a holographic image for patient examination.

[0085] Doctors can also control the display and interaction of three-dimensional medical models through natural gestures, thereby improving the efficiency and accuracy of medical image analysis.

[0086] Embodiment 2:

[0087] See also Figure 2 As shown, a medical model visualization interactive control system based on gesture recognition includes a gesture recognition module, which captures the operator's hand movement by using a camera or a depth sensor;

[0088] The data processing module uses VTK to load medical imaging data and process the medical imaging data into a three-dimensional model;

[0089] The control module converts the gesture recognition results into control instructions for the model;

[0090] The visualization module is used to update the display of the three-dimensional model according to the instructions sent by the control module.

[0091] The camera captures dynamic images of the hand at a high frame rate, providing continuous visual information. The computer vision algorithm is used to analyze the image, extract the key points, contours and motion trajectories of the hand, and thus recognize the gesture.

[0092] The depth sensor is used to provide distance data between the hand and the sensor, helping the algorithm obtain the three-dimensional position and posture of the hand, and accurately identify the shape and posture of the hand through depth information.

[0093] The gesture recognition module is used to remotely control the model through one-hand gestures, and to perform initial recognition by raising one hand to show different initial movements and complete the position transformation of the model.

[0094] As can be seen from the above, the above system enables the operator to remotely control the model through single-hand gestures;

[0095] In specific implementation, the operator raises one hand to show different initial movements for initial recognition: such as fist (expanded initial), full palm elevation (shrinked initial, rotated initial), three fingers (moved initial: thumb, index finger, middle finger). Then the desired movement or rotation operation is performed by moving the position in the plane. In addition: if the fist is expanded, it will turn into a palm, and if it is reduced, it will turn from a palm to a fist;

[0096] Gesture recognition uses a database specifically designed for gesture recognition. Operators can upload gesture information to the database, which can identify key points of the hand and use deep learning technology to enable the system to more accurately understand and respond to the doctor's gestures.

[0097] The system can distinguish different single-hand gestures according to program presets, such as fist-to-palm for zooming in, full-palm elevation for rotation, and three-finger elevation movement for movement, thereby achieving non-contact control of 3D models;

[0098] Gesture recognition technology can achieve real-time recognition and response. Doctors can quickly zoom, rotate, and translate when viewing medical models, thereby more effectively analyzing the patient's condition. This efficient operation method helps doctors obtain the required information faster and improves the accuracy of diagnosis and treatment.

[0099] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, unless they are contradictory.

[0100] In the drawings of the embodiments disclosed in the present invention, only the structures related to the embodiments disclosed in the present invention are involved, and other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of the present invention can be combined with each other.

[0101] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A medical model visualization interactive control method based on gesture recognition, characterized by: include S1. Realize 3D imaging and processing of CT and MRI image data through VTK, including data loading, preprocessing, segmentation, 3D visualization and holographic image generation; S2, collecting images of the operator and capturing images of the operator's gestures; S3, using a gesture segmentation algorithm to perform gesture segmentation on the captured gesture action image; S4, based on gesture segmentation, analyzing the operator's gesture changes; S5. Based on the analysis of gesture changes, the medical three-dimensional model is controlled in real time.

2. The method for visualizing and interactively controlling a medical model based on gesture recognition according to claim 1, characterized in that: The data is loaded using VTK and stored in DICOM format; The data preprocessing includes smoothing and noise reduction processing; The image segmentation uses threshold segmentation or an algorithm to extract the target structure; The three-dimensional visualization converts the processed data into a three-dimensional model by creating a VTK rendering window, a renderer and an interactor, and combining the coronal, sagittal and cross-sectional views to display the original image data; The holographic image generation calculates the phase information of the object surface by simulating the interference of light waves, and further generates a hologram using the interference principle.

3. The method for visualizing and interactively controlling a medical model based on gesture recognition according to claim 2, characterized in that: The data preprocessing adopts Gaussian smoothing technology, and uses Gaussian function to perform convolution operation on the image to reduce the noise in the image and smooth the image. The Gaussian function expression is: Among them, G(x,y) is the value of the Gaussian function, σ is the standard deviation, which controls the width of the Gaussian kernel. The larger the standard deviation, the more obvious the smoothing effect, and (x,y) is the coordinate of the pixel in two-dimensional space.

4. The method for visualizing and interactively controlling a medical model based on gesture recognition according to claim 2, characterized in that: The threshold segmentation distinguishes the foreground and background by a set threshold. If the gray value of the original image is I(x, y), the segmentation is performed by the set threshold T. The segmentation formula is as follows: Among them, S(x,y) is the result image after segmentation, I(x,y) is the grayscale value of the original image, and T is the set threshold.

5. The method for visualizing and interactively controlling a medical model based on gesture recognition according to claim 1, characterized in that: The gesture segmentation algorithm adopts a deep learning approach to perform gesture segmentation by training a convolutional neural network. The network can learn the features of the hand and achieve accurate segmentation. The steps for establishing the gesture segmentation algorithm are as follows: S11, collect and annotate the gesture database, use tools to annotate the gestures, and generate segmentation masks for each image; S12. Build a convolutional neural network model, and select a convolutional neural network architecture including U-Net, SegNet, or DeepLab for gesture segmentation; S13: Perform model training and prepare images and corresponding segmentation masks.

6. The method for visualizing and interactively controlling a medical model based on gesture recognition according to claim 5, characterized in that: The gesture database is used to identify key points of the hand and distinguish different single-hand gestures according to program presets.

7. The method for visualizing and interactively controlling a medical model based on gesture recognition according to claim 5, characterized in that: The model training adopts the following formula: Loss function: Among them, y i is the true label, is the predicted output of the model; The optimizer updates the weight formula: Among them, m t is the average value of the gradient, v t is the average of the squared gradients, α is the learning rate, and e is a small constant to prevent division by zero errors.

8. A medical model visualization interactive control system based on gesture recognition, characterized by: include A gesture recognition module that captures the operator’s hand movements using a camera or depth sensor; The data processing module uses VTK to load medical imaging data and process the medical imaging data into a three-dimensional model; The control module converts the gesture recognition results into control instructions for the model; The visualization module is used to update the display of the three-dimensional model according to the instructions sent by the control module.

9. The medical model visualization interactive control system based on gesture recognition according to claim 8, characterized in that: The camera captures dynamic images of the hand at a high frame rate, provides continuous visual information, and uses a computer vision algorithm to analyze the image, extract key points, contours, and motion trajectories of the hand, thereby recognizing gestures; The depth sensor is used to provide distance data between the hand and the sensor, helping the algorithm to obtain the three-dimensional position and posture of the hand, and accurately identify the shape and posture of the hand through the depth information.

10. The medical model visualization interactive control system based on gesture recognition according to claim 8, characterized in that: The gesture recognition module is used to remotely control the model through single-hand gestures, and to perform initial recognition by raising one hand to show different initial movements, thereby completing the position conversion of the model.

Images