Surgical operation teaching simulation method and system

Abstract

The invention provides a surgical operation teaching simulation method and system. The method comprises the following steps: receiving a user instruction; inputting the operation learning item content into a preset neural network model to obtain a video display item corresponding to the learning item content; the method comprises the following steps: identifying object, scene and character features in a video by using a computer vision technology, and extracting key frames and timestamps corresponding to the key frames from the video; modeling based on a scene in the video; defining various states of the virtual patient in the three-dimensional scene; monitoring the state of the virtual patient in real time in the process of simulating the operation; the instruction is sent to the video player according to the detected state recognition result of the virtual patient, and the progress of the teaching video can be always kept synchronous with the operation progress of the user in the process that the user performs the simulated operation, so that the user can watch the teaching video in real time, and the overall teaching effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical operation simulation, and in particular to a surgical operation teaching simulation method and system. Background Art

[0002] Surgical teaching simulation method is a teaching method that uses modern scientific and technological means to simulate real surgical scenes and operation processes. This method simulates the surgical environment, surgical instruments, surgical steps, etc., so that learners can practice repeatedly in a virtual environment, thereby mastering surgical skills and improving the safety and effectiveness of surgical operations. The surgical teaching simulation system is a comprehensive teaching system that integrates virtual simulation technology, computer graphics, human-computer interaction and other technologies. The system provides learners with a safe, controllable and repeatable practice platform by simulating real surgical scenes and operation processes.

[0003] Existing surgical teaching simulation methods usually use virtual simulation software for surgical simulation training, which can simulate real surgical scenes, including surgical instruments, patient anatomical structures, surgical steps, etc. During the simulation process, corresponding teaching videos are usually provided. Usually, the device will first receive user instructions, and parse the user instructions to obtain the surgical learning item content, and input the surgical learning item content into a preset neural network model to obtain the teaching video corresponding to the learning item content. Users can perform corresponding operations according to the teaching video. However, in the existing operation process, since the video will be played continuously, when the user forgets the relevant operation steps during the simulated operation, the user is required to watch the video again, including the teaching video of the steps after the operation, which reduces the overall teaching quality.

[0004] Therefore, it is necessary to provide a new surgical teaching simulation method and system to solve the above technical problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a surgical operation teaching simulation method and system.

[0006] The surgical operation teaching simulation method provided by the present invention comprises the following steps:

[0007] S1: receiving a user instruction, and parsing the user instruction to obtain surgical learning item content;

[0008] S2: Inputting the surgical learning item content into a preset neural network model to obtain a video display item corresponding to the learning item content;

[0009] S3: Use computer vision technology to identify objects, scenes, and character features in the video, and extract key frames and their corresponding timestamps from the video. The key frames represent the status of the virtual patient in the video.

[0010] S4: Modeling based on the scenes in the video, extracting objects, scenes, and virtual patients from the video to generate a three-dimensional scene;

[0011] S5: in the three-dimensional scene, define various states of the virtual patient, establish a mapping relationship between the virtual patient state in the video key frame and the virtual patient state in the three-dimensional scene, and organize the mapping relationship into a mapping table;

[0012] S6: The user starts a simulated surgery. During the simulated surgery, the state of the virtual patient is monitored in real time. When the state of the virtual patient changes, the state recognition algorithm is used to determine the current state, and the corresponding video key frame and the timestamp corresponding to the key frame are found in the mapping table.

[0013] S7: According to the state recognition result of the detected virtual patient, send instructions to the video player, jump to the timestamp of the corresponding video key frame, and repeat steps S3 to S6 until the simulated surgery process is completed.

[0014] Furthermore, the neural network model pre-stores various medical imaging materials of surgical patients, surgical operation procedures and treatment results.

[0015] Furthermore, the neural network model is specifically a tree-shaped recursive neural network model, so that the user instructions and the obtained surgical learning content items are constructed into a tree structure, and the cases and surgical operations corresponding to the learning item content are determined according to the tree structure, and the video display items are determined accordingly.

[0016] Furthermore, the specific steps for modeling based on the scenes in the video are as follows:

[0017] S01: Import the video file to be modeled into the modeling software;

[0018] S02: Detect stable and identifiable feature points in key frames;

[0019] S03: Estimate the intrinsic and extrinsic parameters of the camera in the modeling software according to the detection results of the feature points;

[0020] S04: Reconstruct the 3D point cloud using the detected feature points and camera parameters. On the basis of the 3D point cloud reconstruction, generate a mesh model through an interpolation optimization algorithm, and map the color information in the video to the 3D model to make it have a more realistic appearance.

[0021] S05: Add animation effects to the model and set a user interaction interface so that the model can interact with the user. The user interaction interface includes buttons, sliders, and touch screen elements.

[0022] Preferably, the specific process steps for detecting feature points in a key frame are as follows:

[0023] S0201: Read key frames in a video or image sequence to be processed into computer memory;

[0024] S0202: Construct the scale space of the image through the box filter. The box filter uses the integral image technology to quickly calculate the sum of the image area;

[0025] S0203: At each scale level, the determinant of the Hessian matrix is ​​used to detect potential key points, and the local extreme points of the determinant are key points;

[0026] S0204: Calculate the Haar wavelet response in the circular area with the key point as the center, and determine the main direction of the key point by counting the accumulated value of the Haar wavelet response in the circular area. Due to the simplicity and orthogonality of the Haar wavelet, the Haar wavelet transform can efficiently decompose the image signal and extract different feature information;

[0027] S0205: Select a rectangular area around the key point, calculate the Haar wavelet response in the area, divide the rectangular area into smaller sub-areas, calculate the cumulative value of the Haar wavelet response of the pixel points in each sub-area, and form a feature descriptor of the key point;

[0028] S0206: Use the generated feature descriptors to perform feature matching and find identifiable feature points.

[0029] Furthermore, during the simulated surgery, the state of the virtual patient is monitored by sensors.

[0030] Furthermore, the specific steps of establishing the mapping relationship between the virtual patient state in the video key frame and the virtual patient state in the three-dimensional scene are as follows:

[0031] A1: Synchronize the element states in the 3D scene to the corresponding moments according to the timestamps corresponding to the key frames in the video;

[0032] A2: Map the parameter information of the video key frame to the corresponding elements in the 3D scene.

[0033] The present invention also provides a surgical operation teaching system, which is applied to the above-mentioned surgical operation teaching simulation method, comprising:

[0034] Instruction receiving module: used to receive user instructions and parse the user instructions to obtain the content of surgical learning items;

[0035] Video acquisition module: used to input the surgical learning item content into a preset neural network model to obtain a video display item corresponding to the learning item content;

[0036] Model generation module: Modeling is performed based on the scenes in the video, objects, scenes, and virtual patients are extracted from the video, a three-dimensional scene is generated, and various states of the virtual patients are defined, and a mapping relationship between the virtual patient states in the key frames of the video and the virtual patient states in the three-dimensional scene is established;

[0037] Operation module: used for users to start simulated surgery, monitor the status of virtual patients in real time through sensors, and send status information of virtual patients;

[0038] Adjustment module: According to the state recognition result of the detected virtual patient, send instructions to the video player to jump to the timestamp of the corresponding video key frame.

[0039] Furthermore, the sensors include a heart rate sensor, a blood pressure sensor, and a breathing sensor, and the above sensors are used to collect and transmit the physiological data of the virtual patient in real time.

[0040] Furthermore, the heart rate sensor is used to monitor and record the heart rate changes of the virtual patient in real time; the blood pressure sensor is used to monitor the blood pressure condition of the virtual patient; and the breathing sensor is used to monitor the breathing frequency and depth of the virtual patient.

[0041] Compared with the related art, the surgical operation teaching simulation method and system provided by the present invention have the following advantages:

[0042] Beneficial effects:

[0043] The present invention utilizes computer vision technology to identify features of objects, scenes, and characters in a video, extracts key frames and timestamps corresponding to the key frames from the video, wherein the key frames represent the state of a virtual patient in the video, performs modeling based on the scenes in the video, extracts objects, scenes, and virtual patients from the video, and generates a three-dimensional scene. In the three-dimensional scene, various states of the virtual patient are defined, and a mapping relationship between the state of the virtual patient in the video key frames and the state of the virtual patient in the three-dimensional scene is established. When a user starts a simulated operation and the state of the virtual patient is monitored in real time during the simulated operation, a state recognition algorithm is used to determine the current state, and the corresponding video key frames and timestamps corresponding to the key frames in the mapping table are searched. Subsequently, according to the state recognition result of the detected virtual patient, an instruction is sent to a video player to jump to the timestamp of the corresponding video key frames. The above method enables the progress of the teaching video to always be synchronized with the operation progress of the user during the simulated operation, so that the user can watch the teaching video in real time, thereby improving the overall teaching effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 An operation flow chart of the surgical operation teaching simulation method provided by the present invention;

[0045] Figure 2 An operation flow chart of modeling based on scenes in a video provided by the present invention;

[0046] Figure 3 This is a block diagram of the surgical teaching simulation system provided by the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described below in conjunction with the accompanying drawings and implementation modes.

[0048] In the specific implementation process, Figure 1-Figure 3 As shown, a surgical operation teaching simulation method comprises the following steps:

[0049] S1: receiving a user instruction, and parsing the user instruction to obtain surgical learning item content;

[0050] Specifically, the operator's command input method is to complete the user through a device that receives the user's command through a touch screen;

[0051] S2: Inputting the surgical learning item content into a preset neural network model to obtain a video display item corresponding to the learning item content;

[0052] The neural network model pre-stores various medical records and imaging materials of surgical patients, surgical procedures, and treatment results;

[0053] The neural network model is specifically a tree-shaped recursive neural network model, which constructs the user instructions and the obtained surgical learning content items into a tree structure, and determines the cases and surgical operations corresponding to the learning item content according to the tree structure, and determines the video display items accordingly;

[0054] S3: Use computer vision technology to identify objects, scenes, and character features in the video, and extract key frames and their corresponding timestamps from the video. The key frames represent the status of the virtual patient in the video.

[0055] S4: Modeling is performed based on the scenes in the video, and objects, scenes, and virtual patient features are extracted from the video to generate a three-dimensional scene.

[0056] The specific steps for modeling based on the scenes in the video are as follows:

[0057] S01: Import the video file to be modeled into the modeling software;

[0058] Among them, the modeling software mentioned above is Blender, which supports the creation and publishing of real-time 3D scene interactive content. Open the "Import" function in Blender, select the video file, ensure that the video file is correctly loaded into the timeline or media library of the software, preview the video in the software, confirm its integrity and quality, and prepare for subsequent processing;

[0059] S02: Detect stable and identifiable feature points in key frames, using the SURF algorithm to detect identifiable feature points in key frames:

[0060] S03: Estimate the intrinsic and extrinsic parameters of the camera in the modeling software through a camera calibration algorithm according to the detection results of the feature points;

[0061] S04: 3D point cloud reconstruction is performed using the detected feature points and camera parameters. On the basis of the 3D point cloud reconstruction, an interpolation optimization algorithm is used. The interpolation optimization algorithm is mainly used in 3D point cloud reconstruction to convert discrete point cloud data into a continuous mesh model, while optimizing the smoothness, topological structure and geometric properties of the mesh. The mesh model is generated by the interpolation optimization algorithm, and the color information in the video is mapped to the 3D model to make it have a more realistic appearance.

[0062] S05: adding animation effects to the model and setting a user interaction interface so that the model can interact with the user, wherein the model animation effects include: bone binding: creating a bone system for the mesh model and binding the bones to the mesh of the model;

[0063] Animation: Use keyframe animation, motion capture data, or physical simulation methods to add animation effects to the model, such as simulation of surgical operations, rotation and scaling of the model;

[0064] The user interface includes buttons, sliders, and touch screen elements;

[0065] Every step the user takes during the simulated surgery will be reflected in the virtual patient's status in real time, updating the virtual patient's status.

[0066] S5: In the three-dimensional scene, various states of the virtual patient are defined, a mapping relationship between the virtual patient state in the video key frame and the virtual patient state in the three-dimensional scene is established, and the mapping relationship is organized into a mapping table.

[0067] Among them, the status of the virtual patient can be:

[0068] Behavior status: standing, walking, sitting, lying down, raising hands, bending over, talking to the doctor

[0069] Physiological status: including the virtual patient's heart rate, respiratory rate, blood pressure, and body temperature.

[0070] The specific process of establishing the mapping relationship between the virtual patient state in the video key frame and the virtual patient state in the three-dimensional scene is as follows:

[0071] A1: Synchronize the element states in the 3D scene to the corresponding moments according to the timestamps corresponding to the key frames in the video;

[0072] Specifically: A01: Timestamp extraction: extract the timestamp of each key frame from the video file. These timestamps mark the moments of important events or state changes in the video, ensuring the accuracy and consistency of the timestamps so that the element states in the three-dimensional scene can be accurately matched to them.

[0073] A02: In the 3D modeling or animation software, set the timeline corresponding to the video timeline and create a time controller to manage the passage of time in the 3D scene, ensuring that the time unit of the 3D scene matches the time unit of the video;

[0074] A03: For each element in the 3D scene, i.e., objects, scenes, virtual patients, its state at different time points is recorded, and these state records are associated with the key frame timestamps in the video;

[0075] A04: When the video plays to a key frame, the corresponding time point is found in the 3D scene according to the timestamp of the key frame, and the state of the relevant elements in the 3D scene is adjusted to match the state shown in the video key frame.

[0076] A2: Map the parameter information of the video key frame to the corresponding elements in the 3D scene;

[0077] Specifically, combining videos with three-dimensional scenes not only enriches the content presentation, enhances the content's expressiveness and appeal, but also improves the overall quality of teaching.

[0078] S6: The user starts simulating surgery. During the simulation, the state of the virtual patient is monitored in real time. When the state of the virtual patient changes, the state recognition algorithm is used to determine the current state, and the corresponding video key frame and the timestamp corresponding to the key frame are found in the mapping table.

[0079] During the simulated surgery, the state of the virtual patient is monitored through sensors, which monitor the progress of the surgical steps and changes in the patient's vital signs.

[0080] The sensors include a heart rate sensor, a blood pressure sensor, and a breathing sensor, which are used to collect and transmit the physiological data of the virtual patient in real time, and these data reflect the patient's vital signs status.

[0081] S7: according to the state recognition result of the detected virtual patient, send an instruction to the video player, jump to the timestamp of the corresponding video key frame, and repeat steps S3 to S6 until the simulated surgery process is completed;

[0082] Specifically, the above method ensures that the progress of the teaching video can always be synchronized with the user's operation progress during the user's simulated surgery, so that the user can watch the teaching video in real time, thereby improving the overall teaching effect.

[0083] Among them, the specific process steps of step S02 are as follows:

[0084] S0201: Read key frames in a video or image sequence to be processed into computer memory;

[0085] S0202: Construct the scale space of the image through the box filter. The box filter uses the integral image technology to quickly calculate the sum of the image area;

[0086] S0203: At each scale level, the determinant of the Hessian matrix is ​​used to detect potential key points, and the local extreme points of the determinant are key points;

[0087] S0204: Calculate the Haar wavelet response in the circular area with the key point as the center, and determine the main direction of the key point by counting the accumulated value of the Haar wavelet response in the circular area. Due to the simplicity and orthogonality of the Haar wavelet, the Haar wavelet transform can efficiently decompose the image signal and extract different feature information;

[0088] S0205: Select a rectangular area around the key point, calculate the Haar wavelet response in the area, divide the rectangular area into smaller sub-areas, calculate the cumulative value of the Haar wavelet response of the pixel points in each sub-area, and form a feature descriptor of the key point;

[0089] S0206: Use the generated feature descriptors to perform feature matching and find identifiable feature points.

[0090] The specific process steps of step S03 are as follows:

[0091] S0301: Setting the initial parameters of the camera calibration algorithm, including the initial values ​​of the preset camera intrinsic and extrinsic parameters;

[0092] S0302: Process the detected feature points through the Zhang calibration algorithm, and use the least squares method and other optimization methods to solve the camera's intrinsic and extrinsic parameters. Zhang calibration is a commonly used method. It collects pictures of different postures of the calibration plate, extracts the pixel coordinates of the corner points in the picture, calculates the initial values ​​of the camera's intrinsic and extrinsic parameters through the homography matrix, and then estimates the distortion coefficient using the nonlinear least squares method, and uses the maximum likelihood estimation method to optimize the parameters;

[0093] S0303: After solving the intrinsic and extrinsic parameters of the camera, the image is distorted using the distortion coefficient.

[0094] A surgical operation teaching system, applied to the above surgical operation teaching simulation method, comprising:

[0095] Instruction receiving module: used to receive user instructions and parse the user instructions to obtain the content of surgical learning items;

[0096] Video acquisition module: used to input the surgical learning item content into the preset neural network model to obtain the video display item corresponding to the learning item content;

[0097] Model generation module: Modeling is performed based on the scenes in the video, objects, scenes, and virtual patients are extracted from the video, a three-dimensional scene is generated, and various states of the virtual patients are defined, and a mapping relationship between the virtual patient states in the key frames of the video and the virtual patient states in the three-dimensional scene is established;

[0098] Operation module: used for users to start simulated surgical operations, while monitoring the status of virtual patients in real time through sensors and sending status information of virtual patients.

[0099] Adjustment module: According to the state recognition result of the detected virtual patient, send instructions to the video player to jump to the timestamp of the corresponding video key frame.

[0100] The sensors include a heart rate sensor, a blood pressure sensor, and a breathing sensor, which are used to collect and transmit the physiological data of the virtual patient in real time.

[0101] The heart rate sensor is used to monitor and record the heart rate changes of virtual patients in real time. The patient's heart rate is an important indicator for evaluating his or her vital signs and surgical stress response. Through the heart rate sensor, the system can simulate the immediate impact of surgical operations on the patient's heart rate.

[0102] The blood pressure sensor is used to monitor the blood pressure condition of the virtual patient. The system can simulate the blood pressure fluctuations that may occur during the operation through the blood pressure sensor.

[0103] The respiratory sensor is used to monitor the breathing rate and depth of the virtual patient. Breathing is a basic process of life activities. The patient's breathing condition during surgery is often affected by multiple factors such as anesthesia, pain, and tension. Through the respiratory sensor, the system can simulate the patient's breathing changes in different surgical scenarios.

[0104] The circuits and controls involved in the present invention are all prior art and will not be described in detail here.

[0105] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A surgical operation teaching simulation method, It is characterized in that The following steps are involved: S1: receiving a user instruction, and parsing the user instruction to obtain surgical learning item content; S2: Inputting the surgical learning item content into a preset neural network model to obtain a video display item corresponding to the learning item content, wherein the neural network model pre-stores various medical history imaging materials of surgical patients, surgical operation processes, and treatment results; S3: Use computer vision technology to identify objects, scenes, and character features in the video, and extract key frames and their corresponding timestamps from the video. The key frames represent the status of the virtual patient in the video. S4: Modeling based on the scenes in the video, extracting objects, scenes, and virtual patients from the video to generate a three-dimensional scene; S5: in the three-dimensional scene, define various states of the virtual patient, establish a mapping relationship between the virtual patient state in the video key frame and the virtual patient state in the three-dimensional scene, and organize the mapping relationship into a mapping table; S6: The user starts a simulated surgery. During the simulated surgery, the state of the virtual patient is monitored in real time. When the state of the virtual patient changes, the state recognition algorithm is used to determine the current state, and the corresponding video key frame and the timestamp corresponding to the key frame are found in the mapping table. S7: According to the state recognition result of the detected virtual patient, send instructions to the video player, jump to the timestamp of the corresponding video key frame, and repeat steps S3 to S6 until the simulated surgery process is completed.

2. The surgical teaching simulation method according to claim 1, It is characterized in that The neural network model is specifically a tree-shaped recursive neural network model, which constructs user instructions and obtained surgical learning content items into a tree structure, and determines the cases and surgical operations corresponding to the learning item content based on the tree structure, and determines the video display items accordingly.

3. The surgical operation teaching simulation method according to claim 2, It is characterized in that The specific steps for modeling based on the scenes in the video are as follows: S01: Import the video file to be modeled into the modeling software; S02: Detect feature points in key frames; S03: Estimate the intrinsic and extrinsic parameters of the camera in the modeling software according to the detection results of the feature points; S04: Reconstruct the 3D point cloud using the detected feature points and camera parameters. On the basis of the 3D point cloud reconstruction, generate a mesh model through an interpolation optimization algorithm to map the color information in the video onto the 3D model. S05: Add animation effects to the model and set a user interaction interface so that the model can interact with the user. The user interaction interface includes buttons, sliders, and touch screen elements.

4. The surgical teaching simulation method according to claim 3, It is characterized in that in, The specific steps of detecting feature points in key frames are as follows: S0201: Read key frames in a video or image sequence to be processed into computer memory; S0202: Construct the scale space of the image through the box filter. The box filter uses the integral image technology to quickly calculate the sum of the image area; S0203: At each scale level, the determinant of the Hessian matrix is ​​used to detect potential key points, and the local extreme points of the determinant are key points; S0204: Calculate the Haar wavelet response in the circular area with the key point as the center, and determine the main direction of the key point by counting the accumulated value of the Haar wavelet response in the circular area. Due to the simplicity and orthogonality of the Haar wavelet, the Haar wavelet transform can efficiently decompose the image signal and extract different feature information; S0205: Select a rectangular area around the key point, calculate the Haar wavelet response in the area, divide the rectangular area into smaller sub-areas, calculate the cumulative value of the Haar wavelet response of the pixel points in each sub-area, and form a feature descriptor of the key point; S0206: Use the generated feature descriptors to perform feature matching and find identifiable feature points.

5. The surgical operation teaching simulation method according to claim 1, It is characterized in that During the simulated surgery, the status of the virtual patient is monitored through sensors.

6. The surgical operation teaching simulation method according to claim 5, It is characterized in that The specific steps of establishing the mapping relationship between the virtual patient state in the video key frame and the virtual patient state in the three-dimensional scene are as follows: A1: Synchronize the element states in the 3D scene to the corresponding moments according to the timestamps corresponding to the key frames in the video; A2: Map the parameter information of the video key frame to the corresponding elements in the 3D scene.

7. A surgical teaching system, applied to the surgical teaching simulation method according to any one of claims 1 to 6, It is characterized in that include: Instruction receiving module: used to receive user instructions and parse the user instructions to obtain the content of surgical learning items; Video acquisition module: used to input the surgical learning item content into a preset neural network model to obtain a video display item corresponding to the learning item content; Model generation module: Modeling is performed based on the scenes in the video, objects, scenes, and virtual patients are extracted from the video, a three-dimensional scene is generated, and various states of the virtual patients are defined, and a mapping relationship between the virtual patient states in the key frames of the video and the virtual patient states in the three-dimensional scene is established; Operation module: used for users to start simulated surgery, monitor the status of virtual patients in real time through sensors, and send status information of virtual patients; Adjustment module: According to the state recognition result of the detected virtual patient, send instructions to the video player to jump to the timestamp of the corresponding video key frame.

8. The surgical teaching system according to claim 7, It is characterized in that The sensors include a heart rate sensor, a blood pressure sensor, and a breathing sensor.

9. The surgical teaching system according to claim 8, It is characterized in that The heart rate sensor is used to monitor and record the heart rate changes of the virtual patient in real time; the blood pressure sensor is used to monitor the blood pressure condition of the virtual patient; and the breathing sensor is used to monitor the breathing frequency and depth of the virtual patient.

Images