An interactive thoracoscopic surgery nursing training system based on virtual reality technology

Abstract

The present invention discloses an interactive thoracoscopic surgery nursing training system based on virtual reality technology, wherein the system comprises: a teacher's intelligent terminal for teaching management, a server, and several VR mobile devices at the student end, the teacher's intelligent terminal and the VR mobile device are connected to the server, the teacher's intelligent terminal issues nursing training instructions and nursing assessment instructions on the server, and the student logs in to the server through identity authentication via the VR mobile device; wherein the server includes a functional subsystem and a scene simulation subsystem; the thoracic surgery nursing training selection or thoracic surgery nursing assessment selection is performed on the functional subsystem in the server via the VR mobile device, and the functional subsystem performs corresponding scene simulation in the scene simulation subsystem according to the selection. In an embodiment of the present invention, providing VR thoracoscopic surgery nurse cooperation training can increase participation, improve learning interest and effect, thereby ensuring teaching quality and effect.

Description

Technical Field

[0001] The invention relates to the technical field of virtual reality, and in particular to an interactive thoracoscopic surgery nursing training system using virtual reality technology. Background Art

[0002] Lung cancer is one of the malignant tumors with high morbidity and mortality. It is particularly important to strengthen the prevention and treatment of lung cancer and improve the prognosis of patients. Radical surgical resection is the recommended first choice for the treatment of early lung cancer. With the continuous development of medical technology, surgical treatment of lung cancer has moved from traditional thoracotomy to video-assisted thoracoscopic surgery (VATS). Compared with traditional surgery, thoracoscopic surgery has smaller incisions, less impact on shoulder joint function, better protection of lung function, significantly improved patients' quality of life after surgery and promoted rapid recovery. With the continuous development of medical technology, thoracoscopic surgery has gradually moved from 2D flat video surgery to the era of 3D thoracoscopic surgery with a more three-dimensional and layered surgical field of view, which is closer to the naked eye. As an important member of the surgical medical team, operating room nurses play an important role to ensure the smooth and safe progress of the operation. With the continuous development of thoracoscopic surgery technology, the requirements for the cooperation of operating room nurses are getting higher and higher. It is particularly important to strengthen the training of operating room nurses in thoracoscopic surgery.

[0003] At present, most of the surgical cooperation training for operating room nurses in hospitals adopts traditional teaching methods: classroom teaching method, face-to-face one-to-one or one-to-many clinical teaching. The existing teaching methods can meet the basic requirements for the training of knowledge and operational skills of operating room nurses. However, the existing training technology has the following problems: First, the teaching scene and teaching model cannot truly reflect the anatomy of the lungs and mediastinum under thoracoscopic surgery and the key points of cooperation required for different processes such as processing lung tissue, blood vessels, and trachea. In addition, the existing teaching model is relatively boring, and the learning compliance and efficiency of learners are low, which is difficult to meet the actual application needs. Moreover, the training of thoracoscopic surgery nurses is not limited to the hospital. Some large medical centers and general hospitals need to undertake the training of advanced training and specialist nurses, while traditional offline teaching requires venues, time and personnel preparation, consumes a lot of manpower and material resources, and cannot meet the role of remote training. Therefore, a new system is needed to achieve better thoracoscopic surgery training for operating room nurses. Virtual reality technology (VR) is a computer simulation system that can create and experience a virtual world, which has been widely used in medical teaching and training. The current problems and defects in the field of virtual reality and medical technology training are: the application of virtual reality technology is concentrated on the training of surgeons, and there is a lack of attention to its application in the training of operating room nurses. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art. The present invention provides an interactive thoracoscopic surgery nursing training system based on virtual reality technology, which provides VR thoracoscopic surgery nurse cooperation training, can increase participation, improve learning interest and effect, thereby ensuring teaching quality and effect.

[0005] In order to solve the above technical problems, an embodiment of the present invention provides an interactive thoracoscopic surgery nursing training system using virtual reality technology, the system comprising: a teacher's intelligent terminal for teaching management, a server and several VR mobile devices at the student end, the teacher's intelligent terminal and the VR mobile device are connected to the server based on the Internet, after the teacher's intelligent terminal is connected to the server, the nursing training instructions and nursing assessment instructions are issued on the server, and the student logs in to the server through identity authentication using the VR mobile device; wherein the server contains a functional subsystem and a scene simulation subsystem;

[0006] Students use VR mobile devices to select thoracic surgery nursing training or thoracic surgery nursing assessment on the functional subsystem in the server, and the functional subsystem performs corresponding scene simulation in the scene simulation subsystem according to the students' selection.

[0007] Optionally, the functional subsystem includes a login module, a training module, an assessment module, a statistical analysis module and an interactive function module;

[0008] The login module is the external port of the server, which is used for registration and login of student users and loading of teaching and training resources for teacher users;

[0009] The training module is for the student user to enter the server through the login module and select 3D virtual reality real surgery resources in the scene simulation subsystem for operation guidance learning and training;

[0010] The assessment module is used to assess the student user's mastery of thoracoscopic surgery cooperation knowledge and cooperation process, and to conduct an evaluation test on the student user. The teacher user sends an assessment instruction through the server, and selects a video in the scene simulation subsystem to send to the corresponding student user. After logging in through a VR mobile device, the student user directly enters the assessment module for a simulated assessment;

[0011] The statistical analysis module receives data through the background of the server, and collects data on the training and / or assessment of each student user, and stores and analyzes the data;

[0012] The interactive function module is a module that provides text and / or voice interaction.

[0013] Optionally, the login module uses CANVAS canvas technology to draw a login registration interface, and the login registration interface is drawn by dragging and setting coordinates through UGUI components to form input boxes, buttons and text components; then use C# scripts to bind the account and password input box components in the login registration interface, and read the data entered by the student user, use MySql database technology to log in and store the account and password information entered by the student user, and query and verify or store the login user information in the database;

[0014] After the student user enters the server, the C# script is used to control the loading of resources and scenes, the real-time loading amount of resources is read, and the sliding bar in the UGUI is used to display the loading progress in the simulation interface. After entering, the corresponding thoracoscopic surgery is used for teaching resources.

[0015] Optionally, the assessment module is used for uploading and resetting test questions, and testing and assessing test questions; wherein the uploading and resetting of test questions are achieved by establishing a test question table and a question table in an EXCEL table, each question contains question information and question selection items, and then the C# code reads the data in the corresponding format and stores it in a MySql database, and the teacher user's account is used to manage the uploading and resetting of test questions, and the teacher user is required to directly delete the question information in the MySql database by selecting the corresponding test question. The test and assessment function of the test question utilizes the drawing board, text, and check box components in the UGUI to layout the answering interface, and utilizes the interactive function to implement the assessment at each node of thoracoscopic surgery cooperation, and the test results are presented in the form of a score table, and the test results include score number ID, student ID, test question ID, score, and score grade, and the assessment results are saved in the MySql database.

[0016] Optionally, the interactive function module is used to perform linear planning of video nodes for the real 3D thoracoscopic surgical video according to the surgical process and required cooperation points, and to clarify the interaction points based on the linear planning results; for the interaction points, create text and voice interaction content corresponding to the interaction points required for the cooperation of the surgical nurses according to the actual surgical cooperation situation; and insert the text and voice interaction content corresponding to the interaction points into the real 3D thoracoscopic surgical video in the form of text boxes and voices for prompt processing.

[0017] Optionally, the scene simulation subsystem includes a surgery scene simulation module and a thoracoscopic surgery video module;

[0018] The surgical scene simulation module is used to simulate the scene of a real surgery, including the scene of the patient, surgeon, anesthesiologist, assistant, mirror holder, and instrument nurse standing before the operation begins;

[0019] The thoracoscopic surgery video module is used to simulate the actual situation of the surgeon performing thoracoscopic surgery during the thoracoscopic surgery; the actual situation is produced by a real thoracoscopic surgery recording video.

[0020] Optionally, when simulating the construction of a real surgery scene, the surgical scene simulation module uses 3ds Max 3D software to perform 3D reconstruction processing, restores the real objects in the operating room scene, and performs surface rendering or mapping processing on the scene model;

[0021] When constructing the character models used in the simulation of real surgery, the personnel situation before the surgery is simulated, including the character simulation of the patient in the side-lying position before the surgery, the surgeon, the anesthesiologist, the assistant, the mirror holder, and the instrument nurse; wherein the instrument nurse is a perspective character model and can perform free perspective activities in the surgical scene;

[0022] When constructing the multimedia drawing board in the simulation of real surgery, the position of the thoracoscopic machine is simulated to play the real 3D thoracoscopic surgery cooperation video; through VideoManager, the script is implemented using the message mechanism processing framework implemented in virtual reality.

[0023] Optionally, when constructing the character model in the simulation of the real surgery, it also includes:

[0024] Through NURBS surface modeling technology, photos of the surgeon, anesthesiologist, assistant, mirror holder, and instrument nurse were taken respectively, and then imported into the 3DS Max three-dimensional view. The photos were fixed and the character contour curves were outlined with a raster image editor. Then, the character surface was created using curve grids and straight lines. The surface was processed using bridging, cropping, extrusion, chamfering, and smoothing commands in conjunction with modifiers to complete the production of the character model.

[0025] Optionally, the thoracoscopic surgery video module uses a mobile storage device to connect the mobile storage device to a mobile thoracoscopic device to collect real thoracoscopic surgery video data from the Department of Thoracic Surgery of the National Center for Respiratory Medicine, and records and obtains different thoracoscopic surgery videos according to the classification of thoracoscopic surgery and the different surgical approaches corresponding to each category. The surgical categories are divided into lung wedges, lobes, segments, subsegments, and whole lungs; wherein the wedges include: left upper lobe wedge, left lower lobe wedge, left upper lobe wedge Lobe + lower lobe wedge, right upper lobe wedge, right middle lobe wedge, right lower lobe wedge, right upper, middle and lower lobe combined wedge; lobes include right upper lobe, right middle lobe, right lower lobe, left upper lobe and left lower lobe; segments include left apical posterior segment, left anterior segment, left upper and lower lingual segments, left dorsal segment, right apical tip, right posterior end, right anterior segment, right lateral segment, right medial segment, right anterior basal segment, right posterior basal segment, right lateral basal segment and right medial basal segment; 2D thoracoscopic surgery video resources of various surgical methods are collected;

[0026] Reset the 2D thoracoscopic surgery video processing in the 2D thoracoscopic surgery video resources to 3D thoracoscopic surgery video using virtual reality technology.

[0027] Optionally, the processing and resetting of the 2D thoracoscopic surgery video in the 2D thoracoscopic surgery video resource into a 3D thoracoscopic surgery video using virtual reality technology includes:

[0028] Using structural reconstruction technology to extract scene depth image information from 2D thoracoscopic surgery videos in 2D thoracoscopic surgery video resources, obtain corresponding depth images, and perform initialization processing, wherein the initialization processing includes smoothing, noise reduction, and edge sharpening processing;

[0029] Filling virtual viewpoint holes using DIBR depth image rendering technology on the initialized depth image; the hole filling includes drawing and reconstructing the virtual viewpoint;

[0030] The images after the virtual viewpoint holes are filled are verified based on the image synthesis algorithm to generate 3D thoracoscopic gesture videos.

[0031] In the embodiment of the present invention, the advantages of VR technology are used to truly feedback the actual situation of thoracoscopic surgery clinical practice, and effective training can be carried out on the key points of each step of the surgical process; by providing VR thoracoscopic surgery nurse cooperation training, the interactive teaching mode can be used in the training process to increase participation, improve learning interest and effect, thereby ensuring the quality and effect of teaching; by using VR technology, the limitations of time and space can be broken, and learning and remote training can be realized anytime and anywhere; it can help regional medical centers to use VR professional training tools to conduct long-distance multi-center training. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 It is a schematic diagram of the structure of an interactive thoracoscopic surgery nursing training system using virtual reality technology in an embodiment of the present invention;

[0034] Figure 2 It is a schematic diagram of the structural composition of the server in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] For example, see Figure 1 , Figure 1 It is a schematic diagram of the structure of an interactive thoracoscopic surgery nursing training system using virtual reality technology in an embodiment of the present invention.

[0037] like Figure 1 As shown, an interactive thoracoscopic surgery nursing training system using virtual reality technology, the system includes: a teacher's intelligent terminal 11 for teaching management, a server 12, and several VR mobile devices 13 at the student end, the teacher's intelligent terminal 11 and the VR mobile device 13 are connected to the server 12 based on the Internet, after the teacher's intelligent terminal 11 is connected to the server 12, the nursing training instructions and nursing assessment instructions are issued on the server 12, and the students log in to the server 12 through identity authentication through the VR mobile device 13; wherein the server 12 contains a functional subsystem and a scene simulation subsystem;

[0038] Students use the VR mobile 13 device to select thoracic surgery nursing training or thoracic surgery nursing assessment on the functional subsystem in the server 12, and the functional subsystem performs corresponding scene simulation in the scene simulation subsystem according to the student's selection.

[0039] The above system can enable students to experience the thoracoscopic surgery coordination process realistically, increase participation, improve learning interest, ensure teaching quality and effectiveness, and at the same time use this technology to realize remote training and achieve long-distance multi-center training effects.

[0040] In the specific implementation process of the present invention, the functional subsystem includes a login module, a training module, an assessment module, a statistical analysis module and an interactive function module; the login module is the external port of the server, which is used for registration and login of student users and loading of teaching and training resources for teacher users; the training module is for the student user to enter the server through the login module, and to select 3D virtual reality real surgical resources in the scene simulation subsystem for operation guidance learning and training; the assessment module is for assessing the student user's mastery of thoracoscopic surgery cooperation knowledge and cooperation process, and conducting an evaluation test on the student user. The teacher user sends an assessment instruction through the server, and selects the video in the scene simulation subsystem to send it to the corresponding student user. After logging in through a VR mobile device, the student user directly enters the assessment module for a simulated assessment; the statistical analysis module receives data through the background of the server, and counts the training and / or assessment data of each student user, and stores and analyzes them; the interactive function module is a module that provides text and / or voice interaction.

[0041] Furthermore, the login module uses CANVAS curtain technology to draw a login and registration interface, and the login and registration interface is drawn by dragging and setting coordinates through UGUI components to form input boxes, buttons and text components; then use C# scripts to bind the account and password input box components in the login and registration interface, and read the data entered by the student user, use MySql database technology to log in and store the account and password information entered by the student user, and query and verify or store the login user information in the database; after the student user enters the server, use C# scripts to control the loading of resources and scenes, read the real-time loading amount of resources, use the slider in UGUI, and display the loading progress in the simulation interface, and enter the corresponding thoracoscopic surgery with teaching resources.

[0042] Furthermore, the assessment module is used for uploading and resetting test questions, and testing and assessing test questions; wherein the uploading and resetting of test questions are achieved by establishing a test question table and a question table in an EXCEL table, each question contains question information and question selection items, and then the C# code reads the data in the corresponding format and stores it in the MySql database, and the teacher user's account is used to manage the uploading and resetting of test questions, and the teacher user is required to directly delete the question information in the MySql database by selecting the corresponding test question. The test and assessment function of the test question utilizes the drawing board, text, and check box components in the UGUI to layout the answering interface, and utilizes the interactive function to implement the assessment at each node of thoracoscopic surgery cooperation, and the test results are presented in the form of a score table, which includes the score number ID, student ID, test question ID, score, and score grade, and the assessment results are saved in the MySql database.

[0043] Furthermore, the interactive function module is used to perform linear planning of video nodes for the real 3D thoracoscopic surgical video according to the surgical process and the required cooperation points, and to clarify the interaction points based on the linear planning results; for the interaction points, the text and voice interaction content corresponding to the interaction points required for the cooperation of the surgical nurses is created according to the actual surgical cooperation situation; the text and voice interaction content corresponding to the interaction points are inserted into the real 3D thoracoscopic surgical video in the form of text boxes and voices for prompt processing.

[0044] Specifically, the functional subsystem includes a login module, a training module, an assessment module, a statistical analysis module and an interactive function module.

[0045] Among them, the login module is the external port for entering the server. The login module includes student user registration, student user login, and loading of teaching and training resources. Students register with real-name information, and the login module sends the registered information to the server for automatic inspection. If the information does not exist, it will be stored and retained in the background database. If it does exist, the interface will prompt that it has been registered and prompt to log in. When logging in, students need to enter their account and password, log in to the server in the background and determine whether the student user account information exists in the data management library. At the same time, check whether the password is correct. Only when all are correct can they obtain permission to enter the server. After entering the server, the student user enters the loading interface and enters the corresponding thoracoscopic surgery cooperation teaching resources.

[0046] That is, the CANVAS curtain technology is used to draw the login and registration interface. The registration interface is drawn by dragging and setting the coordinates through the UGUI component to make Input Field (input box), Button (button) and Text (text) components, and then the C# script is used to bind the account and password input box components in the login and registration interface, and read the data entered by the student user. The MySql database technology is used to log in and store the account, password and other information entered by the user, and query and verify or store the login information in the database to realize the student user's login and registration function for the system.

[0047] After students enter the server, the technical points for implementing the loading interface include: using C# scripts to control the loading of resources and scenes, reading the real-time loading amount of resources, using the slider in UGUI to display the loading progress in the simulation interface, and then entering the corresponding thoracoscopic surgery teaching resources.

[0048] The training module is that after students log in to the server, they select the 3D virtual reality real surgery resources in the thoracoscopic surgery video module to learn the operation guidance.

[0049] The function of the assessment module is mainly to assess students' mastery of thoracoscopic surgery knowledge and coordination processes, and to conduct evaluation tests on users. Teachers send instructions through the background management system, select the video in the thoracoscopic surgery video module, and send it to the student end. The student end logs in through the login module, enters the assessment module, and conducts a simulated assessment.

[0050] The technical implementation of the assessment module includes test question uploading and resetting, and test question testing and assessment. The test questions are uploaded and reset by creating a test question table and a question table in the EXCEL table. Each question contains question information and question selection items. The C# code then reads the data in the corresponding format and stores it in the MySql database. The administrator's account is used to manage the upload and reset of the test questions. It is necessary to further implement the reset, that is, to directly delete the question information in the database by selecting the corresponding test question through the administrator account. The test and assessment function of the test questions uses the drawing board, text, and check box components in the UGUI to layout the answering interface, and uses the interactive function to implement the assessment at each node of thoracoscopic surgery cooperation. The test results are presented in the form of a score table, which stores the results of each user's test, including the score number ID, student ID, test question ID, score, and score level. The corresponding assessment results are also saved through the background database.

[0051] The statistical analysis module receives data through the background, counts the training and assessment status of each student, and stores and analyzes any problems that arise.

[0052] The interactive function module provides a text and voice interactive function module to increase student participation, enhance learning interest and effect, thereby ensuring teaching quality and effect.

[0053] The interactive function module is implemented by providing a voice + text interactive module in different thoracoscopic cooperation nodes. The technical solution is to use computer and network technology to integrate the interactive experience into the linear video, allowing interaction through the display of text and voice in the conversation window, so that students can improve their viewing experience and participation. The specific form includes the above-mentioned 3D thoracoscopic real surgery video, according to the surgeon's surgical process around the video linear node representing the progress of the operation, according to the progress of different types of operations, the required nurses' cooperation points are used for text and voice windows, interactive questions and selection of linear video nodes representing the surgical process, and the cooperation points such as instruments and materials required by nurses in real scenes are interactively asked questions to achieve the real training purpose. The specific plan includes: linear planning of video nodes according to the surgical process and the required cooperation points for the 3D thoracoscopic real surgery video, and clarifying the interactive points. For the interactive points of the 3D thoracoscopic real surgery video, according to the real surgical cooperation situation, create the text and voice interactive content required for the cooperation of the surgical nurses. The text and voice interactive content of the surgical process video nodes are inserted into the 3D thoracoscopic surgery video in the form of text boxes and voices to provide prompts, so that simulated interaction can be achieved in the video. For the surgical video cooperation points, when the student user finds the interaction point and interacts, he will proceed to the next surgical cooperation interaction point, and finally complete the entire thoracoscopic surgery cooperation and simulation training.

[0054] In the specific implementation process of the present invention, the scene simulation subsystem includes a surgical scene simulation module and a thoracoscopic surgery video module; the surgical scene simulation module is used to simulate the scene conditions of real surgery, and the scene conditions include the scene conditions of the patient, surgeon, anesthesiologist, assistant, mirror holder, and instrument nurse before the operation begins; the thoracoscopic surgery video module is used to simulate the real situation of the surgeon performing thoracoscopic surgery during thoracoscopic surgery; the real situation is produced by a real thoracoscopic surgery recording video.

[0055] Furthermore, when the surgical scene simulation module simulates the construction of the scene of a real surgery, 3dsMax 3D software is used for 3D reconstruction, and the real objects in the operating room scene are restored, and the scene model is surface rendered or textured; when the character model in the simulation of the real surgery is constructed, the personnel situation before the operation is simulated, including the character simulation of the patient in the lateral position before the operation, the surgeon, the anesthesiologist, the assistant, the mirror holder, and the instrument nurse; wherein the instrument nurse is a perspective character model and can perform free perspective activities in the surgical scene; when the multimedia drawing board in the simulation of the real surgery is constructed, the position of the thoracoscopic machine is simulated to play the real 3D thoracoscopic surgery cooperation video; through VideoManager, the script is implemented using the message mechanism processing framework implemented in virtual reality.

[0056] Furthermore, when building the character model in the simulation of real surgery, it also includes: using NURBS surface modeling technology, taking photos of the surgeon, anesthesiologist, assistant, mirror holder, and instrument nurse respectively, and then importing the photos into the 3DS Max three-dimensional view, fixing the photos, and using a raster image editor to outline the character contour curves, and then using curve grids and straight lines to create character role surfaces, and then using bridging, cropping, extrusion, chamfering, and smoothing commands in conjunction with modifiers to process the surfaces to complete the production of the character model.

[0057] Furthermore, the thoracoscopic surgery video module uses a mobile storage device to connect the mobile storage device to a mobile thoracoscopic device to collect real thoracoscopic surgery video data from the Department of Thoracic Surgery of the National Center for Respiratory Medicine, and records and obtains different thoracoscopic surgery videos according to the classification of thoracoscopic surgery and the different surgical approaches corresponding to each category. The surgical categories are divided into lung wedges, lobes, segments, subsegments, and whole lungs; among them, the wedges include: left upper lobe wedge, left lower lobe wedge, left upper lobe + lower lobe wedge, right upper lobe wedge, right middle lobe wedge, right Lower lobe wedge, right upper, middle and lower lobe combined wedge; lobes include right upper lobe, right middle lobe, right lower lobe, left upper lobe and left lower lobe; lung segments include left apical posterior segment, left anterior segment, left upper and lower lingular segments, left dorsal segment, right tip, right posterior end, right anterior segment, right lateral segment, right medial segment, right anterior basal segment, right posterior basal segment, right lateral basal segment and right medial basal segment; 2D thoracoscopic surgery video resources of various surgical methods are obtained; 2D thoracoscopic surgery video resources are processed and reset into 3D thoracoscopic surgery videos using virtual reality technology.

[0058] Furthermore, the processing of resetting the 2D thoracoscopic surgery video in the 2D thoracoscopic surgery video resource to the 3D thoracoscopic surgery video of virtual reality technology includes: extracting the depth image information of the scene from the 2D thoracoscopic surgery video in the 2D thoracoscopic surgery video resource using the structural reconstruction technology to obtain the corresponding depth image, and performing initialization processing, wherein the initialization processing includes smoothing, noise reduction and edge sharpening processing; filling the virtual viewpoint holes in the depth image after the initialization processing using the DIBR depth image rendering technology; the hole filling includes drawing and reconstructing the virtual viewpoint; and verifying the image after the virtual viewpoint hole is filled based on the image synthesis algorithm to generate a 3D thoracoscopic gesture video.

[0059] Specifically, the scene simulation subsystem includes a surgical scene simulation module and a thoracoscopic surgery video module; the surgical scene simulation module is used to simulate real surgical scene conditions, including the simulated positions of patients, surgeons, anesthesiologists, assistants, scope holders, and instrument nurses before the start of the operation; the thoracoscopic surgery video module simulates the real situation of the surgeon performing thoracoscopic surgery during thoracoscopic surgery. It is produced from real thoracoscopic surgery recorded videos and is selected for use in the above-mentioned training and assessment.

[0060] The surgical scene simulation module is realized through 3D modeling technology, simulating the real operating room scene simulation and thoracoscopic surgery process simulation. Its technical content includes the construction of scene models and scene optimization and integration; scene model construction includes real surgical scene simulation construction and thoracoscopic surgery video production; real surgical scene construction covers the room structure, characters, and multimedia drawing boards of the operating room.

[0061] The construction of the surgical site structure is used to simulate the real operating room scene. By using 3ds Max 3D software to implement 3D reconstruction technology, the real objects in the operating room scene are restored, and the scene model is surface rendered or mapped to realize the construction of the surgical site structure.

[0062] The character model is built to simulate the personnel situation before the operation begins, including the surgical patient in the side-lying position, the surgeon, the anesthesiologist, the assistant, the mirror hand, and the instrument nurse. The instrument nurse is a perspective character model that can perform free perspective activities in the surgical scene. This is to give users a more realistic experience. It is achieved by the derivation of the close-up observation algorithm. A further technical solution is to use NURBS surface modeling technology to take photos of the surgeon, anesthesiologist, assistant, mirror hand, and instrument nurse, and then import the pictures into the 3DS Max three-dimensional view, fix the pictures, and use the raster image editor to outline the character contour curve, and then use the curve grid and straight lines to create the character role surface, and use the bridge, crop split, extrude, chamfer, smooth and other commands with the modifier to process the surface to complete the production of the character model.

[0063] The multimedia drawing board is used to simulate the position and display function of the thoracoscopic machine and to play a real 3D thoracoscopic surgery cooperation video. A further technical solution is to implement the script through VideoManager using the message mechanism processing framework implemented in virtual reality.

[0064] The thoracoscopic surgery video module simulates the actual situation of the surgeon performing thoracoscopic surgery during thoracoscopic surgery. It is produced from real thoracoscopic surgery recordings. Its technical points include the production of VR-based 3D videos. Technical points include the acquisition of thoracoscopic surgery videos and the production of thoracoscopic surgery virtual reality 3D videos.

[0065] The technical solution for acquiring real thoracoscopic surgery videos is as follows: using a mobile storage device, the mobile storage device is connected to a mobile thoracoscope to collect real thoracoscopic surgery video data from the Department of Thoracic Surgery of the National Center for Respiratory Medicine, and different thoracoscopic surgery videos are recorded and acquired according to the classification of thoracoscopic surgery and the different surgical approaches corresponding to each category. The surgical categories are divided into lung wedge, lung lobe, lung segment, lung subsegment, and whole lung. Among them, wedges include: left upper lobe wedge, left lower lobe wedge, left upper lobe + lower lobe wedge, right upper lobe wedge, right middle lobe wedge, right lower lobe wedge, and right upper, middle and lower lobe combined wedge; lobes include right upper lobe, right middle lobe, right lower lobe, left upper lobe, and left lower lobe; lung segments include left apical posterior segment, left anterior segment, left upper and lower lingual segments, left dorsal segment, right apical tip, right posterior end, right anterior segment, right lateral segment, right medial segment, right anterior basal segment, right posterior basal segment, right lateral basal segment, and right medial basal segment. Finally, real 2D thoracoscopic surgery video resources of various surgical methods are summarized.

[0066] The production of 3D virtual reality video of thoracoscopic surgery is a technical solution for processing and resetting the real 2D surgical video obtained above into a 3D video using virtual reality technology:

[0067] (1) Use structural reconstruction technology to extract the depth image information of the scene from the original thoracoscopic 2D video, obtain the corresponding depth map, and perform initialization processing;

[0068] (2) Further use the obtained depth image for post-processing smoothing, noise reduction, and edge sharpening to improve image quality;

[0069] (3) Based on the DIBR deep image rendering technology, the virtual viewpoint holes are filled, and the virtual viewpoints are further drawn and reconstructed. The acquired thoracoscopic surgery video images are occluded by using the hole filling method.

[0070] (4) Utilize the image synthesis algorithm to generate a 3D thoracoscopic surgery video.

[0071] (5) The various 3D thoracoscopic surgery videos produced are uploaded to the video server through the transmission module of the teaching platform with the help of hypertext transfer protocol (HTTP).

[0072] Scene optimization and integration include exporting the prepared model to FBX format, importing it into Unity3D, and finally adjusting the size, position, direction, etc. of the model to complete the integration of the scene.

[0073] Using the Prefabs folder in Unity3D, all the models are copied into the folder, and the above 3D models are placed in the virtual coordinate space. The models in the space are scaled to a reasonable size to establish the preliminary form of the entire surgical scene. The relative spatial coordinates between the models are further adjusted, and the surgical patient model is fixed in the middle of the operating table in the lateral position. The surgeon and the mirror holder are on the anterior ventral side of the patient model in the lateral position, the surgical assistant is on the dorsal side of the patient model in the lateral position, and the instrument nurse is at the back as the perspective character model. The multimedia drawing board is located in front of the visual field to simulate the position of the thoracoscopic machine for playing thoracoscopic surgery videos.

[0074] First, teachers and students register their personal information on the login module through the mobile terminal. The login module sends the registered information to the server system for automatic inspection. If the information does not exist, it will be stored and retained in the background database. If it does exist, the interface prompts that it has been registered and prompts to log in. When logging in, students need to enter their account password. The background login system determines whether the student user account information exists in the data management library, and at the same time checks whether the password is correct. Only when all are correct can they obtain permission to enter the system; after the student user enters the system, he enters the loading interface and enters the corresponding teaching training and assessment module. During training, students log in and enter the system. Through the VR mobile terminal, they can select videos in the real thoracoscopic surgery module according to their learning needs, and select the videos they need for simulation training and assessment. The optional modules include lung wedges, lobes, segments, subsegments, and whole lungs. Wedges include: left upper lobe wedge, left lower lobe wedge, left upper lobe + lower lobe wedge, right upper lobe wedge, right middle lobe wedge, right lower lobe wedge, right upper, middle and lower lobe combined wedge; lobes include right upper lobe, right middle lobe, right lower lobe, left upper lobe, left lower lobe; segments include left apex posterior segment, left anterior segment, left upper and lower lingual segments, left dorsal segment, right apex, right posterior end, right anterior segment, right lateral, right medial segment, right anterior basal segment, right posterior basal segment, right lateral basal segment, right medial basal segment, and select After the selection, the simulation training will start, and the loading scene will be entered. The first interface, the operating room scene simulation, will display the real surgical scene. The scene includes the simulated positions of the patient, surgeon, anesthesiologist, assistant, mirror holder, and instrument nurse before the operation. The voice + text interaction is displayed in the interface. The introduction includes: ① surgical staff positions; ② patient position; ③ anesthesia method; ④ medical history introduction; ⑤ planned surgical method. This step guides students to learn the cooperation before the start of thoracoscopic surgery. For further learning, the multimedia drawing board in the scene module simulates the position and display function of the thoracoscopic machine, and starts playing the 3D thoracoscopic surgery cooperation video selected in the thoracoscopic surgery module. Training is also conducted at different nodes of the operation, using text boxes and voice to conduct question-based training at cooperation points such as when dealing with lung fissures, arteries, veins, bronchi, removing lung pathology, clearing lymph nodes, verifying whether there is lung leakage, closing the incision, and at the end of the operation: "Question format: This node performs XX operation, and the required cooperation includes: Option A... Option B... Option C... Option D..." After the simulation training is completed, the statistical analysis module will be entered, and the background data will be used to count the interactive cooperation of students at each cooperation node and stored by the server system. The teacher can use the statistical analysis module to discover the students' learning cooperation and mastery, the problems that arise, and conduct statistical analysis to guide subsequent teaching decisions.

[0075] In the embodiment of the present invention, the advantages of VR technology are used to truly feedback the actual situation of thoracoscopic surgery clinical practice, and effective training can be carried out on the key points of each step of the surgical process; by providing VR thoracoscopic surgery nurse cooperation training, the interactive teaching mode can be used in the training process to increase participation, improve learning interest and effect, thereby ensuring the quality and effect of teaching; by using VR technology, the limitations of time and space can be broken, and learning and remote training can be realized anytime and anywhere; it can help regional medical centers to use VR professional training tools to conduct long-distance multi-center training.

[0076] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0077] In addition, the above is a detailed introduction to an interactive thoracoscopic surgery nursing training system using virtual reality technology provided by an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An interactive thoracoscopic surgery nursing training system based on virtual reality technology, characterized in that: The system includes: a teacher's smart terminal for teaching management, a server, and several VR mobile devices on the student side. The teacher's smart terminal and the VR mobile device are connected to the server based on the Internet. After the teacher's smart terminal is connected to the server, it publishes nursing training instructions and nursing assessment instructions on the server. Students log in to the server through identity authentication using VR mobile devices; wherein the server contains a functional subsystem and a scene simulation subsystem; The student selects thoracic surgery nursing training or thoracic surgery nursing assessment on the functional subsystem in the server through a VR mobile device, and the functional subsystem performs corresponding scene simulation in the scene simulation subsystem according to the student's selection; The scene simulation subsystem includes a surgery scene simulation module and a thoracoscopic surgery video module; The surgical scene simulation module is used to simulate the scene of a real surgery, including the scene of the patient, surgeon, anesthesiologist, assistant, mirror holder, and instrument nurse standing before the operation begins; The thoracoscopic surgery video module is used to simulate the actual situation of the surgeon performing thoracoscopic surgery during thoracoscopic surgery; the actual situation is produced by a real thoracoscopic surgery recording video; When simulating the scene of a real surgery, the surgical scene simulation module uses 3ds Max 3D software to perform 3D reconstruction, restore the real objects in the operating room scene, and perform surface rendering or mapping on the scene model; When constructing the character models used in the simulation of real surgery, the personnel situation before the surgery is simulated, including the character simulation of the patient in the side-lying position before the surgery, the surgeon, the anesthesiologist, the assistant, the mirror holder, and the instrument nurse; wherein the instrument nurse is a perspective character model and can perform free perspective activities in the surgical scene; When constructing the multimedia drawing board in the simulation of real surgery, the position of the thoracoscopic machine is simulated to play the real 3D thoracoscopic surgery cooperation video; the script is implemented through VideoManager using the message mechanism processing framework implemented in virtual reality; The thoracoscopic surgery video module uses a mobile storage device to connect the mobile storage device to a mobile thoracoscopic device to collect real thoracoscopic surgery video data from the Department of Thoracic Surgery of the National Center for Respiratory Medicine, and records and obtains different thoracoscopic surgery videos according to the classification of thoracoscopic surgery and the different surgical approaches corresponding to each category. The surgical categories are divided into lung wedges, lobes, segments, subsegments, and whole lungs; among them, the wedges include: left upper lobe wedge, left lower lobe wedge, left upper lobe wedge + Lower lobe wedge, right upper lobe wedge, right middle lobe wedge, right lower lobe wedge, right upper, middle and lower lobe combined wedge; lobes include right upper lobe, right middle lobe, right lower lobe, left upper lobe and left lower lobe; segments include left apical posterior segment, left anterior segment, left upper and lower lingual segments, left dorsal segment, right apex, right posterior end, right anterior segment, right lateral segment, right medial segment, right anterior basal segment, right posterior basal segment, right lateral basal segment and right medial basal segment; 2D thoracoscopic surgery video resources of various surgical methods are collected; Reset the 2D thoracoscopic surgery video processing in the 2D thoracoscopic surgery video resources to 3D thoracoscopic surgery video using virtual reality technology; The method of resetting the 2D thoracoscopic surgery video processing in the 2D thoracoscopic surgery video resource to a 3D thoracoscopic surgery video using virtual reality technology includes: Using structural reconstruction technology to extract scene depth image information from 2D thoracoscopic surgery videos in 2D thoracoscopic surgery video resources, obtain corresponding depth images, and perform initialization processing, wherein the initialization processing includes smoothing, noise reduction, and edge sharpening processing; Filling virtual viewpoint holes using DIBR depth image rendering technology on the initialized depth image; the hole filling includes drawing and reconstructing the virtual viewpoint; The images after the virtual viewpoint holes are filled are verified based on the image synthesis algorithm to generate 3D thoracoscopic gesture videos.

2. The interactive thoracoscopic surgery nursing training system according to claim 1 is characterized in that: The functional subsystem includes a login module, a training module, an assessment module, a statistical analysis module and an interactive function module; The login module is the external port of the server, which is used for registration and login of student users and loading of teaching and training resources for teacher users; The training module is for the student user to enter the server through the login module and select 3D virtual reality real surgery resources in the scene simulation subsystem for operation guidance learning and training; The assessment module is used to assess the student user's mastery of thoracoscopic surgery cooperation knowledge and cooperation process, and to conduct an evaluation test on the student user. The teacher user sends an assessment instruction through the server, and selects a video in the scene simulation subsystem to send to the corresponding student user. After logging in through a VR mobile device, the student user directly enters the assessment module for a simulated assessment; The statistical analysis module receives data through the background of the server, and collects data on the training and / or assessment of each student user, and stores and analyzes the data; The interactive function module is a module that provides text and / or voice interaction.

3. The interactive thoracoscopic surgery nursing training system according to claim 2 is characterized in that: The login module uses CANVAS canvas technology to draw a login registration interface. The login registration interface is drawn by dragging and setting coordinates through UGUI components to form input boxes, buttons and text components; then use C# scripts to bind the account and password input box components in the login registration interface, and read the data entered by the student user, use MySql database technology to log in and store the account and password information entered by the student user, and query and verify or store the login user information in the database; After the student user enters the server, the C# script is used to control the loading of resources and scenes, the real-time loading amount of resources is read, and the sliding bar in the UGUI is used to display the loading progress in the simulation interface. After entering, the corresponding thoracoscopic surgery is used for teaching resources.

4. The interactive thoracoscopic surgery nursing training system according to claim 2 is characterized in that: The assessment module is used for uploading and resetting test questions, and testing and assessing test questions; wherein the uploading and resetting of test questions are achieved by establishing a test question table and a question table in an EXCEL table, each question contains question information and question selection items, and then the C# code reads the data in the corresponding format and stores it in the MySql database, and the teacher user's account is used to manage the uploading and resetting of test questions, and the teacher user is required to directly delete the question information in the MySql database by selecting the corresponding test question. The test and assessment function of the test question utilizes the drawing board, text, and check box components in the UGUI to layout the answering interface, and utilizes the interactive function to implement the assessment at each node of thoracoscopic surgery cooperation, and the test results are presented in the form of a score table, which includes the score number ID, student ID, test question ID, score, and score grade, and the assessment results are saved in the MySql database.

5. The interactive thoracoscopic surgery nursing training system according to claim 2 is characterized in that: The interactive function module is used to perform linear programming of video nodes for the real 3D thoracoscopic surgery video according to the surgical process and the required coordination points, and to identify the interactive points based on the linear programming results; For the interactive points, based on the actual surgical cooperation situation, create text and voice interactive content corresponding to the interactive points required for surgical nurses to cooperate; The text and voice interactive contents corresponding to the interactive points are inserted into the real 3D thoracoscopic surgery video for prompt processing in the form of text boxes and voices.

6. The interactive thoracoscopic surgery nursing training system according to claim 1, characterized in that: When the character model in the simulation of real surgery is constructed, it also includes: Through NURBS surface modeling technology, photos of the surgeon, anesthesiologist, assistant, mirror holder, and instrument nurse were taken respectively, and then imported into the 3DS Max three-dimensional view. The photos were fixed and the character contour curves were outlined with a raster image editor. Then, the character surface was created using curve grids and straight lines. The surface was processed using bridging, cropping, extrusion, chamfering, and smoothing commands in conjunction with modifiers to complete the production of the character model.

Images