Natural orifice operation control system, operation control method and computer equipment

Abstract

The embodiment of the invention is suitable for the technical field of computer-aided medical technology and natural orifice surgery, and provides a natural orifice surgery control system, a surgery control method and computer equipment, the system comprises an image processing module used for identifying a tissue structure in an intraoperative endoscope image to obtain image identification information; the path navigation module is used for planning the movement path of the instrument in the natural orifice according to the image identification information and dynamically adjusting the movement path in the movement process of the instrument; the mechanical arm control module is used for controlling the instrument to move according to the movement path and controlling the instrument to execute corresponding surgical operation in the surgical area after the instrument moves to the surgical area; wherein the operation area is marked and displayed in the intraoperative endoscope image by the image processing module according to the image identification information. By adopting the system, the operation precision of the natural orifice surgery can be improved.

Description

TECHNICAL FIELD

[0001] The embodiment of the present application belongs to the technical field of computer-aided medical technology and natural orifice surgery technology, and particularly relates to a natural orifice surgery control system, a surgery control method and a computer device. BACKGROUND

[0002] Natural orifice surgery refers to natural orifice translumenal endoscopic surgery (NOTES), which is a minimally invasive surgery method of performing endoscopic operation in the human body through natural cavities of the human body such as the oral cavity and the nasal cavity. Compared with traditional endoscopic surgery, natural orifice surgery has the advantages of small trauma, fast postoperative recovery and hidden scars. However, the operation space of natural orifice surgery is small, the vision is limited, and the instrument control is difficult, so the surgical risk is also correspondingly increased, and the surgeon needs to have very high technical proficiency to successfully complete such surgery.

[0003] At present, the surgeon mainly operates the endoscope and surgical instruments in a manual manner during natural orifice surgery, and the surgical effect is closely related to the operation experience of the surgeon, so it is difficult to ensure the precision of the surgical operation. SUMMARY

[0004] Therefore, the embodiment of the present application provides a natural orifice surgery control system, a surgery control method and a computer device to improve the operation precision of natural orifice surgery.

[0005] The first aspect of the embodiment of the present application provides a natural orifice surgery control system, comprising:

[0006] An image processing module, configured to identify tissue structures in an intraoperative endoscope image and obtain image recognition information;

[0007] A path navigation module, configured to plan a movement path of an instrument in a natural orifice according to the image recognition information, and dynamically adjust the movement path in the process of movement of the instrument;

[0008] A mechanical arm control module, configured to control the instrument to move according to the movement path, and after the instrument moves to a surgical area, control the instrument to perform a corresponding surgical operation in the surgical area; wherein the surgical area is marked and displayed in the intraoperative endoscope image by the image processing module according to the image recognition information.

[0009] Optionally, the intraoperative endoscope image comprises a natural orifice image, the image recognition information comprises a natural orifice inner wall, and the path navigation module is specifically configured to:

[0010] Based on the natural cavity inner wall identified in the natural cavity image, a motion path of the instrument in the natural cavity is planned, and the motion path is prompted in the natural cavity image; wherein the distance between the motion path and any side of the natural cavity inner wall is greater than or equal to a distance threshold.

[0011] Optionally, the path navigation module is further configured to:

[0012] In the process of the motion of the instrument in the natural cavity, the position of the instrument tip is monitored in real time.

[0013] When the instrument tip moves to a distance between the instrument tip and any side of the natural cavity inner wall being less than the distance threshold or when the angle deviation between the motion direction of the instrument and the forward direction of the planned motion path is greater than an angle threshold, motion path correction information is generated, and the motion path is corrected according to the motion path correction information.

[0014] Optionally, the natural cavity surgery control system further comprises:

[0015] A safety monitoring module is configured to determine a deformation amount of a contact area according to the intraoperative endoscope image when the instrument contacts the tissue in the patient's body, and to perform a safety warning on the current surgical operation when the deformation amount is greater than a preset deformation threshold; wherein the contact information between the instrument and the tissue in the patient's body is provided to the safety monitoring module by the image processing module.

[0016] Optionally, the natural cavity surgery control system further comprises:

[0017] A user interaction module is configured to receive a user operation instruction and control related modules based on the user operation instruction; wherein the user operation instruction includes any one or more of a foot pedal operation instruction, a gesture operation instruction and a voice operation instruction; and the related modules include the image processing module and the mechanical arm control module.

[0018] Optionally, the user interaction module is specifically configured to:

[0019] In response to the voice operation instruction, the image processing module is controlled to mark the surgical area in the intraoperative endoscope image, and the marked intraoperative endoscope image is displayed through a display unit.

[0020] Optionally, the natural cavity surgery control system further comprises:

[0021] An instrument identification module is configured to read identity information of the instrument that has been connected, and verify the instrument according to the identity information; in a case where the instrument verification is passed, control information matched with the instrument is acquired and transmitted to the robot arm control module, so as to instruct the robot arm control module to control the instrument according to the control information, wherein the control information comprises a kinematic model matched with the instrument.

[0022] Optionally, the instrument identification module is specifically configured to:

[0023] verify, according to the identity information, an adaptation relationship of the instrument with a current natural orifice surgery, a service life of the instrument, and compliance information of the instrument;

[0024] in a case where the instrument is adapted to the current natural orifice surgery, the instrument is within a safe service life, and the instrument is a standard instrument, determine that the instrument passes the verification.

[0025] A second aspect of the embodiments of the present application provides a natural orifice surgery control method, comprising:

[0026] identifying a tissue structure in an intraoperative endoscope image to obtain image recognition information;

[0027] planning a motion path of an instrument in a natural orifice according to the image recognition information, and dynamically adjusting the motion path in a process of motion of the instrument;

[0028] controlling the instrument to move according to the motion path, and marking a surgical region in the intraoperative endoscope image according to the image recognition information after the instrument moves to the surgical region;

[0029] controlling the instrument to perform a corresponding surgical operation in the surgical region.

[0030] A third aspect of the embodiments of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, when the processor executes the computer program, the computer device implements the following method realized by each module of the natural orifice surgery control system according to any one of the first aspect:

[0031] identifying a tissue structure in an intraoperative endoscope image to obtain image recognition information;

[0032] planning a motion path of an instrument in a natural orifice according to the image recognition information, and dynamically adjusting the motion path in a process of motion of the instrument;

[0033] controlling the instrument to move according to the movement path, and marking the surgical region in the intraoperative endoscope image according to the image recognition information after the instrument moves to the surgical region;

[0034] controlling the instrument to perform a corresponding surgical operation in the surgical region.

[0035] A fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is executed by a computer, the computer program implements the following method realized by each module of the natural orifice surgery control system according to any one of the first aspect:

[0036] identifying a tissue structure in an intraoperative endoscope image to obtain image recognition information;

[0037] planning a movement path of an instrument in a natural orifice according to the image recognition information, and dynamically adjusting the movement path in a process of movement of the instrument;

[0038] controlling the instrument to move according to the movement path, and marking the surgical region in the intraoperative endoscope image according to the image recognition information after the instrument moves to the surgical region;

[0039] controlling the instrument to perform a corresponding surgical operation in the surgical region.

[0040] A fifth aspect of the embodiment of the present application provides a computer program product, which includes a computer program, and when the computer program runs, the following method realized by each module of the natural orifice surgery control system according to any one of the first aspect is executed:

[0041] identifying a tissue structure in an intraoperative endoscope image to obtain image recognition information;

[0042] planning a movement path of an instrument in a natural orifice according to the image recognition information, and dynamically adjusting the movement path in a process of movement of the instrument;

[0043] controlling the instrument to move according to the movement path, and marking the surgical region in the intraoperative endoscope image according to the image recognition information after the instrument moves to the surgical region;

[0044] controlling the instrument to perform a corresponding surgical operation in the surgical region.

[0045] Compared with the prior art, the embodiment of the present application has the following beneficial effects:

[0046] The natural cavity operation control system provided in the embodiments of the present application can process the intraoperative image collected by the endoscope in real time, identify the tissue structure in the image as reference information for path planning and navigation and surgical operation, and assist the doctor in completing the natural cavity operation. The embodiments of the present application assist in surgical operation through the natural cavity operation control system, so that the doctor can pay more attention to the operation strategy rather than the operation details, can significantly improve the operation precision and reduce the operation complexity while ensuring the safety of the operation.

[0047] The natural cavity operation control system and the operation control method provided in the embodiments of the present application can be used for natural cavity operation, can assist the doctor in instrument navigation, automatic obstacle avoidance, visual enhancement and stable control operation through real-time analysis of the operation field image, reduce the influence of human factors such as the doctor's personal experience on the operation, and thus improve the operation precision and safety and the operation efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0049] Figure 1 is a schematic diagram of a natural cavity operation control system provided in the embodiments of the present application;

[0050] Figure 2 is a schematic diagram of mechanical arm motion control provided in the embodiments of the present application;

[0051] Figure 3 is a schematic diagram of surgical region marking display provided in the embodiments of the present application;

[0052] Figure 4 is a schematic diagram of another natural cavity operation control system provided in the embodiments of the present application;

[0053] Figure 5 is a schematic diagram of a natural cavity operation control method provided in the embodiments of the present application;

[0054] Figure 6 is a schematic diagram of a computer device provided in the embodiments of the present application. DETAILED DESCRIPTION

[0055] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0056] As described above, in the current natural orifice surgery, most of which rely on the experience of the surgeon, mainly using manual operation of endoscope and surgical instruments to complete the operation, which is low in efficiency and difficult to guarantee the precision and safety of the operation. In view of the above problems, the natural orifice surgery control system provided by the embodiments of the present application integrates image processing, surgical path planning and navigation, automatic posture control of surgical instruments and other functions to assist the surgeon in spatial navigation, image recognition and operation stability control during the natural orifice surgery, thereby improving the operation precision and efficiency of the natural orifice surgery under the premise of ensuring the safety of the operation.

[0057] The technical solutions of the present application will be described below through specific embodiments.

[0058] Referring to Figure 1 , a schematic diagram of a natural orifice surgery control system provided by the embodiments of the present application is shown, Figure 1 The control system 100 shown in the figure can specifically include an image processing module 102, a path navigation module 104 and a mechanical arm control module 106, wherein:

[0059] The image processing module 102 is configured to identify the tissue structure in the intraoperative endoscope image and obtain image recognition information.

[0060] The path navigation module 104 is configured to plan the movement path of the instrument in the natural orifice according to the image recognition information, and dynamically adjust the planned movement path during the movement of the instrument.

[0061] The mechanical arm control module 106 is configured to control the instrument to move according to the planned movement path, and after the instrument moves to the surgical area, control the instrument to perform the corresponding surgical operation in the surgical area. Wherein, the above surgical area can be marked and displayed in the intraoperative endoscope image by the image processing module 102 according to the identified image recognition information. That is, the image processing module 102 can determine the specific surgical area in the intraoperative endoscope image according to the image recognition information, and mark the surgical area in the above intraoperative endoscope image to prompt the surgeon to perform the surgical operation in the area.

[0062] In the embodiments of the present application, the intraoperative endoscopic image can be collected by using a camera module on an endoscope that enters the human body. When performing a natural orifice surgery, the endoscope and the surgical instrument can enter the natural orifice under the control of a mechanical arm. In this process, the endoscope will remain in a working state and collect images of the in-vivo environment including the natural orifice in real time. The image collected by the endoscope is the intraoperative endoscopic image, which can be transmitted to the image processing module 102 of the control system 100 through the communication module on the endoscope for processing. Unless otherwise specified, the instruments referred to in the embodiments of the present application are surgical instruments, and the intraoperative image is the intraoperative endoscopic image, i.e., the surgical instruments and instruments in the present application refer to the same technical features, and the intraoperative image and the intraoperative endoscopic image refer to the same technical features. Figure 1

[0063] The image processing module 102 in the control system 100 can process the received intraoperative endoscopic image, such as identifying the tissue structure, lesion or surgical area in the image, etc.

[0064] In a possible implementation manner of the embodiments of the present application, the image processing module 102 can process the intraoperative endoscopic image based on the Unet algorithm. Unet is a convolutional neural network architecture for image segmentation, designed for medical image segmentation. Its core features include U-shaped network structure, skip connection and feature fusion mechanism, which can achieve high-precision segmentation on small sample data sets. Before processing the intraoperative endoscopic image, the image processing module 102 can also pre-process the image, such as morphological optimization of the received intraoperative endoscopic image, to further improve the accuracy and precision of data processing. After completing image segmentation to determine the lesion area in the image, the image processing module 102 can also enhance the image through post-processing to highlight the significant area in the image. The embodiments of the present application do not limit the specific image processing manner performed by the image processing module 102.

[0065] In the embodiments of the present application, the image recognition information obtained by recognizing the intraoperative endoscopic image can include the shape and position of the tissue structure in the image, and can also include the relevant feature information of the lesion area, such as the specific position and size of the polyp in the image, etc. The image recognition information output by the image processing module 102 can be provided to other modules, such as the path navigation module 104.

[0066] The path navigation module 104 in the control system 100 can plan a movement path of the surgical instrument in the natural orifice according to the image recognition information, and guide the surgical instrument to move along the planned movement path. In addition, the path navigation module 104 can also dynamically adjust the movement path in real time during the movement of the surgical instrument to ensure the safety of the entire instrument movement.​

[0067] Exemplarily, the intraoperative endoscopic image can include a natural cavity image captured when the endoscope is in the natural cavity, and the image recognition information obtained by the image processing module 102 from the natural cavity image after the organization recognition and other processing can include information of the inner wall of the natural cavity. Therefore, the path navigation module 104 can plan a movement path of the surgical instrument in the natural cavity where the instrument is currently located based on the natural cavity inner wall recognized in the natural cavity image. In order to ensure the safety of the instrument movement process, for example, to ensure that the instrument movement will not cause damage to the inner wall of the cavity, the distance between the movement path planned by the path navigation module 104 and the inner wall of the natural cavity on any side can be greater than or equal to a certain distance threshold.

[0068] In the embodiment of the present application, the path navigation module 104 can plan the shortest and safest movement path in the constrained three-dimensional space based on the Dijkstra algorithm combined with the image depth information. The Dijkstra algorithm is an algorithm for calculating the shortest path from a fixed source point to all other vertices in a graph. Specifically, the path navigation module 104 can construct a three-dimensional space according to the depth information of the image to determine the position of the inner wall of the natural cavity and the advancing direction of the instrument. For example, the advancing direction of the instrument can be the direction corresponding to the maximum value of the depth information in the image. The movement path planned by the path navigation module 104 can start from the current position of the endoscope or the instrument, and plan the optimal movement path along the determined advancing direction while avoiding the direction towards the inner wall of the cavity.

[0069] In addition, after the intraoperative endoscopic image is transmitted to the control system 100, the control system 100 can display the image in real time through the display unit or the display module. After the path navigation module 104 plans the movement path, the path navigation module 104 can also prompt the movement path in the displayed intraoperative endoscopic image. For example, the advancing direction of the instrument is prompted in the natural cavity image to guide the instrument to advance in the direction along the planned path.

[0070] In the embodiment of the present application, the dynamic adjustment of the planned movement path by the path navigation module 104 during the instrument movement process can ensure that the instrument movement will not deviate from the specified direction or path. This process can be realized by monitoring the position of the instrument tip in real time.

[0071] Specifically, the path navigation module 104 can monitor the position of the instrument tip in real time during the movement of the instrument in the natural cavity. The monitoring of the position of the instrument tip by the path navigation module 104 can be based on the image recognition information output by the image processing module 102. Exemplarily, the image recognition information can include the position information of the instrument tip and the relevant information of the inner wall of the natural cavity where the instrument is currently located, and the path navigation module 104 can monitor the position of the instrument tip according to these information.

[0072] When the instrument tip moves to a distance from the inner wall of the natural cavity on either side that is less than the aforementioned distance threshold, or when the angle deviation between the direction of the movement of the instrument and the advancing direction of the planned movement path is greater than a certain angle threshold, the path navigation module 104 can generate movement path correction information and correct the planned movement path according to the movement path correction information. By correcting the movement path in real time, it can be strictly ensured that the instrument movement process will not damage the inner wall of the natural cavity. For example, it can be ensured that the instrument tip will not scratch the inner wall of the cavity, will not cause the inner wall of the cavity to be punctured or cause the inner wall to bleed, and the like.

[0073] In the embodiments of the present application, the movement of the surgical instrument and the endoscope in the natural cavity and the like in the human body environment can be realized under the operation of the mechanical arm. The mechanical arm control module 106 in the control system 100 can control the operation of the mechanical arm, guide the surgical instrument to move according to the planned movement path, and reach the designated surgical area.

[0074] When the surgical instrument moves to the surgical area, the mechanical arm control module 106 can control the surgical instrument to perform corresponding surgical operations in the current surgical area. For example, the mechanical arm control module 106 can operate the surgical instrument to remove polyps in the surgical area.

[0075] As Figure 2 shown is a schematic diagram of the movement control of the mechanical arm provided in the embodiments of the present application, Figure 2 shows an example of the process of controlling the corresponding surgical operation performed by the mechanical arm under the control of the aforementioned natural cavity surgery control system.

[0076] Referring to Figure 2The intraoperative endoscope image collected by the endoscope can be transmitted to the control system and processed by the image processing module 102 in the control system. According to the processing result, for example, the aforementioned image recognition information, the control system can transmit control instructions to the mechanical arm control module 106 through CAN communication. In this way, the mechanical arm control module 106 can control the mechanical arm to perform corresponding actions according to the received control instructions, for example, guide the instrument to move or perform corresponding surgical operations in the surgical area. The execution process of the above-mentioned actions can be realized by the teleoperation master hand based on the code value. In addition, the position and state of the mechanical arm during the operation can also be transmitted to the control system through CAN communication to determine the specific instructions for executing the next action. Figure 2 The mechanical arm shown in the figure can be a micro mechanical arm.

[0077] In the embodiment of the present application, when the surgical instrument moves to the surgical area, the image processing module 102 can mark the surgical area in the intraoperative image based on the recognized image recognition information. For example, the surgical area is displayed in an enlarged manner. In this way, by marking the surgical area in the image, the surgeon can be prompted to perform corresponding surgical operations in the area, thereby improving the efficiency and accuracy of the surgical operation.

[0078] As shown in Figure 3 , it is a schematic diagram of surgical area marking display provided by an embodiment of the present application. Figure 3 The image 300 shown in the figure can be an intraoperative endoscope image collected by an endoscope. Through the recognition processing of the image processing module 102, the corresponding surgical area 301 can be marked in the image 300. As shown in Figure 3 It can be seen that the surgical area 301 includes two polyps 3011, and subsequent surgical operations can be completed by the operation of the mechanical arm control module 106 to remove the two polyps 3011.

[0079] The natural orifice surgery control system provided by the embodiment of the present application can process the intraoperative image collected by the endoscope in real time, recognize the tissue structure in the image as reference information for path planning and navigation and surgical operation, to assist the doctor to complete the natural orifice surgery. The embodiment of the present application assists in performing surgical operations through the natural orifice surgery control system, so that the doctor can pay more attention to the surgical strategy rather than the operation details, can significantly improve the accuracy of the surgical operation and reduce the operation complexity while ensuring the safety of the operation.

[0080] On the basis of Figure 1 , referring to Figure 4 , a schematic diagram of another natural orifice surgery control system provided by an embodiment of the present application is shown, Figure 4 The control system 400 shown in the figure includes Figure 1In addition to the image processing module 102, the path navigation module 104 and the mechanical arm control module 106 shown in the image processing module 102, the path navigation module 104 and the mechanical arm control module 106, the image acquisition module 101, the instrument identification module 103, the user interaction module 105, the safety monitoring module 107 and the operation record module 108 are further included. The functions implemented by each module are introduced below.

[0081] The image acquisition module 101 can be used for image acquisition. In an example, the image acquisition module 101 can be an endoscope of the access control system, and the corresponding image acquisition function is realized through a camera device on the endoscope to obtain an intraoperative endoscope image. In another example, the image acquisition module 101 can be a functional module connected with the endoscope, which functions as a component of the image acquisition module 101 to realize the image acquisition function together with the endoscope. The embodiments of the present application do not limit this.

[0082] The intraoperative endoscope image acquired by the image acquisition module 101 can be transmitted to the image processing module 102 in real time for preoperative observation and real-time feedback of the operation screen during operation to guide subsequent operation.

[0083] The image processing module 102 is mainly used for processing the intraoperative endoscope image acquired by the image acquisition module 101, such as tissue recognition, lesion analysis and operation area marking of the intraoperative image. The image processing module 102 can output image recognition information for further processing by other related modules. The functions implemented by the image processing module 102 can be referred to Figure 1 The corresponding embodiments are introduced in the embodiment part, which will not be repeated here.

[0084] Compared with Figure 1 the control system shown in the control system, Figure 4 The control system can further include an instrument identification module 103, which can be used to identify instruments with different functions and ensure that the control mode of the corresponding instrument is called to control the instrument.

[0085] In the embodiments of the present application, when a certain instrument is connected to the control system, the instrument identification module 103 can read the identity information of the instrument that has been plugged in. For example, an identity chip can be embedded in the instrument during production or manufacturing, and the relevant identity information of the instrument can be written into the chip. When the instrument is connected to the control system, the instrument identification module 103 can read the identity information written in the identity chip and verify the instrument according to the read identity information.

[0086] In a possible implementation of the embodiment of the present application, the verification of the instrument can include verifying the instrument from the read identity information in terms of the adaptation relationship of the instrument to the current natural orifice surgery, the service life of the instrument, and compliance information of the instrument, etc. If the instrument identification module 103 confirms that the instrument is adapted to the current natural orifice surgery, the instrument is within the safe service life, and the instrument is a standard instrument, it can be determined that the accessed instrument passes the verification.

[0087] Specifically, the verification of the adaptation relationship of the instrument to the current natural orifice surgery can ensure that the instrument accessed to the system is the instrument required by the current surgery. For example, in the preoperative planning stage, the instruments required by the current surgery can be planned in advance. In the process of verifying the instrument, it can be verified whether the instrument is in the list of instruments required by the current surgery. For example, the current accessed instrument is an electrotome, and the instrument identification module 103 can verify whether the electrotome is included in the planned instrument list. If the electrotome is included in the instrument list, it can be considered that the electrotome will be used for surgical operation in the current natural orifice surgery, and at this time the accessed electrotome is adapted to the current surgery. Otherwise, if the electrotome is not included in the instrument list, it is considered that the electrotome is not adapted to the current instrument, and the instrument verification fails.

[0088] The verification of whether the instrument is within the safe service life can ensure the safety of subsequent surgical operation. Only instruments within the safe service life can be used in the natural orifice surgery. Any instrument beyond the service life cannot be used continuously.

[0089] The verification of the compliance information of the instrument can refer to verifying whether the accessed instrument is a legal instrument rather than a counterfeit. The legal instrument can be identified and judged by the identity information.

[0090] The instrument identification module 103 can set different verification items for each instrument. Only when each verification item passes the verification, the instrument is considered to pass the verification. For example, the verification items of an instrument include the adaptation relationship of the instrument to the current natural orifice surgery, the service life of the instrument, and the compliance information of the instrument in the foregoing examples. Only when the instrument is adapted to the current natural orifice surgery, the instrument is within the safe service life, and the instrument is a standard instrument, it can be considered that the instrument passes the verification. If any verification item fails the verification, it can be considered that the instrument fails the verification.

[0091] In the case that the instrument passes the verification, the instrument identification module 103 can obtain the control information matched with the instrument, and transmit the control information to the mechanical arm control module 104, so as to instruct the mechanical arm control module 104 to control the instrument according to the control information. The above control information can include a kinematic model adapted to the instrument.

[0092] Exemplarily, the arm of the forceps instrument has 6 degrees of freedom, and the electrotome has 3 degrees of freedom, so different control modes need to be adopted when using the two instruments, and accordingly different kinematic models need to be adapted for different instruments.

[0093] The path navigation module 104 can be used to plan a motion path of an instrument in a natural cavity according to image recognition information, and dynamically adjust the motion path in the process of instrument motion. For example, in the process of entering a natural cavity, such as the process of entering a lower digestive tract, the path navigation module 104 can plan a motion path of a surgical instrument and an endoscope, and prompt a forward direction to prevent the inner wall of the digestive tract from being injured in the process of advancing. When the surgical instrument deviates from the planned direction in the process of motion, the path navigation module 104 can automatically correct the advancing direction to ensure that the surgical instrument moves along the guided direction. The specific functions implemented by the path navigation module 104 can be referred to in the introduction of the corresponding embodiment part, which will not be described here in detail. Figure 1 The corresponding embodiment part is introduced, which will not be described here in detail.

[0094] As shown in Figure 4 The natural cavity surgery control system provided by the embodiments of the present application can further include a user interaction module 105. The module can be used to receive user operation instructions, and control related modules based on the received user operation instructions. For example, the related modules can include any module in the control system, such as the image acquisition module 101, the image processing module 102, the instrument recognition module 103, the path navigation module 104, the mechanical arm control module 106, the safety monitoring module 107, and the surgery recording module 108, and the like.

[0095] In a possible implementation manner of the embodiments of the present application, the above-mentioned user operation instructions can include any one or more of a foot pedal operation instruction, a gesture operation instruction, and a voice operation instruction.

[0096] The foot pedal operation instruction can be an instruction control mode operated by a doctor through a foot pedal. For example, the doctor can operate through a foot pedal when he cannot free his hands in the process of surgical operation. The foot pedal device can be located at the lower part of the master console cart. Through the foot pedal operation instruction, the doctor can realize functions such as switching the electrotome mode, starting or disconnecting the control of the surgical instrument, and keeping the surgical instrument in a clamping state, and the like.

[0097] The voice operation instruction can be a control mode of sending an instruction to the control system in the form of voice. Figure 4The control system shown in the figure can configure a voice recognition unit to recognize voice operation instructions issued by an intraoperative physician and perform corresponding operations. For example, in response to a voice operation instruction, the user interaction module 105 can control the image processing module 102 to mark a surgical area in an intraoperative endoscope image and display the marked intraoperative endoscope image through the display unit. In addition, the voice operation instruction can also be used to control the magnification of the endoscope display, adjust the display mode, mark the polyp area in the operation, and adjust the height of the operation table of the master console, the height of the surgeon's seat, and the like.

[0098] The gesture operation instruction can be realized through touch screen control. The physician can perform gesture operations such as sliding and clicking on the interface provided by the display unit of the control system to realize parameter setting, device state detection, and the like. At the same time, the operations or functions that can be realized through the foot pedal operation instruction and the voice operation instruction can also be realized through the gesture operation instruction, which is not limited in the embodiments of the present application.

[0099] The mechanical arm control module 106 can be used to control the instrument to move along the planned motion path, and after the instrument moves to the surgical area, control the instrument to perform corresponding surgical operations in the surgical area.

[0100] In the embodiments of the present application, the control system can connect the lower computer through the CAN device to issue instructions to the micro mechanical arm, control the operation of the micro mechanical arm, and then realize the operation of the endoscope and surgical instrument. The specific functions realized by the mechanical arm control module 106 can be referred to Figure 1 The corresponding embodiments are described in the introduction, which will not be repeated here.

[0101] As described above, the path navigation module 104 can plan the motion path based on the image recognition obtained by the image processing module 102, and dynamically adjust the path in real time during the instrument motion, so as to ensure the safety of the instrument motion. For example, to prevent the instrument tip from poking the natural cavity wall, and the like. In the embodiments of the present application, in addition to the safety control realized by the dynamic adjustment of the motion path of the path navigation module 104, the control system also provides a safety monitoring module 107, which further ensures the safety of the operation through the related processing of the safety monitoring module 107.

[0102] In the embodiments of the present application, in the case where the instrument contacts the tissue in the patient's body, the safety monitoring module 107 can determine the deformation amount of the contact area according to the intraoperative endoscope image. In this way, when the deformation amount is greater than a preset deformation threshold, the safety monitoring module 107 can perform a safety warning for the current operation. The contact information between the instrument and the tissue in the patient's body can be provided by the aforementioned image processing module 102 to the safety monitoring module 107.

[0103] Specifically, the image processing module 102 can process the obtained image recognition information to include contact information between the instrument and the in-vivo tissue, such as whether the two are in contact, the specific location of the contact area, and the like. In the case where the instrument is in contact with the in-vivo tissue, such as inside a certain side of the cavity, the image processing module 102 can transmit the contact information to the safety monitoring module 107. The safety monitoring module 107 can calculate the deformation amount of the contact area in the in-vivo endoscopic image. When the contact in a certain direction causes the deformation amount to exceed a set deformation threshold, it indicates that the instrument should not move in that direction any more, and the safety monitoring module 107 can timely give a warning to the current operation to prompt the doctor to adjust the surgical operation.

[0104] The operation record module 108 can be used to save each operation instruction in the operation process, so as to ensure the traceability of the operation process. In addition, when saving each operation instruction in the operation process, the operation record module 108 can also selectively retain the operation video corresponding to the surgical field in the operation according to actual needs. For example, for the saved operation instruction, the operation record module 108 can correspondingly save the surgical field image displayed in the in-vivo endoscopic image during the execution of the operation instruction. In this way, not only the traceability of the operation process is ensured, but also the operation process can be used as a case for analysis and learning by other doctors, so as to improve the operation level of the doctors.

[0105] In combination with the natural orifice surgery control system shown in Figure 1 and Figure 4 , the embodiment of the present application further provides a natural orifice surgery control method. As shown in Figure 5 , it is a schematic diagram of the natural orifice surgery control method provided by the embodiment of the present application. The method can specifically include the following steps:

[0106] S501, identifying the tissue structure in the in-vivo endoscopic image to obtain image recognition information.

[0107] S502, planning the movement path of the instrument in the natural orifice according to the image recognition information, and dynamically adjusting the movement path during the movement of the instrument.

[0108] S503, controlling the instrument to move according to the movement path, and after the instrument moves to the operation area, marking the operation area in the in-vivo endoscopic image according to the image recognition information.

[0109] S504, controlling the instrument to perform a corresponding operation in the operation area.

[0110] It should be noted that, Figure 5The natural orifice surgery control method shown can be applied to a computer device, which can be an electronic device capable of realizing computer-assisted medical related functions, such as a desktop computer, a cloud server, and the like. In an example, the computer device can serve as the bearing device of the natural orifice surgery control system shown in the foregoing Figure 1 and Figure 4 . Therefore, the computer device can realize the natural orifice surgery control method provided in the embodiments of the present application by realizing the functions that can be realized by the various modules of the foregoing natural orifice surgery control system.

[0111] Exemplarily, the computer device can be configured with the various modules in the foregoing Figure 4 in the form of hardware or software, such as an image acquisition module, an image processing module, an instrument recognition module, a path navigation module, a user interaction module, a mechanical arm control module, a safety monitoring module, and a surgery recording module.

[0112] In the embodiments of the present application, the identification of the tissue structure in the intraoperative endoscope image in S501 to obtain image recognition information, and the marking of the surgical area in the intraoperative endoscope image according to the image recognition information after the instrument moves to the surgical area in S503 can be realized by the image processing module in the computer device; the planning of the movement path of the instrument in the natural orifice according to the image recognition information in S502, and the dynamic adjustment of the movement path during the movement of the instrument can be realized by the path navigation module; the movement of the instrument according to the movement path in S503, and the execution of the corresponding surgical operation by the instrument in the surgical area after the instrument moves to the surgical area in S504 can be realized by the mechanical arm control module.

[0113] In addition, the other various modules can also realize corresponding functions in the process of realizing the method. For example, the image acquisition module can realize the acquisition of the intraoperative endoscope image, the instrument recognition module can realize the identification and verification of the instrument, the safety monitoring module can give a warning for the contact deformation in the case of contact between the instrument and the tissue inner wall, the surgery recording module can record the entire surgical operation process, and the like. Through the cooperation of the various modules in the computer device, the natural orifice surgery operation is completed, and the safety and accuracy of the surgery are ensured.

[0114] Since the implementation process of the method is similar to the methods or functions realized by the various modules of the foregoing natural orifice surgery control system, the relevant parts can be referred to the descriptions of the foregoing various embodiments, which will not be repeated here.

[0115] The natural cavity operation control system and the operation control method provided by the embodiments of the present application can greatly improve the operation precision based on image intelligent analysis and precise mechanical arm control. The motion path automatic planning and navigation and real-time mechanical feedback realized by the related modules and algorithms can enable the doctor to focus more on the operation strategy rather than the operation details, thereby greatly reducing the burden of the doctor. The embodiments of the present application can also prevent important tissues from being injured in the operation process through contact deformation monitoring and marking of the lesion area and the operation area, thereby enhancing the intraoperative safety. The natural cavity operation control system and the operation control method provided by the embodiments of the present application have wide adaptability and can be used to implement various types of natural cavity operations.

[0116] With reference to Figure 6 , a schematic diagram of a computer device provided by an embodiment of the present application is shown. As shown in Figure 6 , the computer device 600 in the embodiment of the present application includes a processor 610, a memory 620, and a computer program 621 stored in the memory 620 and executable on the processor 610. The processor 610 implements the steps in the natural cavity operation control method embodiments described above when executing the computer program 621, such as the steps S501 to S504 shown in Figure 5 . Alternatively, the processor 610 implements the functions of the modules / units in the system embodiments described above when executing the computer program 621, such as the functions of the modules 101 to 108 shown in Figure 4 .

[0117] For example, the computer program 621 can be divided into one or more modules / units, which are stored in the memory 620 and executed by the processor 610 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which can be used to describe the execution process of the computer program 621 in the computer device 600. For example, the computer program 621 can be divided into an image processing module, a path navigation module, and a mechanical arm control module, and the specific functions of the modules are as follows:

[0118] The image processing module is configured to identify the tissue structure in the intraoperative endoscope image and obtain image recognition information.

[0119] The path navigation module is configured to plan the motion path of the instrument in the natural cavity according to the image recognition information and dynamically adjust the motion path during the motion of the instrument.

[0120] The mechanical arm control module is configured to control the instrument to move along the movement path, and control the instrument to perform a corresponding surgical operation in a surgical region after the instrument moves to the surgical region. The surgical region is marked and displayed in the intraoperative endoscope image by the image processing module according to the image recognition information.

[0121] The computer device 600 can be a device capable of realizing the functions of the various modules in the various system embodiments described above. The computer device 600 can be a desktop computer, a cloud server, or the like. The computer device 600 can include, but is not limited to, a processor 610 and a memory 620. Those skilled in the art can understand that the computer device 600 can include more or fewer components, or combine some components, or include different components, such as an input / output device, a network access device, a bus, and the like. Figure 6 The computer device 600 is only an example and does not limit the computer device 600, which can include more or fewer components, or combine some components, or include different components, such as an input / output device, a network access device, a bus, and the like.

[0122] The processor 610 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, or the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0123] The memory 620 can be an internal storage unit of the computer device 600, such as a hard disk or a memory of the computer device 600. The memory 620 can also be an external storage device of the computer device 600, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like. Further, the memory 620 can include both the internal storage unit and the external storage device of the computer device 600. The memory 620 is used to store the computer program 621 and other programs and data required by the computer device 600. The memory 620 can also be used to temporarily store data that has been output or will be output.

[0124] The embodiment of the present application further discloses a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, a natural orifice surgery control method realized by each module in each system embodiment is implemented.

[0125] The embodiment of the present application further discloses a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and when the computer program is executed by a computer, a natural orifice surgery control method realized by each module in each system embodiment is implemented.

[0126] The embodiment of the present application further discloses a computer program product, comprising a computer program, and when the computer program is executed on a computer, the computer is caused to execute a natural orifice surgery control method realized by each module in each system embodiment.

[0127] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them. Although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent ones; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A natural cavity surgical control system, characterized in that, include: The image processing module is used to identify tissue structures in intraoperative endoscopic images and obtain image recognition information; The path navigation module is used to plan the movement path of the instrument within the natural cavity based on the image recognition information, and to dynamically adjust the movement path during the movement of the instrument. The robotic arm control module is used to control the instrument to move according to the motion path, and after the instrument moves to the surgical area, it controls the instrument to perform corresponding surgical operations in the surgical area; wherein, the surgical area is marked and displayed in the intraoperative endoscopic image by the image processing module according to the image recognition information.

2. The system according to claim 1, characterized in that, The intraoperative endoscopic images include natural cavity images, the image recognition information includes the inner wall of the natural cavity, and the path navigation module is specifically used for: Based on the natural cavity wall identified in the natural cavity image, the movement path of the instrument within the natural cavity is planned, and the movement path is indicated in the natural cavity image; wherein the distance between the movement path and the natural cavity wall on either side is greater than or equal to a distance threshold.

3. The system according to claim 2, characterized in that, The path navigation module is also used for: During the movement of the instrument within the natural cavity, the position of the instrument's end point is monitored in real time. When the distance between the end of the instrument and the inner wall of the natural cavity on either side is less than the distance threshold, or when the angular deviation between the direction of the instrument's movement and the forward direction of the planned movement path is greater than the angle threshold, movement path correction information is generated. The motion path is then corrected based on the motion path correction information.

4. The system according to claim 1, characterized in that, Also includes: The safety monitoring module is used to determine the deformation of the contact area based on the intraoperative endoscopic images when the instrument comes into contact with the patient's internal tissues. When the deformation exceeds a preset deformation threshold, a safety warning is issued for the current surgical procedure; wherein, the contact information between the instrument and the patient's internal tissues is provided to the safety monitoring module by the image processing module.

5. The system according to any one of claims 1 to 4, characterized in that, Also includes: The user interaction module is used to receive user operation instructions and control related modules based on the user operation instructions; wherein, the user operation instructions include any one or more of foot pedal operation instructions, gesture operation instructions, and voice operation instructions; the related modules include the image processing module and the robotic arm control module.

6. The system according to claim 5, characterized in that, The user interaction module is specifically used for: In response to the voice operation command, the image processing module is controlled to mark the surgical area in the intraoperative endoscopic image, and the marked intraoperative endoscopic image is displayed through the display unit.

7. The system according to any one of claims 1 to 4 or 6, characterized in that, Also includes: The device identification module is used to read the identity information of the connected device and verify the device based on the identity information. If the device passes verification, control information matching the device is acquired and transmitted to the robotic arm control module to instruct the robotic arm control module to control the device according to the control information. The control information includes a kinematic model adapted to the device.

8. The system according to claim 7, characterized in that, The instrument identification module is specifically used for: The device's compatibility with current natural cavity surgery, its lifespan, and its compliance information are verified based on the identity information. The device is deemed to have passed verification if it is compatible with current natural cavity surgery, is within its safe service life, and is a standard device.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the computer device to implement the following methods implemented by the various modules of the natural cavity surgical control system as described in any one of claims 1 to 8: Identify tissue structures in intraoperative endoscopic images to obtain image recognition information; The movement path of the instrument within the natural cavity is planned based on the image recognition information, and the movement path is dynamically adjusted during the movement of the instrument. The instrument is controlled to move along the motion path, and after the instrument moves to the surgical area, the surgical area is marked in the intraoperative endoscopic image according to the image recognition information. The instrument is controlled to perform the corresponding surgical procedure in the surgical area.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed, the following methods implemented by the various modules of the natural cavity surgical control system as described in any one of claims 1 to 8 are performed: Identify tissue structures in intraoperative endoscopic images to obtain image recognition information; The movement path of the instrument within the natural cavity is planned based on the image recognition information, and the movement path is dynamically adjusted during the movement of the instrument. The instrument is controlled to move along the motion path, and after the instrument moves to the surgical area, the surgical area is marked in the intraoperative endoscopic image according to the image recognition information. The instrument is controlled to perform the corresponding surgical procedure in the surgical area.

Images