Human-intelligence collaborative laparoscopic surgery robot system

Abstract

The present invention provides a human-intelligence-cooperative laparoscopic surgery mirror-holding robot system, which relates to the fields of artificial intelligence and robotics. The system comprises: a laparoscope, a mirror-holding robot, a voice interaction module and a control module; wherein the laparoscope is used to collect intracavitary images of an intraoperative object and send the intracavitary images to the control module; the voice interaction module is used to receive voice signals of the controlled object and send voice signals to the control module; the control module is used to generate action instructions for adjusting the field of view based on the intracavitary images and voice signals, and send the action instructions to the mirror-holding robot; and the mirror-holding robot is used to fix the laparoscope and control the movement of the laparoscope in response to the action instructions to adjust the intraoperative field of view.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and robotics, and in particular to a laparoscopic surgery mirror-holding robot system in collaboration with humans and intelligence. Background Art

[0002] In laparoscopic surgery, in order to obtain the surgical field of view in the abdominal cavity, a dedicated medical staff is required to support and control the endoscope to adjust the field of view during surgery. The shaking of the hands of the medical staff holding the endoscope will cause the display screen of the microscope to shake during surgery. The long-term holding of the endoscope by the medical staff holding the endoscope will cause physical fatigue, and there will be communication ambiguity between the medical staff holding the endoscope and the doctor, thus affecting the efficiency of the surgery. In order to solve these problems, the relevant technology uses a robot holding the endoscope to replace the medical staff holding the endoscope to control the endoscope.

[0003] In the related technology, the mirror-holding robot can use passive control to control the endoscope. The surgeon sends control instructions to the mirror-holding robot through foot pedals, voice, handles, etc., and the mirror-holding robot passively executes the corresponding control instructions until the ideal surgical field of view is generated. The passive control method requires the surgeon to issue control instructions multiple times, which will interfere with the surgeon's normal surgical process and thus affect the efficiency of the operation. The mirror-holding robot can also control the endoscope by identifying and tracking surgical instruments. However, surgical instruments often produce some auxiliary actions that do not require adjustment of the surgical field of view. Therefore, the quality of the surgical field of view generated by simply tracking the surgical instruments is poor. Summary of the invention

[0004] In view of this, the present invention provides a human-intelligence collaborative laparoscopic surgery mirror-holding robot system that can generate a high-quality surgical field of view.

[0005] A human-intelligence-cooperative laparoscopic surgical mirror-holding robot system as an embodiment of the present invention includes: a laparoscope, a mirror-holding robot, a voice interaction module and a control module; wherein the laparoscope is used to collect intracavitary images of the object during surgery, and send the intracavitary images to the control module; the voice interaction module is used to receive voice signals of the controlled object, and send the voice signals to the control module; the control module is used to generate action instructions for adjusting the field of view based on the intracavitary images and the voice signals, and send the action instructions to the mirror-holding robot; and the mirror-holding robot is used to fix the laparoscope, and control the movement of the laparoscope in response to the action instructions to adjust the intraoperative field of view.

[0006] According to an embodiment of the present invention, the control module includes: a speech recognition unit, a depth perception unit, a scene perception unit, a human-intelligence collaboration unit and a visual servo unit; wherein the speech recognition unit is used to recognize the speech signal to obtain a speech recognition result; the depth perception unit is used to perform depth perception based on the intracavitary image to obtain a depth map; the scene perception unit is used to determine a plurality of position points based on the intracavitary image, wherein the plurality of position points include at least one instrument position point and at least one tissue position point; the human-intelligence collaboration unit is used to determine a desired field of view point based on the speech recognition result and the plurality of position points; and the visual servo unit is used to output the action instruction based on the depth map and the desired field of view point.

[0007] According to an embodiment of the present invention, the scene perception unit includes: an image segmentation subunit, used to perform image segmentation processing on the intracavitary image to obtain a mask image; an instrument position point determination subunit, used to extract an instrument mask image block from the mask image, and determine the at least one instrument position point based on the instrument mask image block; and a tissue position point determination subunit, used to extract a tissue mask image block from the mask image, and determine the at least one tissue position point based on the tissue mask image block.

[0008] According to an embodiment of the present invention, the above-mentioned instrument position point determination subunit is specifically used to: divide the above-mentioned instrument mask image block into connected areas to obtain at least one first instrument connected area; perform noise filtering on the above-mentioned at least one first instrument connected area to obtain at least one second instrument connected area; for each of the above-mentioned second instrument connected areas, skeletonize the above-mentioned second instrument connected areas to obtain an instrument skeleton; based on the principal component analysis method, extract instrument line segments used to represent the initial and terminal positions of the instrument from the above-mentioned instrument skeleton; determine the end points of the instrument based on the positions of the above-mentioned instrument line segments in the above-mentioned intracavitary image; and obtain the instrument position points corresponding to the above-mentioned second instrument connected areas based on the end points of the above-mentioned instrument.

[0009] According to an embodiment of the present invention, the above-mentioned tissue position point determination subunit is specifically used to: determine at least one key tissue connected area related to the key tissue from the at least one tissue connected area included in the above-mentioned tissue mask image based on the respective labels of the at least one tissue connected area; and for each of the above-mentioned key tissue connected areas, obtain the tissue position point corresponding to the above-mentioned key tissue connected area based on the first-order moment of the above-mentioned key tissue connected area.

[0010] According to an embodiment of the present invention, the above-mentioned human intelligence collaboration unit is specifically used for: when the above-mentioned voice recognition result indicates that the above-mentioned voice signal is valid, determining the instruction type based on the voice instruction information included in the above-mentioned voice recognition result; and determining the above-mentioned expected field of view point based on the above-mentioned multiple position points by utilizing a processing strategy corresponding to the above-mentioned instruction type.

[0011] According to an embodiment of the present invention, the above-mentioned instruction type includes a direction instruction type; wherein the above-mentioned human intelligence collaboration unit is specifically used to: determine the target image area from the multiple image areas included in the above-mentioned intracavitary image based on the above-mentioned voice instruction information, wherein the above-mentioned multiple image areas are obtained based on the center division of the above-mentioned intracavitary image; in the case where the above-mentioned target image area only contains the first position point, determine the above-mentioned first position point as the above-mentioned expected field of view point, wherein the above-mentioned first position point belongs to the above-mentioned multiple position points; in the case where the above-mentioned target image area contains multiple second position points, and the above-mentioned multiple second position points are all instrument position points, determine the above-mentioned expected field of view point based on the centroid position of the above-mentioned multiple second position points, wherein the above-mentioned multiple second position points belong to the above-mentioned multiple position points; the above-mentioned target image area contains multiple third position points, and the above-mentioned multiple third position points are all tissue position points In the case where the target image area includes multiple fourth position points, and the multiple fourth position points include instrument position points and tissue position points, the first centroid position is determined based on the instrument position points included in the multiple fourth position points, the second centroid position is determined based on the tissue position points included in the multiple fourth position points, and the expected field of view point is determined based on the line between the first centroid position and the second centroid position, wherein the multiple fourth position points belong to the multiple position points; and in the case where the target image area does not include the above-mentioned position points, the reverse direction boundary is determined based on the voice instruction information, and the expected field of view point is determined based on the distance between each of the at least one instrument position point and the reverse direction boundary.

[0012] According to an embodiment of the present invention, the above-mentioned instruction type includes a directional attention instruction type; wherein the above-mentioned human intelligence collaboration unit is specifically used to: determine the first target device or target tissue based on the above-mentioned voice instruction information; and determine the above-mentioned expected field of view point based on the device position point corresponding to the above-mentioned first target device or the tissue position point corresponding to the above-mentioned target tissue among the above-mentioned multiple position points.

[0013] According to an embodiment of the present invention, the above-mentioned instruction type includes an automatic tracking instruction type; wherein the above-mentioned human intelligence collaboration unit is specifically used to: determine the tracking point based on the above-mentioned voice instruction information; and whenever the above-mentioned tracking point is outside a preset area, determine the above-mentioned expected field of view point based on the above-mentioned tracking point.

[0014] According to an embodiment of the present invention, the system is configured to switch the working mode from the initial mode to the direction mode, the directional focus mode or the automatic tracking mode in response to the voice signal; wherein, when the system is configured to switch the working mode to the direction mode or the directional focus mode, the human-intelligence collaboration unit is configured to output one of the expected viewing points, and after the mirror-holding robot controls the laparoscope to complete an action in response to an action instruction generated based on the expected viewing point, the system is configured to switch the working mode to the initial mode; when the system is configured to switch the working mode to the automatic tracking mode, the mirror-holding robot is configured to control the laparoscope to complete an action each time the human-intelligence collaboration unit outputs the expected viewing point; the system is also configured to switch the working mode to the working mode corresponding to the new voice signal in response to receiving a new voice signal.

[0015] According to an embodiment of the present invention, a laparoscope is supported and controlled by a mirror-holding robot, and the information of the intracavitary image of the object during surgery is identified by a control module, the voice signal of the controlled object is analyzed, and action instructions are generated to adjust the surgical field of view. The human-intelligence collaborative laparoscopic surgery mirror-holding robot system of an embodiment of the present invention can understand the adjustment intention of the surgical field of view of the controlled object only through a voice interaction module, which reduces the operating burden of the controlled object and does not interrupt the surgical process. Therefore, it can provide a high-quality surgical field of view for laparoscopic surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings.

[0017] Figure 1 A principle schematic diagram of a human-intelligence-cooperative laparoscopic surgery mirror-holding robot system according to an embodiment of the present invention is shown.

[0018] Figure 2 A schematic diagram showing the working principle of a control module of a human-intelligence-cooperative laparoscopic surgery mirror-holding robot system according to an embodiment of the present invention is shown.

[0019] Figure 3 A schematic diagram of determining instrument position points of a human-intelligence-cooperative laparoscopic surgery mirror-holding robot system according to an embodiment of the present invention is shown.

[0020] Figure 4A A schematic diagram showing the principle of determining a desired field of view point based on a first position point in a target image area according to an embodiment of the present invention is shown.

[0021] Figure 4B A schematic diagram showing the principle of determining a desired field of view point based on a plurality of second position points within a target image region according to an embodiment of the present invention is shown.

[0022] Figure 4C A schematic diagram showing the principle of determining a desired field of view point based on a plurality of third position points within a target image region according to an embodiment of the present invention is shown.

[0023] Figure 4D A schematic diagram showing the principle of determining a desired field of view point based on a plurality of fourth position points within a target image region according to an embodiment of the present invention is shown.

[0024] Figure 4E A schematic diagram showing the principle of determining a desired field of view point when a target image region does not contain a position point according to an embodiment of the present invention is shown.

[0025] Figure 5A A schematic diagram showing the principle of determining an expected field of view point according to an automatic tracking instruction type according to an embodiment of the present invention.

[0026] Figure 5B A schematic diagram showing the principle of determining an expected field of view point according to an automatic tracking instruction type according to another embodiment of the present invention.

[0027] Figure 5C A schematic diagram showing the principle of determining an expected field of view point according to an automatic tracking instruction type according to yet another embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0029] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0030] All terms used herein, including technical and scientific terms, have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0031] When using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted as the meaning of the expression generally understood by those skilled in the art. For example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc. When using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted as the meaning of the expression generally understood by those skilled in the art. For example, "a system having at least one of A, B, or C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.

[0032] It should also be noted that the directional terms mentioned in the embodiments, such as "up", "down", "in", "out", "left", "right", etc., are only reference directions of the drawings and are not intended to limit the scope of protection of the present invention. Throughout the drawings, the same elements are represented by the same or similar reference numerals. Conventional structures or configurations will be omitted when they may cause confusion in the understanding of the present invention.

[0033] In the related art, the scope-holding robot uses passive control to control the endoscope, which cannot accurately control the direction and amount of movement used to adjust the surgical field of view, and requires the surgeon to send field of view adjustment instructions multiple times. For example, when using foot pedals and voice control, there are only four instructions: up, down, left, and right. When oblique movement is required, for example, the upper left direction, it is necessary to move left and then up to obtain the ideal direction of movement. Moreover, the field of view adjustment distance is fixed each time the command is given, and the doctor needs to issue multiple "left" commands, or keep pressing the left direction button on the foot pedal to achieve the ideal amount of movement to the left. Therefore, the control process using passive control is very cumbersome.

[0034] In the related art, the mirror-holding robot controls the endoscope by identifying and tracking surgical instruments. The mirror-holding robot actively adjusts the surgical field of view, although it does not require the intervention of the surgeon, and only uses the end position of the surgical instrument as the basis for adjusting the field of view. However, during surgery, there is often a situation where the surgical field of view does not need to be adjusted, but the surgical instrument is always in motion. Therefore, it is difficult to obtain an accurate surgical field of view by simply tracking the end of the surgical instrument. In other words, the way the mirror-holding robot actively adjusts the surgical field of view cannot accurately capture the adjustment intention of the surgical field of view, and the quality of the generated surgical field of view is poor.

[0035] The present invention provides a human-intelligence-cooperative laparoscopic surgery mirror-holding robot system, the system comprising: a laparoscope, a mirror-holding robot, a voice interaction module and a control module; wherein the laparoscope is used to collect intracavitary images of an object during surgery and send the intracavitary images to the control module; the voice interaction module is used to receive voice signals of the controlled object and send voice signals to the control module; the control module is used to generate action instructions for adjusting the field of view based on the intracavitary images and voice signals, and send the action instructions to the mirror-holding robot; and the mirror-holding robot is used to fix the laparoscope and control the movement of the laparoscope in response to the action instructions to adjust the intraoperative field of view.

[0036] Figure 1 A principle schematic diagram of a human-intelligence-cooperative laparoscopic surgery mirror-holding robot system according to an embodiment of the present invention is shown.

[0037] like Figure 1 As shown, the human-intelligence-cooperative laparoscopic surgery mirror-holding robot system includes a laparoscope 1, a mirror-holding robot 2, a voice interaction module 3 and a control module 4.

[0038] Laparoscopic surgery is a minimally invasive surgical technique that involves making several small holes in the abdominal wall and inserting laparoscopic surgical instruments into the abdominal cavity for surgery. Clinical applications of laparoscopic surgery include cholecystectomy, liver cyst fenestration and drainage, hernia repair, etc. Laparoscopic surgery also has diagnostic functions and can be used for puncture biopsy, ultrasound positioning, etc. Laparoscopic surgical equipment generally includes an insufflator for regulating intra-abdominal pressure and an optical system for providing a surgical field of view. Laparoscopic surgical instruments include a puncture device for establishing and maintaining an abdominal wall channel and hand-controlled instruments such as graspers, scissors, and electric hooks.

[0039] According to an embodiment of the present invention, the intraoperative object 5 may be a patient undergoing laparoscopic surgery, and the controlled object 6 may be a surgeon performing laparoscopic surgery. The intracavitary image may be a two-dimensional image.

[0040] In one example, if Figure 1 As shown, the laparoscope 1 is used to collect intracavitary images of the abdominal cavity of the object 5 during surgery, and can also be used as an input source to transmit the intracavitary images to the control module 4. The mirror holding robot 2 is used to fix the laparoscope 1, and also responds to the action instructions from the control module 4 to complete the field of view adjustment in a specific operation. The voice interaction module 3 is used to receive the voice signal sent by the control object 6 and send the voice signal to the control module 4. The control module 4 generates the action instructions for adjusting the surgical field of view based on the intracavitary image and the voice signal, and sends the action instructions to the mirror holding robot 2.

[0041] According to an embodiment of the present invention, a laparoscope is supported and controlled by a mirror-holding robot, and the information of the intracavitary image of the object during surgery is identified by a control module, the voice signal of the controlled object is analyzed, and action instructions are generated to adjust the surgical field of view. The human-intelligence collaborative laparoscopic surgery mirror-holding robot system of an embodiment of the present invention can understand the adjustment intention of the surgical field of view of the controlled object only through a voice interaction module, which reduces the operating burden of the controlled object and does not interrupt the surgical process. Therefore, it can provide a high-quality surgical field of view for laparoscopic surgery.

[0042] Figure 2 A schematic diagram showing the working principle of a control module of a human-intelligence-cooperative laparoscopic surgery mirror-holding robot system according to an embodiment of the present invention is shown.

[0043] like Figure 2 As shown, the control module 4 includes a speech recognition unit 41, a scene perception unit 42, a depth perception unit 43, a human intelligence collaboration unit 44 and a visual servoing unit 45.

[0044] According to an embodiment of the present invention, the speech recognition unit 41 is used to recognize the speech signal 401, obtain the speech recognition result 403, and send the speech recognition result 403 to the human intelligence collaboration unit 44. The depth perception unit 43 is used to perform depth perception based on the intracavitary image 402, obtain the depth map 404, and continuously send the depth map 404 to the visual servo unit 45. The scene perception unit 42 is used to determine a plurality of position points based on the intracavitary image 402, wherein the plurality of position points include at least one instrument position point 406 and at least one tissue position point 407. The scene perception unit 42 continuously sends the plurality of position points to the human intelligence collaboration unit 44. The human intelligence collaboration unit 44 is used to determine the desired field of view point 408 based on the speech recognition result 403 and the plurality of position points. The visual servo unit 45 is used to output an action instruction 409 based on the depth map 404 and the desired field of view point 408.

[0045] It should be noted that during the preoperative preparation, or during the operation to wipe the lens, or before the end of the operation and the system is shut down, the laparoscope will also continue to send the intracavitary image 402 to the control module. In this case, the intracavitary image 402 may not show instrument position points or tissue position points, which may be non-critical tissues, or the controlled object may not use surgical instruments. When the controlled object sends a voice signal 401 and hopes to adjust the surgical field of view, the multiple position points in the intracavitary image 402 processed by the control module include at least one instrument position point 406 and at least one tissue position point 407.

[0046] According to an embodiment of the present invention, the instrument position point 406 represents the pixel position representation of the surgical instrument used by the control subject in the laparoscopic surgery in the intracavitary image 402. The tissue position point 407 represents the pixel position representation of the tissue in the abdominal cavity of the subject during the laparoscopic surgery in the intracavitary image 402. The desired field of view point 408 represents the center of the surgical field of view that the control subject expects to obtain.

[0047] In one example, the speech recognition unit 41 can convert the speech signal 401 into a signal understandable by the control module by means of deep learning, for example, by using a deep learning model. For example, the speech signal emitted by the control object can be "up", "down", "look at the gallbladder", or "track the electric hook".

[0048] In one example, the depth sensing unit 43 may obtain the depth map 404 by deep learning, and extract the estimated value of the depth information from the depth map 404. The deep learning method may be any depth information sensing model, for example, a convolutional neural network. For example, a deep learning model using an encoder-decoder structure may be used.

[0049] In one example, based on the depth information of the depth map 404 and the two-dimensional camera coordinates of the intracavitary image 401, the visual servo unit 45 can obtain the three-dimensional coordinates of the intra-abdominal surgical scene, and then obtain the three-dimensional coordinates of the desired view point 408 in the intra-abdominal surgical scene. The visual servo unit 45 can continuously output the motion instruction 409 to the mirror holding robot to move its body through visual servoing and inverse kinematics. The motion instruction 409 includes the angle and speed of each joint of the mirror holding robot to move.

[0050] In one example, when the distance between the position point of the center of the intracavitary image 402 and the desired field of view point 408 is less than a threshold, the visual servo unit 45 stops sending the action instruction 409 to the mirror holding robot. When the voice recognition unit 41 sends the voice recognition result 403 of "stop", the visual servo unit 45 stops sending the action instruction 409 to the mirror holding robot.

[0051] like Figure 2 As shown, the scene perception unit 42 includes an image segmentation subunit 421 , an instrument position point determination subunit 422 , and a tissue position point determination subunit 423 .

[0052] According to an embodiment of the present invention, the image segmentation subunit 421 is used to perform image segmentation processing on the intracavitary image 402 to obtain a mask image 405. The instrument position point determination subunit 422 is used to extract an instrument mask image block from the mask image 405, and determine at least one instrument position point 406 based on the instrument mask image block. The tissue position point determination subunit 423 is used to extract a tissue mask image block from the mask image 405, and determine at least one tissue position point 407 based on the tissue mask image block.

[0053] According to the embodiment of the present invention, the image information of the intracavitary image 402 mainly includes the information of the surgical instruments in the surgical scene and the information of the tissue in the abdominal cavity of the subject during the operation. The image segmentation subunit 421 can convert the intracavitary image 402 into a mask image 405 .

[0054] In one example, the image segmentation processing method can be any image segmentation model. For example, it can be a fully convolutional network to achieve pixel-level image segmentation. For example, it can be an image segmentation model using an encoder-decoder structure to achieve image segmentation with accurate positioning of segmentation boundaries. Since each frame of the intracavitary image 402 must be processed, a lightweight image segmentation model can be preferred to reduce the time spent on processing each frame.

[0055] In one example, the mask image 405 can be a multi-category segmented image, in which the value of each pixel represents the category to which it belongs. For example, the area with a pixel value of 0 represents the background area, the area with a pixel value of 1 represents the electric hook, the area with a pixel value of 2 represents the gallbladder, etc., which is used to control the area or process of image processing so as to operate on a specific area of ​​the intracavitary image 402 in the next operation.

[0056] According to an embodiment of the present invention, the instrument position point 406 may represent the 2D pixel position coordinates of the surgical instrument tip in the intracavitary image 402. The tissue position point 407 may represent the 2D pixel position coordinates of the tissue position information in the intracavitary image 402.

[0057] According to an embodiment of the present invention, the instrument position point determination subunit is specifically used to: divide the instrument mask image block into connected areas to obtain at least one first instrument connected area. Perform noise filtering on at least one first instrument connected area to obtain at least one second instrument connected area. For each second instrument connected area, skeletonize the second instrument connected area to obtain an instrument skeleton. Based on the principal component analysis method, extract instrument line segments used to represent the start and end positions of the instrument from the instrument skeleton. Based on the position of the instrument line segments in the intracavitary image, determine the end point of the instrument. Based on the end point of the instrument, obtain the instrument position point corresponding to the second instrument connected area.

[0058] According to an embodiment of the present invention, the surgical instrument is a carrier for controlling the implementation of the surgical action of the object. The surgical instrument can be slender, can be multiple, and can have multiple directions of surgical action.

[0059] In one example, there may be two surgical instruments, for example, a grasping forceps and an electric hook. The connected region division is to find and mark the pixel regions with the same pixel value and adjacent positions in the instrument mask image block, and obtain the first instrument connected regions of the grasping forceps and the electric hook respectively. The first instrument connected regions of the grasping forceps and the electric hook are noise filtered respectively, and the first instrument connected regions with an area less than a threshold are removed to obtain the second instrument connected regions of the grasping forceps and the electric hook respectively. The second instrument connected region of the grasping forceps is skeletonized, and the shape of the second instrument connected region of the grasping forceps is simplified into a thin line to obtain the instrument skeleton of the grasping forceps. Based on the principal component analysis method, the most important instrument direction is fitted from the multiple line segments of the instrument skeleton to generate the instrument line segments of the grasping forceps. Similarly, the second instrument connected region of the electric hook is skeletonized, and based on the principal component analysis method, the instrument line segments of the electric hook are obtained.

[0060] Figure 3 A schematic diagram of determining instrument position points of a human-intelligence-cooperative laparoscopic surgery mirror-holding robot system according to an embodiment of the present invention is shown.

[0061] In one example, the intracavitary image may be a rectangular image or an elliptical image. Figure 3 The square frame in the solid line mode represents the screen of the intracavitary image, the straight line segment in the solid line mode represents the instrument segment of the electric hook obtained by the human intelligence collaboration unit, and the straight line segment in the dotted line mode is used to simulate the partial entity of the electric hook outside the screen of the intracavitary image. The position of the instrument segment of the electric hook close to the edge of the screen in the screen of the intracavitary image is determined as the starting point B of the instrument segment of the electric hook, and the starting point B is represented by a hollow dot. Then, the other end of the instrument segment of the electric hook is determined as the end point A of the instrument of the electric hook, and the end point A is represented by a solid dot. Based on the position of the end point A of the instrument of the electric hook, the instrument position point of the electric hook is obtained. Similarly, the instrument position point of the grasping forceps can be obtained.

[0062] According to an embodiment of the present invention, the tissue position point determination subunit is specifically used to: determine at least one key tissue connected area related to the key tissue from at least one tissue connected area based on the label of each of the at least one tissue connected area included in the tissue mask image. For each key tissue connected area, a tissue position point corresponding to the key tissue connected area is obtained based on the first-order moment of the key tissue connected area.

[0063] According to an embodiment of the present invention, based on a multi-category segmentation image, the value of each pixel represents the category to which it belongs, and the corresponding tissue connected regions have respective labels.

[0064] According to an embodiment of the present invention, key tissues in a surgical scene, such as the gallbladder and the cystic duct in a cholecystectomy, have representative shapes and can form obvious connected areas, which are the core operating tissues of the corresponding laparoscopic surgery.

[0065] According to an embodiment of the present invention, non-critical tissues in a surgical scene, such as mucosa and fat in an intracavitary image, occupy a large portion of the intracavitary image. The connected area formed based on the non-critical tissue mask is very large and the information it carries is not very useful. Therefore, the non-critical tissue mask is directly discarded and no corresponding position point is generated.

[0066] In one example, for example, the gallbladder and the cystic duct in a cholecystectomy. Through the connected area division, the tissue connected areas of the gallbladder and the cystic duct are obtained respectively. The first-order moment of the tissue connected area of ​​the gallbladder is confirmed as the tissue position point of the gallbladder. The first-order moment of the tissue connected area of ​​the cystic duct is confirmed as the tissue position point of the cystic duct.

[0067] According to an embodiment of the present invention, the human intelligence collaboration unit is specifically used to: determine the instruction type based on the voice instruction information included in the voice recognition result when the voice recognition result indicates that the voice signal is valid. Determine the desired field of view point based on multiple position points using a processing strategy corresponding to the instruction type.

[0068] In one example, the controlled object emits speech in the form of isolated words, for example, "up", "down", "look at the gallbladder". The speech interaction module obtains speech signals with different electromagnetic wave waveforms, and the speech recognition unit outputs corresponding speech recognition results according to the speech signals with different electromagnetic wave waveforms. The speech interaction module can be received through an ear-hook Bluetooth headset or a microphone. The ear-hook Bluetooth headset can be firmly fixed on the auricle of the controlled object, and can also capture the speech signals emitted from the mouth of the controlled object in a direction, and can suppress noise signals from other directions.

[0069] According to an embodiment of the present invention, four types of instructions are determined based on voice instruction information, including direction instructions, directional attention instructions, automatic tracking instructions and "stop". Direction instructions include: "up", "down", "left", "right", "in", and "out". Directional attention instructions are in the form of a combination of "look" + "focus position", such as "look at the gallbladder" and "look at the electric hook". Automatic tracking instructions include: "automatic" and a combination of "tracking" + "instrument type", such as "tracking the electric hook".

[0070] According to an embodiment of the present invention, when the speech recognition unit outputs a corresponding speech recognition result that is any of the above-mentioned speech instruction information, it indicates that the speech signal is valid. In this case, the speech recognition unit sends the above-mentioned speech instruction information to the human intelligence collaboration unit. The human intelligence collaboration unit implements the corresponding processing strategy according to the instruction type.

[0071] According to an embodiment of the present invention, when the instruction type is a direction instruction type, the human intelligence collaboration unit 44 implements a corresponding processing strategy:

[0072] Based on the voice instruction information, a target image area is determined from a plurality of image areas included in the intracavitary image, wherein the plurality of image areas are obtained based on the center division of the intracavitary image. In the case where the target image area only includes a first position point, the first position point is determined as a desired field of view point. In the case where the target image area includes a plurality of second position points, and the plurality of second position points are all instrument position points, the desired field of view point is determined based on the centroid positions of the plurality of second position points. In the case where the target image area includes a plurality of third position points, and the plurality of third position points are all tissue position points, the desired field of view point is determined based on the centroid positions of the plurality of third position points. In the case where the target image area includes a plurality of fourth position points, and the plurality of fourth position points include instrument position points and tissue position points, the first centroid position is determined based on the instrument position points included in the plurality of fourth position points, the second centroid position is determined based on the component position points included in the plurality of fourth position points, and the desired field of view point is determined based on the line connecting the first centroid position and the second centroid position. In the case where the target image area does not include a position point, a reverse direction boundary is determined based on the voice instruction information, and the desired field of view point is determined based on the distance between at least one instrument position point and the reverse direction boundary.

[0073] According to an embodiment of the present invention, the first location point, the plurality of second location points, the plurality of third location points and the plurality of fourth location points all belong to the plurality of location points. The centroid position can be represented as a weighted average position of all pixels in the location point.

[0074] When the instruction type is a direction instruction type, the following combination FIG. 4A to FIG. 4E , and further explain the corresponding processing strategies implemented by the human intelligence collaboration unit. Figure 4A A schematic diagram showing the principle of determining a desired field of view point based on a first position point in a target image area according to an embodiment of the present invention is shown. Figure 4B A schematic diagram showing the principle of determining a desired field of view point based on a plurality of second position points within a target image region according to an embodiment of the present invention is shown. Figure 4C A schematic diagram showing the principle of determining a desired field of view point based on a plurality of third position points within a target image region according to an embodiment of the present invention is shown. Figure 4DA schematic diagram showing the principle of determining a desired field of view point based on a plurality of fourth position points within a target image region according to an embodiment of the present invention is shown. Figure 4E A schematic diagram showing the principle of determining a desired field of view point when a target image region does not contain a position point according to an embodiment of the present invention is shown.

[0075] In one example, for example, the voice command information is "left" and the command type is a direction command type. The human intelligence collaboration unit divides the intracavitary image into an upper half area and a lower half area, or a left half area and a right half area, based on the center of the intracavitary image. The human intelligence collaboration unit determines the left half area as the target image area based on the voice command information being "left".

[0076] In one example, if FIG. 4A to FIG. 4E As shown, the solid line box on the left represents the image processing schematic diagram of the intracavitary image after receiving the voice command information "left", and the left half area is determined as the target image area. The straight line in the solid line mode represents an exemplary surgical instrument currently used by the control object, and the instrument position point of the surgical instrument is represented by a solid circle A. After determining the desired field of view point, the control module controls the mirror holding robot to move in the command direction, and the solid line box on the right represents the intracavitary image after the mirror holding robot moves.

[0077] like Figure 4A As shown, when the left half area only includes one first position point A1, the first position point A1 is determined as the desired field of view point E.

[0078] like Figure 4B As shown, when the left half area contains three second position points (instrument position point A2, instrument position point A3, instrument position point A4), the centroid position X of all second position points in the left half area is calculated, and the centroid position X is determined to be the expected field of view point E.

[0079] like Figure 4C As shown, when the left half area contains three third position points (tissue position point C1, tissue position point C2, tissue position point C3), the centroid position Y of all third position points in the left half area is calculated, and the centroid position Y is determined to be the expected field of view point E.

[0080] like Figure 4DAs shown, when the left half area contains six fourth position points (instrument position point A5, instrument position point A6, instrument position point A7, tissue position point C4, tissue position point C5, tissue position point C6), the first centroid position X0 of the three fourth position points (instrument position point A5, instrument position point A6, instrument position point A7) in the left half area is calculated, and the second centroid position Y0 of the three fourth position points (tissue position point C4, tissue position point C5, tissue position point C6) in the left half area is calculated, and the instrument position points included in the multiple fourth position points are given more weights, and based on the line connecting the first centroid position X0 and the second centroid position Y0, the position point Z on the line close to the first centroid position X0 is determined as the expected field of view point E.

[0081] like Figure 4E As shown, when the left half area does not contain a position point, that is, the controlled object does not want to obtain the surgical field of view of the surgical instrument or the surgical field of view of the key tissue, the human intelligence collaboration unit will still generate the desired field of view point. The human intelligence collaboration unit calculates the minimum value d of the distance between the instrument position point A and the instrument position point A8 and the boundary of the right half area, and takes the position point F with a distance d from the center position point D of the screen as the desired field of view point E. While the mirror-holding robot moves in the command direction, it can also ensure that the surgical instrument will not remove the range of the surgical field of view.

[0082] According to an embodiment of the present invention, when the instruction type is a directional attention instruction type, the human intelligence collaboration unit implements a corresponding processing strategy: based on the voice instruction information, a first target instrument or target tissue is determined. Based on an instrument position point corresponding to the first target instrument or a tissue position point corresponding to the target tissue among multiple position points, a desired field of view point is determined.

[0083] In one example, for example, the voice command information is "look at the gallbladder", the voice command information is "look" + "pay attention to key tissues", the command type is a directional attention command type, and the target tissue is determined to be the gallbladder. The human intelligence collaboration unit determines the tissue position point of the gallbladder as the expected field of view point, and the mirror-holding robot responds to the corresponding action command once. For example, the voice command information is "look at the electric hook", the voice command information is "look" + "pay attention to key instruments", the command type is a directional attention command type, and the target instrument is determined to be the electric hook. The human intelligence collaboration unit determines the instrument position point of the electric hook as the expected field of view point, and the mirror-holding robot responds to the corresponding action command once.

[0084] According to an embodiment of the present invention, when the instruction type is an automatic tracking instruction type, a tracking point is determined based on the voice instruction information, and whenever the tracking point is outside the preset area, a desired field of view point is determined based on the tracking point.

[0085] When the instruction type is the automatic tracking instruction type, the following combination FIG. 5A to FIG. 5C, and further explain the corresponding processing strategies implemented by the human intelligence collaboration unit. Figure 5A A schematic diagram showing the principle of determining an expected field of view point according to an automatic tracking instruction type according to an embodiment of the present invention. Figure 5B A schematic diagram showing the principle of determining an expected field of view point according to an automatic tracking instruction type according to another embodiment of the present invention. Figure 5C A schematic diagram showing the principle of determining an expected field of view point according to an automatic tracking instruction type according to yet another embodiment of the present invention.

[0086] According to an embodiment of the present invention, the control object will give an "automatic" voice signal, the voice instruction information is "automatic", the instruction type is an automatic tracking instruction type, and the control module can enable the mirror-holding robot to enter a state of actively adjusting the surgical field of view, and use the centroid position of the position points of all surgical instruments as the first tracking point. When there is only one instrument position point, the instrument position point is used as the second tracking point. When the first tracking point or the second tracking point is within the preset area, the human-intelligence collaboration unit will not generate the expected field of view point; when the first tracking point or the second tracking point is outside the preset area, the human-intelligence collaboration unit will determine the first tracking point or the second tracking point as the expected field of view point. The mirror-holding robot responds to the action instruction corresponding to the expected field of view point to automatically track the surgical instrument and adjust the surgical field of view.

[0087] According to an embodiment of the present invention, when there are multiple surgical instruments in a surgical scene, but for auxiliary surgical instruments, tracking is generally not required. The control object will give a voice signal with the instruction information of "tracking" + "instrument type", for example, "tracking electric hook", the instruction type is an automatic tracking instruction type, and the second target instrument is determined to be an electric hook. The control module can enable the mirror-holding robot to enter a state of actively adjusting the surgical field of view, and use the instrument position point of the electric hook as the third tracking point. When the third tracking point is within the preset area, the human-intelligence collaboration unit will not generate the expected field of view point; when the third tracking point is outside the preset area, the human-intelligence collaboration unit will determine the third tracking point as the expected field of view point. The mirror-holding robot responds to the action instruction corresponding to the expected field of view point to automatically track the electric hook and adjust the surgical field of view.

[0088] In one example, if FIG. 5A to FIG. 5C As shown, the box in the solid line mode on the left is a schematic diagram of the intracavitary image after receiving the automatic tracking instruction. The straight line in the solid line mode represents an exemplary surgical instrument currently used by the control object, and the instrument position point of the surgical instrument is represented by a solid circle A. After determining the desired field of view point, the control module controls the mirror holding robot to move, and the box in the solid line mode on the right represents the intracavitary image obtained after the mirror holding robot moves. The gray circular area in the dotted line mode represents the preset area G.

[0089] like Figure 5AAs shown, the intracavitary image includes instrument position point A, instrument position point A9, and instrument position point A10. Based on the automatic tracking instruction type with the instruction type of "automatic", the human intelligence collaboration unit calculates the centroid position X1 of the instrument position point A, instrument position point A9, and instrument position point A10, and determines the centroid position X1 as the first tracking point. When the first tracking point is outside the preset area G, the human intelligence collaboration unit determines the first tracking point as the expected field of view point E. The mirror-holding robot responds to the action instruction corresponding to the expected field of view point E to automatically track the centroid position X1 and adjust the surgical field of view.

[0090] like Figure 5B As shown, the intracavitary image includes an instrument position point A. Based on the automatic tracking instruction type of "automatic", the human intelligence collaboration unit determines the instrument position point A as the second tracking point. When the second tracking point is outside the preset area G, the human intelligence collaboration unit determines the second tracking point as the desired field of view point E. The mirror-holding robot responds to the action instruction corresponding to the desired field of view point E to automatically track the instrument position point A and adjust the surgical field of view.

[0091] like Figure 5C As shown, the intracavitary image includes instrument position point A, instrument position point A11, and instrument position point A12. Based on the automatic tracking instruction type of "tracking" + "instrument type", such as "tracking electric hook", the human intelligence collaboration unit determines the instrument position point A12 of the electric hook as the third tracking point. When the third tracking point is outside the preset area G, the human intelligence collaboration unit determines the third tracking point as the desired field of view point. The mirror-holding robot responds to the action instruction corresponding to the desired field of view point E to automatically track the instrument position point A12 of the electric hook and adjust the surgical field of view.

[0092] like Figure 5A , Figure 5B , Figure 5CAs shown, based on the control object's demand for the surgical field of view, the control object sends a corresponding voice signal. Corresponding tracking points are generated according to different automatic tracking instructions, for example, the centroid position X1 when the instruction information is "automatic", for example, there is only one instrument position point A of a surgical instrument in use, for example, the instrument position point A12 of the "instrument type" when the instruction information is "tracking" + "instrument type". When the tracking point is not within the preset area G, based on the instruction type being the automatic tracking instruction type, the human-intelligence collaboration unit determines the tracking point as the expected field of view point E. The mirror-holding robot responds to the action instruction corresponding to the expected field of view point E, moves the expected field of view point E to the center of the preset area G, and completes an action. When the tracking point leaves the preset area G again, the human-intelligence collaboration unit generates a new expected field of view point E to achieve automatic tracking and adjust the surgical field of view. According to an embodiment of the present invention, the human-intelligence collaborative laparoscopic surgery mirror-holding robot system also includes a display for displaying an intracavitary image to the control object, so that the control object provides a voice signal to the voice interaction module based on the intracavitary image.

[0093] According to an embodiment of the present invention, the system is configured to switch the working mode from the initial mode to the direction mode, the directional attention mode or the automatic tracking mode in response to a voice signal.

[0094] According to an embodiment of the present invention, when the system is configured to switch the working mode to a direction mode or a directional focus mode, the human intelligence collaboration unit is configured to output a desired field of view point, and after the mirror-holding robot responds to an action instruction generated based on an expected field of view point and controls the laparoscope to complete an action, the system is configured to switch the working mode to the initial mode.

[0095] According to an embodiment of the present invention, when the system is configured to switch the working mode to the automatic tracking mode, the mirror-holding robot is configured to control the laparoscope to complete an action in response to the action instruction generated based on the expected field of view point whenever the human intelligence collaboration unit outputs the expected field of view point.

[0096] According to an embodiment of the present invention, the system is further configured to switch the working mode to a working mode corresponding to the new voice signal in response to receiving the new voice signal.

[0097] When the human-intelligence collaborative laparoscopic surgery mirror-holding robot system of the embodiment of the present invention is applied to laparoscopic surgery, based on different voice command information, the mirror-holding robot has the following four working modes: initial mode, direction mode, directional attention mode and automatic tracking mode.

[0098] According to an embodiment of the present invention, the laparoscope continuously sends intracavitary images to the control module, the voice recognition unit continuously sends voice recognition results to the human intelligence collaboration unit, the depth perception unit continuously outputs depth maps, and the scene perception unit continuously outputs instrument position points and tissue position points.

[0099] When the controlled object does not send a voice command signal, the mirror holding robot works in the initial mode. When the controlled object does not send a voice command signal, the voice recognition unit stops sending the voice recognition result to the human intelligence collaboration unit, and the mirror holding robot does not make any specific action to keep the intracavitary image obtained by the laparoscope stable.

[0100] When the voice command signal sent by the controlled object is "stop", the visual servo unit in the control module stops all outputs, so that the mirror-holding robot immediately jumps out of the current motion state or the current state of actively adjusting the surgical field of view and enters the initial mode.

[0101] When the voice command signal issued by the controlled object is of the direction command type, the mirror-holding robot works in the direction mode. The human intelligence collaboration unit generates the expected field of view point based on the specific voice command signal and the corresponding processing strategy. The visual servo unit generates an action command based on the expected field of view point, the depth map and the position point of the center of the intracavitary image, and transmits it to the mirror-holding robot to adjust the surgical field of view. When the visual servo unit determines that the distance between the position point of the center of the intracavitary image after adjustment and the expected field of view point is less than the threshold, the mirror-holding robot completes the action. In response to an action command generated by an expected field of view point, after completing an action, the mirror-holding robot will jump out of the direction mode and switch to the initial mode.

[0102] When the voice command signal issued by the control object is of the directional attention command type, the mirror-holding robot works in the directional attention mode. The human intelligence collaboration unit generates the expected field of view point based on the specific voice command signal and the corresponding processing strategy. The visual servo unit generates an action command based on the expected field of view point, the depth map and the position point of the center of the intracavitary image, and transmits it to the mirror-holding robot to adjust the surgical field of view. When the visual servo unit determines that the distance between the position point of the center of the intracavitary image after adjustment and the expected field of view point is less than the threshold, the mirror-holding robot completes the action. In response to an action command generated by an expected field of view point, after completing an action, the mirror-holding robot will jump out of the directional attention mode and switch to the initial mode.

[0103] When the voice command signal issued by the controlled object is of the automatic tracking command type, the mirror holding robot works in the automatic tracking mode. The human intelligence collaboration unit determines a tracking point and outputs an expected field of view point based on the specific voice command signal and the corresponding processing strategy. The visual servo unit generates an action instruction based on the expected field of view point, the depth map and the position point of the center of the intracavitary image, and transmits it to the mirror holding robot to adjust the surgical field of view. When the visual servo unit determines that the distance between the position point of the center of the adjusted intracavitary image and the expected field of view point is less than the threshold, the mirror holding robot completes an action. After completing an action, the mirror holding robot is still in the automatic tracking mode. When the tracking point leaves the preset area, a new expected field of view point is generated, and the visual servo unit is used to control the mirror holding robot to adjust the surgical field of view again. Until the controlled object issues new voice command information again.

[0104] In the human-intelligence-cooperative laparoscopic surgical mirror-holding robot system of the embodiment of the present invention, the mirror-holding robot can understand the surgical scene information and automatically generate multiple position points for the surgical instruments and key tissues in the intracavitary image of the current operation. Based on these multiple position points representing the positions of surgical instruments and key tissues, multiple potential expected field of view points are generated. When receiving voice command information from the controlled object, such as "left", the mirror-holding robot will automatically select the expected field of view point that best meets the "left" command as the ideal field of view center based on the expected field of view point, and automatically generate a specific movement direction and movement amount to control the mirror-holding robot to adjust the surgical field of view. Therefore, the controlled object only needs to send a simple voice signal once, and the mirror-holding robot can automatically adjust to the most ideal surgical field of view.

[0105] The human-intelligence-cooperative laparoscopic surgery mirror-holding robot system of the embodiment of the present invention is intervened by the control object, and the control object sends an overall adjustment opinion to specifically adjust the surgical field of view. In most cases, this field of view adjustment opinion must be the real ideal surgical field of view that the control object wants. Then the mirror-holding robot completes the specific field of view adjustment work according to the adjustment intention, because the mirror-holding robot completes the field of view adjustment according to the determined adjustment intention, so the field of view adjustment accuracy is greatly improved.

[0106] In summary, the human-intelligence-cooperative laparoscopic surgery mirror-holding robot system of the embodiment of the present invention only requires the controlled object to send the intention of adjusting the field of view once by voice (the most convenient way), which will not cause too much operating burden to the controlled object and will not interrupt the surgical process. The control module actively disassembles the intention of adjusting the field of view into specific movement directions and movement amounts based on the simple but clear intention of adjusting the field of view of the controlled object, and then plans a specific movement route, and finally the mirror-holding robot performs the adjustment of the surgical field of view.

[0107] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A human-intelligence collaborative laparoscopic surgery scope-holding robot system, characterized in that: The system comprises: Laparoscope, scope-holding robot, voice interaction module and control module; Wherein, the laparoscope is used to collect intracavitary images of the object during surgery and send the intracavitary images to the control module; The voice interaction module is used to receive a voice signal of a controlled object and send the voice signal to the control module; The control module is used to generate an action instruction for adjusting the field of view based on the intracavitary image and the voice signal, and send the action instruction to the mirror-holding robot; and The scope holding robot is used to fix the laparoscope and, in response to the action instruction, control the movement of the laparoscope to adjust the intraoperative field of view; Wherein, the control module includes a human intelligence collaboration unit, and the control module is further used to identify the voice signal, obtain a voice recognition result, and determine a plurality of position points based on the intracavitary image, perform depth perception based on the intracavitary image, and obtain a depth map, wherein the plurality of position points include at least one instrument position point and at least one tissue position point, and the human intelligence collaboration unit is used to determine an expected field of view point based on the voice recognition result and the plurality of position points, so that the control module generates the action instruction using the expected field of view point and the depth map, wherein the expected field of view point represents the center of the surgical field of view that the control object expects to obtain; Among them, the human intelligence collaboration unit is specifically used to determine the instruction type based on the voice instruction information included in the voice recognition result when the voice recognition result indicates that the voice signal is valid, and when the instruction type is a direction instruction type, determine the target image area from the multiple image areas included in the intracavitary image based on the voice instruction information, and determine the expected field of view point based on the position point contained in the target image area; wherein the multiple image areas are obtained based on the center division of the intracavitary image.

2. The system according to claim 1, characterized in that The control module also includes: Speech recognition unit, depth perception unit, scene perception unit and visual servoing unit; Wherein, the speech recognition unit is used to recognize the speech signal and obtain a speech recognition result; The depth sensing unit is used to perform depth sensing based on the intracavity image to obtain a depth map; The scene perception unit is used to determine a plurality of position points based on the intracavitary image; and The visual servoing unit is used to output the action instruction based on the depth map and the expected field of view point.

3. The system according to claim 2, characterized in that The scene perception unit comprises: An image segmentation subunit, used for performing image segmentation processing on the intracavitary image to obtain a mask image; an instrument position point determination subunit, configured to extract an instrument mask image block from the mask image, and determine the at least one instrument position point based on the instrument mask image block; and The tissue position point determination subunit is used to extract a tissue mask image block from the mask image, and determine the at least one tissue position point based on the tissue mask image block.

4. The system according to claim 3, characterized in that The instrument position point determination subunit is specifically used for: Performing connected area division on the instrument mask image block to obtain at least one first instrument connected area; Performing noise filtering on the at least one first device communication region to obtain at least one second device communication region; For each of the second device communication regions, skeletonizing the second device communication region to obtain a device skeleton; Based on the principal component analysis method, extracting instrument line segments for indicating the start and end positions of the instrument from the instrument skeleton; determining an end point of the instrument based on a position of the instrument line segment in the intracavitary image; as well as Based on the end point of the instrument, an instrument position point corresponding to the second instrument communication area is obtained.

5. The system according to claim 3, characterized in that The tissue location point determination subunit is specifically used for: Based on the label of each of the at least one tissue connected region included in the tissue mask image, determining at least one key tissue connected region related to the key tissue from the at least one tissue connected region; as well as For each of the key tissue connectivity regions, a tissue position point corresponding to the key tissue connectivity region is obtained based on the first-order moment of the key tissue connectivity region.

6. The system according to claim 2, characterized in that The human intelligence collaboration unit is specifically used for: The desired view point is determined based on the multiple position points using a processing strategy corresponding to the instruction type.

7. The system according to claim 6, characterized in that The instruction type includes a direction instruction type; The human intelligence collaboration unit is specifically used for: In a case where the target image area only includes a first position point, determining the first position point as the desired field of view point, wherein the first position point belongs to the multiple position points; In a case where the target image area includes a plurality of second position points, and the plurality of second position points are all instrument position points, determining the expected field of view point based on the centroid positions of the plurality of second position points, wherein the plurality of second position points belong to the plurality of position points; In a case where the target image area includes a plurality of third position points, and the plurality of third position points are all tissue position points, determining the desired field of view point based on the centroid positions of the plurality of third position points, wherein the plurality of third position points belong to the plurality of position points; In a case where the target image area includes a plurality of fourth position points, and the plurality of fourth position points include instrument position points and tissue position points, a first centroid position is determined based on the instrument position points included in the plurality of fourth position points, a second centroid position is determined based on the tissue position points included in the plurality of fourth position points, and the desired field of view point is determined based on a line connecting the first centroid position and the second centroid position, wherein the plurality of fourth position points belong to the plurality of position points; and In the case that the target image area does not include the position point, a reverse direction boundary is determined based on the voice instruction information, and the desired field of view point is determined based on the distance between each of the at least one instrument position point and the reverse direction boundary.

8. The system according to claim 6, characterized in that The instruction type includes a directed attention instruction type; The human intelligence collaboration unit is specifically used for: Determining a first target device or target tissue based on the voice command information; and The desired field of view point is determined based on an instrument position point corresponding to the first target instrument or a tissue position point corresponding to the target tissue among the plurality of position points.

9. The system according to claim 6, characterized in that The instruction type includes an automatic tracking instruction type; The human intelligence collaboration unit is specifically used for: Based on the voice command information, determining a tracking point; and Whenever the tracking point is outside a preset area, the expected view point is determined based on the tracking point.

10. The system according to claim 2, characterized in that The system is configured to switch the working mode from the initial mode to the direction mode, the directional attention mode or the automatic tracking mode in response to the voice signal; Wherein, when the system is configured to switch the working mode to the direction mode or the directional attention mode, the human intelligence collaboration unit is configured to output one of the desired view points, and after the scope holding robot controls the laparoscope to complete one action in response to an action instruction generated based on one of the desired view points, the system is configured to switch the working mode to the initial mode; In the case where the system is configured to switch the working mode to the automatic tracking mode, the scope holding robot is configured to control the laparoscope to complete an action in response to an action instruction generated based on the expected view point whenever the human intelligence collaboration unit outputs the expected view point; The system is also configured to, in response to receiving a new voice signal, switch the operating mode to an operating mode corresponding to the new voice signal.

Images