Automatic exposure method of image acquisition device, surgical equipment and computer program product

Abstract

The invention provides an automatic exposure method of an image acquisition device, surgical equipment and a computer program product. The method comprises the following steps: acquiring a relative position of a surgical instrument relative to an exposure reference area of the image acquisition device; on the basis of the change of the relative position, when it is monitored that the surgical instrument enters the exposure reference area, the exposure reference area is updated according to the area where the surgical instrument is located in the exposure reference area, and the updated exposure reference area is obtained; and recalculating the exposure parameter by taking the updated exposure reference area as a reference, and performing automatic exposure on the image acquisition device according to the calculated exposure parameter. According to the method, the flickering phenomenon caused by light reflection of the surgical instrument can be effectively avoided, so that the stability and quality of the image are improved. And the image acquisition device can automatically adapt to more scenes, and manual adjustment of exposure parameters is avoided, so that the efficiency of minimally invasive surgery applying the image acquisition device is improved.

Description

Technical Field

[0001] The present application relates to the field of medical devices, and in particular, to an automatic exposure method of an image acquisition device, surgical equipment, and a computer program product. Background Art

[0002] Compared with traditional surgery, minimally invasive surgery has the advantages of less trauma, faster recovery, and reduced pain for patients. Image acquisition devices such as endoscopes in minimally invasive medical devices serve as the "eyes" of doctors, which can effectively help doctors "see" lesions clearly. When doctors perform related surgical operations, they require endoscopes to have extremely high image quality and image stability. Any image error may affect the entire process of the operation. One of the image qualities that doctors are most concerned about is the stability of the image brightness. The technology directly related to it is automatic exposure (there is also manual exposure, but the efficiency is too low and the doctor's operation requirements are high, which is currently not a mainstream technology). There are many factors that affect automatic exposure. When the robustness of the automatic exposure algorithm is poor, it will lead to errors in the decision-making of the image brightness, which will affect the doctor's judgment during the operation.

[0003] Currently, most endoscope host automatic exposure algorithms on the market use a universal metering strategy, that is, full-image or local area weighted metering. Because it is a universal strategy, occasional exposure errors in different scenarios, such as instrument reflections or frequent entry and exit, resulting in "bright and dark" images are inevitable, but this is difficult to accept in high-demand scenarios such as minimally invasive surgery. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an exposure method, surgical equipment and computer program product for a surgical instrument, which can effectively avoid the flickering phenomenon caused by the reflection of the surgical instrument, thereby improving the stability and quality of the image.

[0005] In a first aspect, an embodiment of the present application provides an automatic exposure method for an image acquisition device, comprising: acquiring the relative position of a surgical instrument relative to an exposure reference area of ​​the image acquisition device; based on a change in the relative position, when it is detected that the surgical instrument enters the exposure reference area: updating the exposure reference area according to an area in the exposure reference area where the surgical instrument is located to obtain an updated exposure reference area; and recalculating exposure parameters based on the updated exposure reference area, and performing automatic exposure on the image acquisition device according to the calculated exposure parameters.

[0006] The automatic exposure method of the above-mentioned image acquisition device can effectively avoid the flickering phenomenon caused by the reflection of the surgical instrument by updating the exposure reference area according to the area where the surgical instrument is located in the exposure reference area, and performing exposure with the updated exposure reference area, thereby improving the stability and quality of the image. It can also automatically adapt to more scenes, avoiding manual adjustment of exposure parameters, thereby improving the efficiency of minimally invasive surgery using the image acquisition device.

[0007] In combination with the first aspect, optionally, updating the exposure reference area according to the area where the surgical instrument is located in the exposure reference area includes: judging whether there is overexposure in the area where the surgical instrument is located in the exposure reference area; if it is determined that there is overexposure, eliminating the area in the exposure reference area where the surgical instrument is located or the area where the surgical instrument is located that has overexposure.

[0008] The automatic exposure method of the above-mentioned image acquisition device eliminates the entire area where the surgical instrument is located or the area with overexposure when the area where the surgical instrument is located is overexposed, so that the details of the image, especially the area outside the area where the surgical instrument is located, are clearer, and the influence of other parts of the image appearing too dim due to the overexposed area is eliminated, thereby further improving the stability and quality of the image.

[0009] In combination with the first aspect, optionally, determining whether the area where the surgical instrument is located in the exposure reference area is overexposed includes: determining an overexposed area in the exposure reference area where the surgical instrument is located whose brightness value exceeds a brightness threshold; calculating the area of ​​the area where the surgical instrument is located in the exposure reference area and the area of ​​the overexposed area; and determining whether the area where the surgical instrument is located in the exposure reference area is overexposed based on whether the ratio of the area of ​​the overexposed area to the area of ​​the area where the surgical instrument is located in the exposure reference area exceeds a first ratio threshold.

[0010] The automatic exposure method of the above-mentioned image acquisition device determines that the area with a brightness value exceeding the brightness threshold in the exposure reference area is an overexposed area, and when the ratio of the area of ​​the overexposed area to the area of ​​the area of ​​the area of ​​interest exceeds the first ratio threshold, it is determined that the area of ​​interest is overexposed. The exposure error problem caused by the reflection of the instrument is reduced, and the quality of the image in the image acquisition device is further improved. This also makes the operation smoother.

[0011] In combination with the first aspect, optionally, updating the exposure reference area according to the area where the surgical instrument is located in the exposure reference area includes: eliminating the area where the surgical instrument is located in the exposure reference area.

[0012] The automatic exposure method of the above-mentioned image acquisition device directly eliminates the entire area where the surgical instrument is located, regardless of how the reflectivity of the surgical instrument changes, and ensures that the exposure of the background area is always in the best state. Even if the position and posture of the surgical instrument change during the operation, the exposure parameters can be quickly adapted and adjusted to maintain the stability of the image quality, so that the automatic exposure method of the image acquisition device provided in the embodiment of the present application can adapt to more scenarios. In addition, regardless of whether there is overexposure in the area where the surgical instrument is located, the entire area where the surgical instrument is located is directly eliminated, and the step of determining whether there is overexposure is omitted, thereby simplifying the automatic exposure method of the image acquisition device and improving the response speed of the system.

[0013] In combination with the first aspect, optionally, the removing of the area in the exposure reference area where the surgical instrument is located includes: judging whether the proportion of the area in the exposure reference area where the surgical instrument is located is less than a second proportion threshold; if it is determined that the proportion of the area where the surgical instrument is located is less than the second proportion threshold, removing the area in the exposure reference area where the surgical instrument is located; otherwise, not performing any processing on the exposure reference area.

[0014] The automatic exposure method of the above-mentioned image acquisition device directly eliminates the entire area where the surgical instrument is located only when the proportion of the area where the surgical instrument is located is less than the second proportion threshold, thereby ensuring that the exposure has a sufficient exposure reference area, so that more appropriate exposure parameters can be calculated. Ultimately, the overall quality of the image is ensured.

[0015] In combination with the first aspect, optionally, obtaining the relative position of the surgical instrument relative to the exposure reference area of ​​the image acquisition device includes: obtaining the three-dimensional coordinates of at least two marked points on the surgical instrument and the three-dimensional coordinates of at least two marked points on the image acquisition device; determining the three-dimensional surgical instrument equation of the surgical instrument in the three-dimensional space according to the three-dimensional coordinates of the at least two marked points on the surgical instrument; determining the three-dimensional image acquisition device equation of the image acquisition device in the three-dimensional space according to the three-dimensional coordinates of the at least two marked points on the image acquisition device; and determining the relative position of the surgical instrument relative to the exposure reference area according to the three-dimensional surgical instrument equation and the three-dimensional image acquisition device equation.

[0016] The automatic exposure method of the above-mentioned image acquisition device determines the corresponding equation in the three-dimensional space through the three-dimensional coordinates of the surgical instrument and the mark on the image acquisition device, and determines the relative position of the surgical instrument relative to the exposure reference area according to the equation, thereby improving the accuracy of judging the position of the surgical instrument, thereby being more conducive to calculating more accurate exposure parameters and eliminating the influence of the surgical instrument on the exposure. Finally, the overall quality of the image is further ensured.

[0017] In combination with the first aspect, optionally, determining the relative position of the surgical instrument with respect to the exposure reference area based on the three-dimensional surgical instrument equation and the three-dimensional image acquisition device equation includes: determining the two-dimensional plane boundary equation of the image acquired by the image acquisition device based on the three-dimensional image acquisition device equation and the field of view angle parameter of the image acquisition device; converting the three-dimensional surgical instrument equation into a two-dimensional surgical instrument equation within the two-dimensional plane based on the three-dimensional surgical instrument equation and the two-dimensional plane boundary equation; determining the relative position of the surgical instrument with respect to the exposure reference area based on the two-dimensional surgical instrument equation and the two-dimensional plane boundary equation.

[0018] The automatic exposure method of the above-mentioned image acquisition device further improves the accuracy of judgment by converting the three-dimensional surgical instrument equation into the two-dimensional surgical instrument equation in the image plane acquired by the image acquisition device, and judging whether the surgical instrument is within the field of view of the image acquisition device according to the two-dimensional surgical instrument equation, thereby further improving the efficiency of exposing the surgical instrument.

[0019] In combination with the first aspect, optionally, the three-dimensional image acquisition device equation includes:

[0020]

[0021] The three-dimensional surgical instrument equation includes:

[0022]

[0023] In the formula, x c ,y c 、z c respectively represent the coordinate values ​​of any point on the equation of the three-dimensional image acquisition device, x1, y1, z1 respectively represent the three-dimensional coordinates of one of the marked points on the image acquisition device, x2, y2, z2 respectively represent the three-dimensional coordinates of another marked point on the image acquisition device, x s ,y s 、z sThey respectively represent the coordinate value of any point on the three-dimensional surgical instrument equation, x3, y3, and z3 respectively represent the three-dimensional coordinates of one of the marked points on the surgical instrument, and x4, y4, and z4 respectively represent the three-dimensional coordinates of another marked point on the surgical instrument.

[0024] The automatic exposure method of the above-mentioned image acquisition device is based on the principle of determining a straight line from two points, and the straight line equation corresponding to the image acquisition device in three-dimensional space is determined according to the two marked points on the image acquisition device. The straight line equation corresponding to the surgical instrument in three-dimensional space is determined according to the two marked points on the surgical instrument.

[0025] In a second aspect, the present application also provides a surgical device, including a surgical instrument, an image acquisition device and a processor; the surgical instrument and the image acquisition device are electrically connected to the processor respectively; the surgical instrument is used to perform surgery under the control of the processor; the image acquisition device is used to acquire images under the control of the processor; the processor is used to: obtain the relative position of the surgical instrument relative to an exposure reference area of ​​the image acquisition device; based on the change of the relative position, monitor when the surgical instrument enters the exposure reference area; update the exposure reference area according to the area where the surgical instrument is located in the exposure reference area to obtain an updated exposure reference area; and recalculate exposure parameters based on the updated exposure reference area, and perform automatic exposure on the image acquisition device according to the calculated exposure parameters.

[0026] The above-mentioned surgical equipment has the same beneficial effects as the exposure method of the surgical instrument described in the first aspect, and will not be described in detail here.

[0027] In a third aspect, the present application further provides a computer program product, including a computer program / instruction, which implements the method described in the first aspect when executed by a processor.

[0028] The above-mentioned computer program product has the same beneficial effects as the exposure method of the surgical instrument described in the first aspect, and will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 Demonstration diagram of surgery using surgical instruments and endoscopes;

[0031] Figure 2 A flow chart of an automatic exposure method for an image acquisition device provided in an embodiment of the present application;

[0032] Figure 3 A flow chart of step S240 in the automatic exposure method of the image acquisition device provided in an embodiment of the present application;

[0033] Figure 4 A flow chart of step S241 in the automatic exposure method of the image acquisition device provided in an embodiment of the present application;

[0034] Figure 5 A flow chart of step S243 in the automatic exposure method of the image acquisition device provided in an embodiment of the present application;

[0035] Figure 6 A flow chart of step S220 in the automatic exposure method of the image acquisition device provided in an embodiment of the present application;

[0036] Figure 7 A flow chart of step S224 in the automatic exposure method of the image acquisition device provided in an embodiment of the present application;

[0037] Figure 8 A schematic diagram of the projection of a surgical instrument in the field of view of an image acquisition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0039] In this specification, many specific technical details are described in some places so that those skilled in the art can understand the complete technical solution. However, it should be understood that the embodiments of the present application can be implemented without these specific technical details. Such detailed description of technical details should not be regarded as a limitation to the present application, and the protection scope of the present application is limited only by the claims. In other places, known structures, connection / position relationships, circuits and / or other details may not be shown in detail to avoid misunderstandings by the public about the main points of the invention of the present application.

[0040] In this specification, the drawings show schematic diagrams of several embodiments of the present application. However, the drawings are only schematic, and it should be understood that changes in mechanical structure, connection / position relationship, physical composition, electrical and steps can be made without departing from the spirit and scope of the present application. Such changes can be either replacement or combination of elements of several embodiments of the present application, or replacement or combination of known contents.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0042] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0043] The terms used herein below are only used to describe specific embodiments and are not intended to limit the present application. Spatially relative terms, such as "below", "lower", "above", "upper", "middle", "middle", "inside", "outside", "center", "edge", etc., are used to describe the relationship between one component or feature shown in the figure and another component or feature for the convenience of explanation. It should be understood that spatially relative terms can only be used under the conditions of the positioning direction of the device in use or operation (except for the positioning direction specifically defined in the figure), and are not necessarily unique and unchanging. For example, if the device in the figure is flipped 180° up and down along the paper, then the elements described as being "below" other components or features will become "above" other components or features. Therefore, the exemplary term "below" can cover both the above and below directions, depending on how the device is positioned. The device can also be positioned in other directions (for example, rotated 90° or positioned in other directions), and the spatially relative descriptors used herein are interpreted accordingly.

[0044] As used herein, "several", "one", and "the" are intended to include plural forms as well, unless the context indicates otherwise. It should be further understood that the terms "include" and / or "comprise" specify the presence of stated features, steps, operations, elements, and / or parts, but do not exclude the presence of one or more other features, steps, operations, elements, parts, and / or groups thereof.

[0045] The term "object" generally refers to a component or a group of components. Throughout the specification and claims, the terms "object", "component", "portion", "part", "module", "assembly" and "element" are used interchangeably.

[0046] The terms "instrument", "surgical instrument" and "surgical instrument" are used herein to describe a medical device configured to be inserted into a patient and used to perform a surgical or diagnostic procedure, generally including an end effector. An end effector may be a surgical tool associated with one or more surgical procedures, such as forceps, needle holders, scissors, bipolar cauterizers, tissue stabilizers or retractors, clip appliers, stapling devices, imaging devices (e.g., ultrasound probes), and the like. Some instruments used in embodiments of the present application further provide an articulated support (sometimes referred to as a "wrist joint", "joint seat") for the surgical tool, so that the position and / or orientation of the end effector can be flexibly manipulated with one or more mechanical degrees of freedom relative to the instrument axis. Further, many end effectors include functional mechanical degrees of freedom, such as jaws that open or close or blades that translate along a specific path. The instrument may also contain information stored (e.g., on a PCBA board within the instrument) that is permanent or updateable by the surgical system. Accordingly, the system may provide one-way or two-way information communication between the instrument and one or more system components.

[0047] The term "mating" (sometimes referred to as "connecting", "coupling", "mounting", "assembling") can be broadly understood as any situation in which two or more objects are connected in a manner that allows the mating objects to operate in conjunction with each other. It should be noted that mating does not require a direct connection (e.g., a direct physical or electrical connection), but rather many objects or components can be used to mate two or more objects. For example, objects A and B can be mated using object C. In addition, the terms "removably coupled" or "removably mated" can be interpreted to mean a non-permanent connection or mating situation between two or more objects. This means that the removably coupled objects can be uncoupled and separated so that they no longer operate in conjunction.

[0048] Finally, the terms "or" and "and / or" as used herein should be interpreted as inclusive or meaning any one or any combination. Thus, "A, B, or C" or "A, B and / or C" means any one of the following: A; B; C; A and B; A and C; B and C; A, B, and C. An exception to this definition will only occur when a combination of elements, functions, steps, or actions are inherently mutually exclusive in some way.

[0049] Overview of Master-Slave Teleoperated Laparoscopic Surgical Robot

[0050] Laparoscopic surgical robots usually include a doctor control platform, a patient surgical platform, and an image platform. The surgeon sits on the doctor control platform and watches the two-dimensional or three-dimensional image of the surgical area transmitted by the laparoscope (sometimes called an "endoscope") placed in the patient's body, and controls the movement of the robotic arm on the patient surgical platform, as well as the surgical instruments or laparoscopes attached to the robotic arm. The robotic arm simulates the human arm, and the surgical instrument simulates the human hand. Both provide surgeons with a series of movements that simulate the human wrist, while filtering the tremor of the human hand itself. Therefore, they are increasingly used in surgery, especially in abdominal, thoracic, and general surgery.

[0051] The patient surgical platform generally includes a chassis, a column, a plurality of mechanical arms connected to the column, and one or more surgical instrument manipulators at the end of the support assembly of each mechanical arm. The surgical instrument and / or laparoscope is detachably connected to the surgical instrument manipulator. Each surgical instrument manipulator supports one or more surgical instruments and / or laparoscopes operated at the surgical site in the patient's body. Each surgical instrument manipulator can be allowed to control the relevant surgical instruments in various forms of one or more mechanical degrees of freedom (e.g., all six Cartesian degrees of freedom, five or less Cartesian degrees of freedom, etc.). Typically, each surgical instrument manipulator is limited by mechanical or software constraints to rotate the relevant surgical instrument around the center of motion on the surgical instrument that remains stationary relative to the patient, and the center of motion is generally located at the position where the surgical instrument enters the body wall, and the center of motion is generally referred to as the "distal center" or "fixed point".

[0052] The imaging platform usually includes a video image capture function (commonly an endoscope) and one or more video displays for displaying surgical instruments in the captured images. In some laparoscopic surgical robots, an optical device is included to transmit images from the patient's body to one or more imaging sensors (e.g., CCD or CMOS sensors) at the distal end of the endoscope, and then transmit the video image to the host of the imaging platform through steps such as photoelectric conversion. Subsequently, through image processing, the processed image is displayed on a video display for observation by other doctors or assistants.

[0053] The doctor control platform usually includes a chassis, a pedal assembly, a stereo monitor, a main control arm, and a manual controller connected to the end of the main control arm. The surgeon controls the manual controller and the pedal assembly to achieve specific movements and / or energy excitation of the surgical instrument. The surgeon control platform can be at a single location in the surgical system composed of a laparoscopic surgical robot or it can be distributed at two or more locations in the system, and the remote control master / slave operation can be completed according to the preset control degree, for example, one location is used as the main control for the main operation, and the other location is used as the auxiliary control for the auxiliary operation. The main control completes the main surgical operation, and the auxiliary control completes the auxiliary operation such as laparoscope movement or tissue traction. In some embodiments, the manual controller can be an input device capable of completing one or more manual operations, such as a joystick, an exoskeleton glove, a power and gravity compensation manipulator, etc. These input devices collect the surgeon's operation signal, and after being processed by the control system, generate control signals of the robotic arm and the surgical instrument manipulator, thereby controlling the remote control motor on the surgical instrument manipulator, and the motor further controls the final movement of the surgical instrument.

[0054] Generally, the force generated by the remote motor is transmitted via a transmission system, which transmits the force from the remote motor to the end effector of the surgical instrument. In some teleoperated surgical embodiments, the input device for controlling the manipulator may be located at a location remote from the patient, in or outside the room where the patient is located, or even in a different city. The input signal from the input device is then transmitted to the control system. Those familiar with teleoperation, telecontrol and telepresentation surgery will be familiar with such systems and their components and will not be described in detail here.

[0055] Please refer to Figure 1 , Figure 1 This is a demonstration diagram of a surgical instrument 130 and an endoscope 120 performing a surgery. Taking abdominal surgery as an example, before the formal operation begins, the doctor needs to use a puncture device to make several holes in the patient's abdomen (that is, the body wall 140) and insert cannulas 110 (usually 3 to 5, for the sake of demonstration, Figure 1 Only the endoscope 120 and a surgical instrument 130 are shown in the figure. After the endoscope 120 and the surgical instrument 130 enter the abdominal cavity of the human body through the cannula 110, the doctor can observe the lesion area 150 through the endoscope 120 and operate the surgical robot through the doctor's trolley to complete the relevant surgery.

[0056] Please refer to Figure 2 , Figure 2 : is a flow chart of the automatic exposure method of the image acquisition device provided by the embodiment of the present application. The automatic exposure method of the image acquisition device provided by the embodiment of the present application is executed by a processing host, and the method may include:

[0057] Step S220: Acquire the relative position of the surgical instrument relative to the exposure reference area of ​​the image acquisition device.

[0058] In the above step S220, the surgical instrument may be a surgical forceps, a surgical needle, etc., and may be installed on a surgical robot. The image acquisition device may be an endoscope, etc. During the production and processing of image acquisition devices such as endoscopes, due to irresistible factors such as material processing errors and assembly errors, even if the endoscopes are of the same model, the generated images will have slight differences. It is necessary to calibrate and correct key parameters (such as field of view angle, viewing angle, length, diameter, internal and external parameters of the endoscope) and store them in the endoscope parameter search and matching library. When the endoscope is inserted into the image processing host, the host can load the corresponding parameters attached to it by reading the endoscope identity ID for subsequent image processing.

[0059] Similarly, in the surgical robot system, the surgical instruments need to be calibrated. First, during the production of surgical instruments, the relevant parameters of the surgical instruments will be different due to the different types and uses of surgical instruments. Therefore, relevant parameter calibration is required during the production process.

[0060] After completing the above initialization process, the three-dimensional space reconstruction system can be used to perform real-time detection of the image acquisition device, surgical instruments, etc. according to preset rules, such as position and posture information, so as to obtain the relative position of the surgical instrument relative to the exposure reference area of ​​the image acquisition device.

[0061] Step S230: Based on the change in relative position, monitor whether the surgical instrument enters the exposure reference area.

[0062] In the above step S230, it can be determined whether the surgical instrument is currently located in the exposure reference area based on the relative position monitored by the three-dimensional space reconstruction system.

[0063] If it is detected that the surgical instrument enters the exposure reference area, step S240 is executed: the exposure reference area is updated according to the area where the surgical instrument is located in the exposure reference area to obtain an updated exposure reference area.

[0064] Step S240 updates the area where the surgical instrument is located, which may be to remove the entire area where the surgical instrument is located or to remove the overexposed portion of the area where the surgical instrument is located if the area where the surgical instrument is located is overexposed. Alternatively, the entire area where the surgical instrument is located may be directly removed regardless of whether the area where the surgical instrument is located is overexposed.

[0065] Step S260: recalculating exposure parameters based on the updated exposure reference area, and performing automatic exposure on the image acquisition device according to the calculated exposure parameters.

[0066] In the above step S260, the exposure parameters may include exposure time, gain, etc. Automatic exposure is performed using the exposure parameters calculated based on the updated exposure reference area.

[0067] In the above implementation process, by updating the exposure reference area according to the area where the surgical instrument is located in the exposure reference area, and exposing with the updated exposure reference area, the flickering phenomenon caused by the reflection of the surgical instrument can be effectively avoided, thereby improving the stability and quality of the image. It can also automatically adapt to more scenes, avoiding manual adjustment of exposure parameters, thereby improving the efficiency of minimally invasive surgery using the image acquisition device.

[0068] Please refer to Figure 3 , Figure 3 is a flow chart of step S240 in the automatic exposure method of the image acquisition device provided in the embodiment of the present application. In some optional implementations, step S240 may include:

[0069] Step S241: Determine whether there is overexposure in the area where the surgical instrument is located in the exposure reference area.

[0070] In the above step S241, it can be determined whether there is overexposure in the area where the surgical instrument is located by means of a brightness threshold or a pixel threshold.

[0071] If it is determined that overexposure exists, step S242 is executed: the area where the surgical instrument is located or the area where the surgical instrument is located with overexposure in the exposure reference area is eliminated.

[0072] In the above step S242, the weight of the area where the entire surgical instrument is located or the area where the overexposure phenomenon exists can be reduced or completely excluded, so as to eliminate the area where the entire surgical instrument is located or the area where the overexposure phenomenon exists.

[0073] In the above implementation process, when the area where the surgical instrument is located is overexposed, the entire area where the surgical instrument is located or the area with overexposure is eliminated, so that the details of the image are clearer and the influence of the overexposed area causing other parts of the image to appear too dim is eliminated, thereby further improving the stability and quality of the image.

[0074] Please refer to Figure 4 , Figure 4 is a flow chart of step S241 in the automatic exposure method of the image acquisition device provided in the embodiment of the present application. In some optional implementations, step S241 may include:

[0075] Step S2411: Determine an overexposed area in the exposure reference area where the surgical instrument is located and whose brightness value exceeds a brightness threshold.

[0076] In the above step S2411, illustratively, the brightness threshold of the 8-bit region of interest may be 255. Of course, those skilled in the art may determine the brightness threshold by themselves according to the size of the image.

[0077] Step S2412: Calculate the area of ​​the region where the surgical instrument is located and the area of ​​the overexposed region in the exposure reference area.

[0078] Step S2413: Determine whether the area of ​​the overexposed area where the surgical instrument is located in the exposure reference area is overexposed based on whether the ratio of the area of ​​the overexposed area to the area of ​​the area where the surgical instrument is located in the exposure reference area exceeds a first ratio threshold.

[0079] In the above step S2413, the first ratio threshold may be 40%, 50%, 60%, etc. When the area ratio exceeds the ratio threshold, it can be determined that the region of interest is overexposed. On the contrary, it can be determined that the region of interest is not overexposed.

[0080] It should be understood that the execution order of the above-mentioned steps S2411 and S2412 can be "execute step S2411 first, then execute step S2412", or "execute step S2412 first, then execute step S2411", or "execute step S2411 and step S2412 simultaneously".

[0081] In the above implementation process, the exposure reference area is determined to be an overexposed area by the tendency of the brightness value exceeding the brightness threshold, and when the ratio of the area of ​​the overexposed area to the area of ​​the region of interest exceeds the first ratio threshold, it is determined that the region of interest is overexposed. The exposure error problem caused by the reflection of the instrument is reduced, and the quality of the image in the image acquisition device is further improved. This also makes the operation smoother.

[0082] In some optional implementations, step S240 may include:

[0083] Step S243: Eliminate the area where the surgical instrument is located in the exposure reference area.

[0084] In the above step S243, no matter whether the area where the surgical instrument is located is overexposed or not, the entire area where the surgical instrument is located is directly eliminated.

[0085] In the above implementation process, by directly eliminating the area where the entire surgical instrument is located, no matter how the reflectivity of the surgical instrument changes, the area where the surgical instrument is located is directly eliminated, ensuring that the exposure of the background area is always in the best state. Even if the position and posture of the surgical instrument change during the operation, the exposure parameters can be quickly adapted and adjusted to maintain the stability of the image quality, so that the automatic exposure method of the image acquisition device provided in the embodiment of the present application can adapt to more scenarios. In addition, regardless of whether there is overexposure in the area where the surgical instrument is located, the area where the entire surgical instrument is located is directly eliminated, and the step of determining whether there is overexposure is omitted, thereby simplifying the automatic exposure method of the image acquisition device and improving the response speed of the system.

[0086] Please refer to Figure 5 , Figure 5 is a flow chart of step S243 in the automatic exposure method of the image acquisition device provided in the embodiment of the present application. In some optional implementations, step S243 may include:

[0087] Step S2431: Determine whether the proportion of the area where the surgical instrument is located in the exposure reference area is less than a second proportion threshold.

[0088] If it is determined that the proportion of the area where the surgical instrument is located is less than the second proportion threshold, step S2432 is executed: the area where the surgical instrument is located in the exposure reference area is removed. Otherwise, step S2433 is executed: no processing is performed on the exposure reference area.

[0089] In the above steps, illustratively, if the proportion of the area where the surgical instrument is located is less than 50%, the entire area where the surgical instrument is located is directly eliminated regardless of whether the area where the surgical instrument is located is overexposed. Otherwise, no processing is performed. Of course, those skilled in the art can determine the second proportion threshold to other values, such as 40%, 60%, etc., according to actual application requirements.

[0090] In the above implementation process, the entire area where the surgical instrument is located is directly eliminated only when the proportion of the area where the surgical instrument is located is less than the second proportion threshold, thereby ensuring that the exposure has sufficient exposure reference area, so that more appropriate exposure parameters can be calculated. Ultimately, the overall quality of the image is ensured.

[0091] Please refer to Figure 6 , Figure 6 is a flow chart of step S220 in the automatic exposure method of the image acquisition device provided in the embodiment of the present application. In some optional implementations, step S220 may include:

[0092] Step S221: Acquire the three-dimensional coordinates of at least two marked points on the surgical instrument and the three-dimensional coordinates of at least two marked points on the image acquisition device.

[0093] In the above step S221, illustratively, two marking points can be taken on the image acquisition device, and their coordinates in the three-dimensional space can be determined. Similarly, three marking points can be taken on the surgical instrument, and their coordinates in the three-dimensional space can be determined. It should be noted that the number of these marking points is determined based on the number of moving joints of the image acquisition device or the surgical instrument. Exemplarily, the end of the image acquisition device or the surgical instrument has no moving joints and can be regarded as a line segment, so two (marked) points can determine a line segment, such as a rigid endoscope or an ultrasonic scalpel. Similarly, if there is a moving joint at the end of the surgical instrument, it can be regarded as two line segments, so three marking points are required, such as an electrocoagulation hook or an anastomosis device. By analogy, the end of the surgical instrument has two moving joints and requires four marking points. This is the most common surgical instrument, such as a needle holder, curved scissors, etc.

[0094] Step S222: Determine a three-dimensional surgical instrument equation of the surgical instrument in three-dimensional space based on the three-dimensional coordinates of at least two marked points on the surgical instrument.

[0095] In the above step S222, according to the coordinates of the two marked points on the image acquisition device, the three-dimensional surgical instrument equation of the surgical instrument in the three-dimensional space can be determined. The three-dimensional surgical instrument equation is usually a linear equation. Similarly, in the case where the surgical instrument includes a linkage rod and an execution end, according to the coordinates of the three marked points on the surgical instrument, two three-dimensional surgical instrument equations can be determined, corresponding to the linkage rod and the execution end on the surgical instrument, respectively, and the equation is usually a linear equation.

[0096] Step S223: determining a three-dimensional image acquisition device equation of the image acquisition device in three-dimensional space according to the three-dimensional coordinates of at least two marked points on the image acquisition device.

[0097] In the above step S223, according to the coordinates of the two marked points on the image acquisition device, the three-dimensional image acquisition device equation of the image acquisition device in the three-dimensional space can be determined. The three-dimensional image acquisition device equation is usually a straight line equation.

[0098] Step S224: Determine the relative position of the surgical instrument with respect to the exposure reference area according to the three-dimensional surgical instrument equation and the three-dimensional image acquisition device equation.

[0099] In the above step S224, the image acquisition device usually has parameters such as the field of view angle when it leaves the factory. According to the equation of the three-dimensional image acquisition device and the field of view angle parameters, the equation of the field of view range boundary of the image acquisition device can be determined. According to the equation of the field of view range boundary and the three-dimensional surgical instrument equation, the relative position of the surgical instrument relative to the exposure reference area can be determined.

[0100] In the above implementation process, the corresponding equation in the three-dimensional space is determined by the three-dimensional coordinates of the surgical instrument and the mark on the image acquisition device, and the relative position of the surgical instrument relative to the exposure reference area is determined according to the equation, which improves the accuracy of judging the position of the surgical instrument, thereby being more conducive to calculating more accurate exposure parameters and eliminating the influence of the surgical instrument on the exposure. Finally, the overall quality of the image is further ensured.

[0101] Please refer to Figure 7 , Figure 7 is a flow chart of step S224 in the automatic exposure method of the image acquisition device provided in the embodiment of the present application. In some optional implementations, step S224 may include:

[0102] Step S2241: Determine the two-dimensional plane boundary equation of the image captured by the image capture device according to the three-dimensional image capture device equation and the field of view angle parameters of the image capture device.

[0103] In the above step S2241, the factory parameters of the image acquisition device usually include its preset working distance. Combining the working distance with the field of view boundary equation can determine the two-dimensional plane boundary equation of the image acquired by the image acquisition device in the three-dimensional space.

[0104] Step S2242: According to the three-dimensional surgical instrument equation and the two-dimensional plane boundary equation, the three-dimensional surgical instrument equation is converted into a two-dimensional surgical instrument equation within the two-dimensional plane.

[0105] In the above step S2242, in the above step S2442, please refer to Figure 8 , Figure 8 Schematic diagram of the projection of the surgical instrument in the field of view of the image acquisition device provided by the embodiment of the present application. In the figure, M1, M2, and M3 may be three marked points determined on the surgical instrument. After determining the boundary equation of the two-dimensional plane, the surgical instrument can be projected into the plane of the image captured by the image acquisition device according to the three-dimensional surgical instrument equation of the surgical instrument, thereby obtaining the two-dimensional surgical instrument equation of the surgical instrument in the two-dimensional plane.

[0106] Step S2243: Determine the relative position of the surgical instrument with respect to the exposure reference area based on the two-dimensional surgical instrument equation and the two-dimensional plane boundary equation.

[0107] In the above step S2243, after the surgical instrument is projected into the plane of the image captured by the image acquisition device, the relative position of the surgical instrument with respect to the exposure reference area can be determined based on the projection.

[0108] In the above implementation process, by converting the three-dimensional surgical instrument equation into a two-dimensional surgical instrument equation in the image plane captured by the image acquisition device, and judging whether the surgical instrument is within the field of view of the image acquisition device based on the two-dimensional surgical instrument equation, the accuracy of the judgment is further improved, thereby further improving the efficiency of exposing the surgical instrument.

[0109] In some optional embodiments, the three-dimensional image acquisition device equation may include:

[0110]

[0111] Three-dimensional surgical instrument equations may include:

[0112]

[0113] In the formula, x c ,y c 、z c They represent the coordinate values ​​of any point on the equation of the three-dimensional image acquisition device, x1, y1, z1 represent the three-dimensional coordinates of one of the marked points on the image acquisition device, x2, y2, z2 represent the three-dimensional coordinates of another marked point on the image acquisition device, x s ,y s 、z s They respectively represent the coordinate values ​​of any point on the three-dimensional surgical instrument equation, x3, y3, and z3 respectively represent the three-dimensional coordinates of one of the marked points on the surgical instrument, and x4, y4, and z4 respectively represent the three-dimensional coordinates of another marked point on the surgical instrument.

[0114] In the above implementation process, according to the principle of determining a straight line from two points, the equation of the straight line corresponding to the image acquisition device in three-dimensional space is determined according to the two marked points on the image acquisition device. According to the two marked points on the surgical instrument, the equation of the straight line corresponding to the surgical instrument in three-dimensional space is determined.

[0115] Based on the same concept, an embodiment of the present application provides a surgical device, which may include a surgical instrument, an image acquisition device, and a processor. The surgical instrument and the image acquisition device may be electrically connected to the processor respectively. The surgical instrument may be used to perform surgery under the control of the processor. The image acquisition device may be used to acquire images under the control of the processor. The processor may be used to: obtain the relative position of the surgical instrument relative to the exposure reference area of ​​the image acquisition device. Based on the change in the relative position, when the surgical instrument enters the exposure reference area. Update the exposure reference area according to the area where the surgical instrument is located in the exposure reference area to obtain an updated exposure reference area. And recalculate the exposure parameters based on the updated exposure reference area, and perform automatic exposure on the image acquisition device according to the calculated exposure parameters.

[0116] The above implementation process may be the same as the exposure method of the surgical instrument described above, and will not be described in detail here.

[0117] The embodiment of the present application further provides a computer program product, which includes a computer program / instruction. The computer program / instruction executes the above method when executed by a processor.

[0118] Among them, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0119] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0121] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. An automatic exposure method for an image acquisition device, characterized in that: include: Acquiring a relative position of a surgical instrument relative to an exposure reference area of ​​the image acquisition device; When it is detected that the surgical instrument enters the exposure reference area based on the change in the relative position: updating the exposure reference area according to the area where the surgical instrument is located in the exposure reference area to obtain an updated exposure reference area; as well as The exposure parameters are recalculated based on the updated exposure reference area, and automatic exposure is performed on the image acquisition device according to the calculated exposure parameters.

2. The automatic exposure method of the image acquisition device according to claim 1, characterized in that: The updating of the exposure reference area according to the area where the surgical instrument is located in the exposure reference area comprises: Determine whether there is overexposure in the area where the surgical instrument is located in the exposure reference area; If it is determined that the overexposure phenomenon exists, the area where the surgical instrument is located or the area where the surgical instrument is located where the overexposure phenomenon exists in the exposure reference area is eliminated.

3. The automatic exposure method of the image acquisition device according to claim 2, characterized in that: The determining whether there is an overexposure phenomenon in the area where the surgical instrument is located in the exposure reference area includes: Determine an overexposed area in the exposure reference area where the surgical instrument is located and whose brightness value exceeds a brightness threshold; Calculate the area of ​​the region where the surgical instrument is located and the area of ​​the overexposed region in the exposure reference region; Whether the area of ​​the overexposed area is overexposed or not is determined according to whether the ratio of the area of ​​the overexposed area to the area of ​​the area of ​​the surgical instrument in the exposure reference area exceeds a first ratio threshold.

4. The automatic exposure method of the image acquisition device according to claim 1, characterized in that: The updating of the exposure reference area according to the area where the surgical instrument is located in the exposure reference area comprises: The area where the surgical instrument is located in the exposure reference area is eliminated.

5. The automatic exposure method of the image acquisition device according to claim 4, characterized in that: Eliminating the area where the surgical instrument is located in the exposure reference area includes: Determine whether the proportion of the area where the surgical instrument is located in the exposure reference area is less than a second proportion threshold; If it is determined that the proportion of the area where the surgical instrument is located is less than the second proportion threshold, the area where the surgical instrument is located in the exposure reference area is eliminated; otherwise, no processing is performed on the exposure reference area.

6. The automatic exposure method of the image acquisition device according to claim 1, characterized in that: The obtaining of the relative position of the surgical instrument relative to the exposure reference area of ​​the image acquisition device comprises: Acquire the three-dimensional coordinates of at least two marked points on the surgical instrument and the three-dimensional coordinates of at least two marked points on the image acquisition device; Determining a three-dimensional surgical instrument equation of the surgical instrument in three-dimensional space according to the three-dimensional coordinates of at least two marked points on the surgical instrument; Determining a three-dimensional image acquisition device equation of the image acquisition device in three-dimensional space according to the three-dimensional coordinates of at least two marked points on the image acquisition device; and According to the three-dimensional surgical instrument equation and the three-dimensional image acquisition device equation, the relative position of the surgical instrument with respect to the exposure reference area is determined.

7. The automatic exposure method of the image acquisition device according to claim 6, characterized in that: Determining the relative position of the surgical instrument with respect to the exposure reference area according to the three-dimensional surgical instrument equation and the three-dimensional image acquisition device equation includes: Determine a two-dimensional plane boundary equation of the image captured by the image capture device according to the three-dimensional image capture device equation and the field of view angle parameter of the image capture device; According to the three-dimensional surgical instrument equation and the two-dimensional plane boundary equation, converting the three-dimensional surgical instrument equation into a two-dimensional surgical instrument equation within the two-dimensional plane; The relative position of the surgical instrument with respect to the exposure reference area is determined according to the two-dimensional surgical instrument equation and the two-dimensional plane boundary equation.

8. The automatic exposure method of the image acquisition device according to claim 7, characterized in that: in, The three-dimensional image acquisition device equation includes: The three-dimensional surgical instrument equation includes: In the formula, x c ,y c 、z c respectively represent the coordinate values ​​of any point on the equation of the three-dimensional image acquisition device, x1, y1, z1 respectively represent the three-dimensional coordinates of one of the marked points on the image acquisition device, x2, y2, z2 respectively represent the three-dimensional coordinates of another marked point on the image acquisition device, x s ,y s 、z s They respectively represent the coordinate value of any point on the three-dimensional surgical instrument equation, x3, y3, and z3 respectively represent the three-dimensional coordinates of one of the marked points on the surgical instrument, and x4, y4, and z4 respectively represent the three-dimensional coordinates of another marked point on the surgical instrument.

9. A surgical device, characterized in that: including surgical instruments, image acquisition devices, and processors; The surgical instrument and the image acquisition device are electrically connected to the processor respectively; The surgical instrument is used to perform surgery under the control of the processor; The image acquisition device is used to acquire images under the control of the processor; The processor is used to: obtain the relative position of the surgical instrument relative to the exposure reference area of ​​the image acquisition device; based on the change of the relative position, monitor when the surgical instrument enters the exposure reference area; update the exposure reference area according to the area where the surgical instrument is located in the exposure reference area to obtain an updated exposure reference area; and recalculate exposure parameters based on the updated exposure reference area, and perform automatic exposure on the image acquisition device according to the calculated exposure parameters.

10. A computer program product, characterized in that The method comprises a computer program / instruction which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images