Imaging system, laparoscope, and method for imaging an object.

Abstract

An imaging system (1) is provided, comprising an optical channel (2) configured to transmit light, a first sensor (10) configured to generate first image data by imaging an object (3) along a first optical path (11), and a second sensor (20) configured to generate second image data by imaging the object (3) along a second optical path (21). The first sensor (10) and the second sensor (20) have shifted focuses. Furthermore, the first optical path (11) and the second optical path (21) are guided at least in part through the optical channel (2).

Description

Technical Field

[0001] The present invention relates to an imaging system, a laparoscope, and a method for imaging an object.

Background Art

[0002] An imaging system may be used to acquire an image of an object to be imaged. The imaging system may often be provided in a laparoscope or an endoscope, which is often used as a video scope called a video scope or a video probe. Such a laparoscope is provided with an image sensor for an object of the laparoscope and is configured to acquire an image of the object during inspection. The object during inspection may be another type of object that is difficult to access, such as an internal organ in the body or the inside of a machine. Therefore, the laparoscope is a slim device with a small internal space. Therefore, it has been difficult to incorporate an imaging system that can be used to acquire a detailed image of the object to be imaged inside the laparoscope.

[0003] European Patent Application Publication No. 1691667 discloses a stereoscopic laparoscope device including a laparoscope, a computer adapted to convert and store image information of a patient's affected part from the laparoscope, and a monitor used to output the image information. The laparoscope includes a support unit including a manipulator and a pair of parallel left and right support rods, a flexible tube unit including a pair of left and right flexible tubes adapted to be separated from each other within a predetermined angle range, and a binocular camera assembly including a pair of left and right cameras installed at the tip of the flexible tube unit to photograph the affected part in the abdominal cavity by the operation of the manipulator. With such a configuration, the image information of the patient's affected part can be processed into a stereoscopic photograph, enabling precise diagnosis and laparoscopic surgery.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, stereoscopic technology is merely a technique that creates the illusion of depth in a photograph through binocular stereoscopic vision. To analyze the object being photographed in detail, it is desirable to provide actual depth information.

[0005] Therefore, an object of the embodiments of the present invention is to provide an imaging system, a laparoscope, and a method for imaging an object that can acquire images usable for determining the actual depth information of the object.

[0006] This objective is achieved by an imaging system comprising the features of claim 1, a laparoscope comprising the features of claim 12, and a method for imaging an object comprising the features of claim 13. [Means for solving the problem]

[0007] According to one aspect of the present invention, an imaging system is provided comprising an optical channel configured to transmit light, a first sensor configured to generate first image data by imaging an object along a first optical path, and a second sensor configured to generate second image data by imaging an object along a second optical path, wherein the first and second sensors are focal-shifted, and the first and second optical paths are guided, at least in part, through the optical channel.

[0008] The imaging system may be an optical imaging system, and may use a first sensor and a second sensor to create image data of an object. That is, the first image data and the second image data may capture the same part of the object from the same field of view. This is achieved by providing a dual-sensor imaging system that can quickly generate the first and second image data sequentially or simultaneously. The imaging data preferably includes a sequence of images, e.g., a video stream. Furthermore, since the first and second optical paths are guided through the same optical channel, the direction in which the object is imaged by the first and second sensors (i.e., the field of view) is the same. As a result, the first and second image data are automatically recorded so that they completely overlap each other without performing any additional alignment or recording processing. Since the first and second sensors have a shifted focus, the first image data has a different focus point (i.e., the point at which the object is clearly depicted by the image data) compared to the second image data. That is, the first sensor may focus on a point having a first distance d1 from the distal end of the optical channel, and the second sensor may focus on a further point having a second distance d2 from the distal end of the optical channel. The difference between the first distance d1 and the second distance d2 may be the focus shift of the two sensors. The focus shift may be determined by the hardware of the imaging system. More specifically, the focus shift may be provided by the arrangement of sensors in the imaging system and / or by the optical system in the imaging system. Thus, the focal length and / or optical path may be shortened or lengthened by the specific arrangement of sensors or the optical system (as described in more detail below).

[0009] The imaging system may be housed within a laparoscope used to provide information necessary for the diagnosis and treatment of human or animal internal organs. Furthermore, the laparoscope can be used for the maintenance of large machinery, for example, to determine whether hard-to-reach gears need to be replaced without disassembling the entire machine. Multiple sensors of the imaging system may be housed within the camera head of the laparoscope, so as to provide a camera head with dual sensors, each sensor having its own focus. In other words, multiple sensors may be housed exclusively within the camera head so that no sensors are present in the shaft. Thus, the shaft may be made slimmer, allowing for operation without problems even in narrow cavities, for example. Consequently, multiple sensors may be located outside the cavity being inspected during operation. In particular, this results in the fact that the weight distribution of the endoscope can be advantageously altered so that the shaft is relatively lighter in weight compared to the camera head. Since the endoscope is held by the user or a robot in or near the camera head, the operation of the endoscope may be further simplified. That is, each sensor may have its own focus point such that these sensors are focus-shifted relative to each other. For example, each sensor may have its own focal length; that is, the focal lengths of multiple sensors may differ. Furthermore, the laparoscope may have a shaft that protrudes from the camera head and can be brought near the object to be imaged. That is, the shaft may, for example, be inserted, at least part of it, into the human body or a narrow cavity, and the shaft may include an optical channel. The shaft may be connected to the camera head at its proximal end. The camera head can be detached from the shaft. In other words, multiple sensors can be separated from the shaft. Therefore, it may be possible to easily replace, change, repair, and maintain multiple sensors without disassembling the shaft itself. That is, the shaft can continue to be used, and only the camera head can be replaced. The distal end of the shaft may face the object to be imaged.Not only the shaft, but also the optical channels may be configured to be at least partially flexible. Furthermore, the shaft may be controllably movable (for example, controllably bent to change the direction of the distal end of the shaft). As a result, the laparoscope can adapt to any environment in which it is to be used and can reach areas located behind other objects. Within the shaft, an optical channel may be provided (for example, in the form of a waveguide) configured to guide the first and second optical paths from the proximal end to the distal end of the shaft, independently of the bending of the shaft. Furthermore, at the distal and / or proximal ends of the shaft, optical devices (for example, a single lens or an array of lenses) may be provided, respectively, to appropriately guide the first and second optical paths into and out of the waveguide.

[0010] The shifted focus of the first and second sensors (e.g., different focal lengths for each sensor) can be achieved by providing at least one additional lens in at least one optical path of each sensor. However, only one additional lens may be provided in at least one optical path. Furthermore, additional lenses or optical systems may be provided to appropriately guide light from the object to the multiple sensors. Using the imaging system according to one embodiment of the present invention, first and second image data can be obtained that completely overlap each other and can be processed directly without requiring additional recording processing. For example, completely overlapping images, each having a different focus (e.g., focal point), may be used to determine depth information of an object (further details will be described later).

[0011] The optical channel may have an elongated body. Furthermore, the optical channel may be hollow or made of a transparent material. The optical channel may have a proximal end facing the first and second sensors and a distal end facing the object to be imaged. The distal end may protrude from the imaging system and may be the exit port for the first and second optical paths. Thus, the first and second optical paths are aligned at their distal ends so that the field of view of the first sensor and the field of view of the second sensor are the same. The optical channel may be configured such that the optical path exiting the optical channel at the distal end diverges and extends, preferably with an extension angle of 30° to 90°, preferably 50° to 70° (i.e., the optical path may become conical as it exits the optical channel to capture a larger scene). For example, the optical channel may be made of optical fiber. Furthermore, the optical channel may be at least partially flexible so that it can guide light even if the optical channel is curved. Preferably, the optical channel is flexible according to the shaft of the laparoscope in which it is provided. Therefore, objects located behind obstacles can be easily imaged by selecting a path for the shaft around the obstacle. The imaging system may have only one optical channel. Therefore, the imaging system can be made compact, requiring less space for operation. On the other hand, stereoscopic imaging systems may have two or more optical channels due to their inherent system requirements, and therefore may require more space for operation.

[0012] The first and second sensors may be photosensors, also called image sensors or sensor chips, and are preferably called complementary metal-oxide-semiconductor (CMOS) sensors or complementary symmetric metal-oxide-semiconductor (COS-MOS) sensors. Furthermore, the first and second sensors may be charge-coupled device (CCD) sensors. Also, the first and second sensors may be of the same type. Alternatively, the first sensor may be a different sensor from the second sensor. By providing different types of sensors, different image quality and / or different image information can be obtained. For example, low-resolution image data may be used to provide rough information about an object, while high-resolution images may be used for detailed information in further processing. Preferably, the minimum resolution of at least one sensor is 1080 × 1920 pixels. Preferably, each sensor has the same resolution. Hereinafter, these will be referred to as the first sensor and the second sensor, but the imaging system may have two or more sensors, and each sensor has its own optical path. That is, if three sensors are provided, three optical paths are also provided, and so on. For example, an imaging system can have three, four, five, or more sensors. In this case, each optical path of each sensor is guided, at least partially, through the same optical channel. The use of more sensors is particularly useful when the object being imaged has a large spatial extent or when acquiring very detailed information about the object being imaged. Furthermore, each sensor may have its own shutter configured to control the amount of light illuminating the sensor. As a result, the sensor can be adapted to different lighting conditions by adjusting the shutter speed. Alternatively, the sensor can acquire a continuous video stream of image data.

[0013] When a CMOS sensor is used, the image data will be a voltage signal output by the sensor. Preferably, each pixel of the sensor can have a specific voltage signal. That is, the CMOS sensor can output a digital signal. When a CCD sensor is used, the image data may be a charge signal. Since voltage signals are less susceptible to degradation by electromagnetic fields, it is preferable to use a CMOS sensor as the sensor in the imaging system.

[0014] Image data may be the output of multiple sensors. Furthermore, image data may include luminance information for one or more color channels. A color channel may represent a specific light spectrum. For example, image data may include information for the green color channel, RGB color channels, and / or NIR (near-infrared) color channels. The RGB color channel is considered to include the green color channel, red color channel, and blue color channel. Furthermore, each sensor can provide image data containing information for a different color channel. That is, a first sensor can provide first image data containing information for the green color channel, while a second sensor can provide second image data containing information for the NIR channel. Further combinations of different color channels from each sensor are also possible, such as NIR-NIR, RGB-RGB, green-green, NIR-RGB, green-NIR, green-RGB, etc. Here, each color channel may be defined by a different wavelength band. For further processing of the image data, the imaging system may be connected to or connectable to a processing unit (e.g., a computer), or it may have a control unit. The control unit may be configured to further process the image data. For example, the control unit may determine information contained within the image data, such as a histogram showing the luminance distribution. Furthermore, the image data may consist of images or photographs depicting the object. To cope with poor lighting conditions, the imaging system may include a light source configured to illuminate the object being imaged. More specifically, the light source may be coupled with an additional waveguide configured to direct the light from the light source into or parallel to the optical channel to the object being imaged.

[0015] An optical path is the path through which light rays travel from an object to multiple sensors. That is, light is reflected by the object to be imaged, introduced into the optical path, guided through the optical path, output through the optical path, and can be imaged by the first and second sensors. The optical path may be defined by the light rays received by each sensor. In other words, the length of the optical path can be measured by the distance from the sensor to the object to be imaged. Furthermore, each optical path may contain multiple lenses or arrays of multiple lenses configured to appropriately guide light rays from the object to each sensor. Additionally, the imaging system may include at least one prism configured to split the first and / or second optical path. Preferably, the prism is configured as a beam splitter prism. The prism may be configured to filter specific wavelengths, i.e., to transmit only specific wavelengths of light. Thus, the wavelengths transmitted to the sensors may be predetermined. For example, only wavelengths corresponding to the green color channel may be transmitted by the prism. Furthermore, the prism can also increase the length of one optical path relative to the other. Therefore, because the prism has multiple tasks, the imaging system can be realized with a minimum number of components. Furthermore, the imaging system may include at least one aperture in the optical path or optical channel configured to control the aperture of the imaging system. That is, the wider the aperture opens, the shallower the depth of field of the image, and vice versa. In general, when the aperture is wide open, the image data has better quality and contains more information. Therefore, it is preferable to adjust the aperture to its maximum opening. As a result, the depth field may be relatively narrow. Furthermore, the imaging system may have one aperture for each optical path.

[0016] The shifted focus between the first and second sensors may be provided by giving each sensor a different focal length. The focal lengths of the first and second sensors may be the distance from the principal axis of an optical lens provided in the imaging system to the focus point along their respective optical paths. The optical lens may be configured to project an image onto the sensor. Each sensor may be assigned to a separate lens; that is, if there are two sensors, the imaging system may have two optical lenses. Furthermore, the focal length may be an indicator of the strength with which the imaging system converges or diverges light. A positive focal length indicates convergence of light, and a negative focal length indicates divergence. Systems with shorter focal lengths bend light rays more sharply, focus at shorter distances, and diverge more quickly. For example, the ratio of the focal lengths of the first and second sensors can be in the range of 0.1 to 1, preferably 0.3 to 0.7. Within this range, the best imaging of the object can be archived. In other words, having the above ratio, the distance between the focus point of the first sensor and the focus point of the second sensor is within an optimal range. More specifically, when the first and second image data are combined or compared with each other, the above ratio ensures that no information about the object is lost due to it being completely out of focus. In other words, it is a ratio that prevents the distance between the in-focus parts from becoming excessively large.

[0017] According to the present invention, at least two image data that are fully recorded relative to each other may be received or generated without additional recording processing. That is, image data of an object generated by at least two sensors may have the same size, the same position, and be imaged from the same viewpoint. Therefore, some parts of the object may be in focus on the first image data, and some other parts of the object may be in focus on the second image data. Furthermore, some parts of the object may be partially in focus on the first image data and partially in focus on the second image data. As a result, by combining or comparing both image data, it is possible to generate a whole image or whole data that contains more information about the object (e.g., more detail, depth information, etc.) compared to a single image or a stereoscopic image of the object. The combination or comparison of the image data is easily performed because both image data are recorded relative to each other so that they fully overlap. That is, it is not necessary to record the image data relative to each other before the image data is further processed. In other words, the first and second image data may contain exactly the same part (i.e., scene) of the object. As a result, the present invention provides a highly efficient method for generating image data that fully overlaps and is therefore easily processed. Preferably, the focus shifts of the multiple sensors are determined in advance by the imaging system (i.e., before imaging the object). That is, the focus shifts can be set by the hardware of the imaging system.

[0018] Preferably, the first and second optical paths have different lengths. Therefore, the focus point of the first sensor is positioned differently from that of the second sensor. Preferably, the distance between at least one sensor and the proximal end of the optical path, and / or the distances between multiple sensors, may be adjustable. As a result, the focal lengths of each sensor may differ, i.e., the distance from each sensor to the point of an object sharply depicted by its respective image data may differ. Providing different lengths for the optical paths is a simple and stable way to achieve different focal lengths for the first and second sensors (i.e., focus shifts between the first and second sensors). For example, multiple sensors may be at different distances from the proximal end of the optical channel. That is, the first sensor may be positioned in the imaging system so that it is further from the proximal end of the optical channel than the second sensor. As a result, different focus points (i.e., focus shifts between the first and second sensors) are ensured by the different positions of the multiple sensors in the imaging system, while the same sensors and image settings (e.g., aperture, shutter speed, etc.) can be used. As a result, the same components can be used to acquire both the first and second image data. Therefore, the imaging system can be simplified and manufacturing costs can be reduced.

[0019] Preferably, the first and second sensors are further configured to image the object simultaneously. That is, the first and second image data can be generated or acquired simultaneously. As a result, since there is no possibility of the object moving or changing shape between the time the first image data is acquired and the time the second image data is acquired, both the first and second image data are completely recorded relative to each other. That is, the first and second image data may differ from each other only in their individual focus points (i.e., focus / blur). Therefore, the imaging system can include only one shutter used for both sensors provided within the imaging system. Thus, the system can be further simplified while ensuring complete overlap of the first and second image data.

[0020] Preferably, the system further includes a focus system positioned in the first and / or second optical paths and configured to change the focus of the first and / or second sensors. Thus, the focus (e.g., focus point or focal length) of at least one sensor can be adjusted. In other words, the distance between the focus point of the first sensor and / or the focus point of the second sensor may be changed. As a result, the imaging system can be adapted to different objects having different spatial extensions. Furthermore, the imaging system can be adapted to different applications. For example, when the image processing system is used during laparoscopy (i.e., in combination with a laparoscope), the first sensor can be focused at a distance of 6 cm measured from the distal end of the shaft, and the second sensor can be focused at a distance of 9 cm measured from the distal end of the shaft. Thus, the sensor focus shift is 3 cm. The focus shift may be positive or negative depending on the specific application. For this reason, the imaging system may be used in multiple applications, and multiple objects may be imaged using the imaging system. For example, the focus system may be configured to adjust the focus point of at least one of the multiple sensors so that the above-defined ratio between the focus point of the first sensor and the focus point of the second sensor is obtained.

[0021] Preferably, the system further comprises focusing means configured to control the focusing system so that the focus (e.g., focal length) of the first and / or second sensors can be adjusted. The focusing means may be a focusing ring configured to be held in the user's hand. Thus, the focusing means may be exposed outside the imaging system (and outside the laparoscope) so that it can be held by the user. The position (i.e., rotation) of the focusing ring may depend on the distance of the object being imaged from the multiple sensors. The greater the distance to the object, the more the focusing ring needs to be rotated to sharply depict at least a portion of the object on the multiple sensors, and vice versa. The focusing means can be made so that only a few fingers (e.g., two fingers) of the user can touch it while the other fingers of the user's hand hold the image processing system (i.e., the laparoscope). As a result, it is preferable that the imaging system is operable with one hand; that is, the user does not need to operate the imaging system with both hands for focusing or holding the imaging system. As a result, the imaging system achieves improved operability. Furthermore, the focusing means may be operated automatically by a control unit. More specifically, the focusing means may be controlled by a control unit to adjust the focus of each sensor according to a specific application before acquiring image data. Alternatively or additionally, the focusing means may be controlled by a control unit to adjust the focal length of each sensor in predetermined steps (i.e., increasing the value). In each step, image data may be acquired by the first and second sensors. The step size may be in the range of 0.1 mm to 5 mm, preferably in the range of 0.5 mm to 2 mm. The focusing means may be provided on the camera head of the endoscope. Therefore, the shaft may not have a focusing means. This allows the weight of the shaft to be kept low. As a result, the overall operability of the endoscope can be improved. Furthermore, during operation of the endoscope, the focusing means can be positioned outside the cavity being examined (e.g., the human body).Therefore, the focusing mechanism can be made easily accessible to the user or robot.

[0022] Preferably, the first and second image data represent the same scene, e.g., the exact same part of an object. Depending on the sensor's focus (e.g., focal length) and / or the different types of sensors used, the field of view of the first image data may be smaller than that of the second image data, or vice versa. To provide the same field of view of an object in both the first and second image data, the image data covering a larger field of view may be adjusted to precisely include the same field of view of the object (e.g., by cropping a portion of the image data, e.g., the outer edge). As a result, both image data may be easily comparable or combined. For example, in both image data, the edges of the image data may be used as reference points. In other words, the portion of the object represented by the first image data and the second image data will precisely overlap. Thus, further processing of the image data can be further improved and simplified.

[0023] Preferably, the imaging system further includes a control unit configured to generate depth information of an object based on first and second image data. In other words, the imaging system may include a control unit for further processing the image data. The control unit may be a computer-like device and may be configured to receive input data, process the input data, and output the processed data. In particular, the input data may be image data (e.g., multiple image data) containing the 2D coordinates of the object, and the output data may be a third coordinate of the object (i.e., depth information or 3D shape information). Note that generating depth information is just one example of how multiple image data may be used. That is, the control unit may output a depth map of the object (e.g., 3D image data). Specifically, the depth map may be a scatter plot of points, each having three coordinates that describe the spatial position or coordinates of each point of the object. That is, the control unit may be configured to determine the depth map based on the first and second image data.

[0024] Furthermore, the control unit may be configured to divide the image data into one or more (e.g., >9) pixel segments called patches. The control unit may also be configured to compare the patches of the first image data with the patches of the second image data. The patches may be rectangular or quadratic. In particular, the control unit may be configured to determine how sharply the object is depicted in each patch (i.e., to determine the sharpness of each patch). Patches can have multiple sizes. That is, the size of the patches provided in each image data may vary depending on the object depicted by the image data and / or depending on a specific application (e.g., type of surgery). That is, in areas with a lot of texture, the patch size may be small (e.g., 5x5 pixels), while in areas where the pixel intensity of the image data is more uniform, i.e., areas with less texture, the pixel size may be large (e.g., 50x50 or up to 1000 pixels). For example, areas of image data with many edges may have a lot of texture. Therefore, in areas with a lot of texture, the patch size is reduced (i.e., the resolution is increased) to obtain high-precision information in that area, while in areas with little texture, the patch size is increased to accelerate processing performed by the control unit, thus enabling the imaging system to operate with high efficiency.

[0025] Preferably, the position of at least one or each first patch in the first image data corresponds to the position of at least one or each second patch in the first image data. Preferably, at least one first patch has the same size as at least one second patch, preferably 20 × 20 pixels in size. That is, the first patch has the same position in the first image data as the second patch in the second image data. More specifically, when the first image data and the second image data are recorded together, the first and second patches overlap each other. Therefore, for each patch, a depth value (i.e., Z coordinate) may be determined by the control unit (i.e., the x, y, and z coordinates in 3D space are determined for each patch). Patches can have the size of one pixel or several pixels. However, the required computational resources depend on the number of patches included in the image data. Therefore, preferably, each patch can have the size of 20 × 20 pixels. Such a size ensures system efficiency and accuracy of the depth map.

[0026] Furthermore, the control unit may be configured to determine the sharpness of each patch in the first image data and each patch in the second image data. Note that multiple patches may have the same sharpness. The sharpness of each patch is proportional to the entropy of that patch. In other words, sharpness is synonymous with entropy. Patches in the first image data can be designated as the first patch, and patches in the second image data as the second patch. The first and second patches may completely overlap each other (i.e., depict part of the same scene / object). In other words, the first and second patches (also called a pair of patches) may be located at the same location in the first and second image data. As described above, the focal lengths of the first and second sensors may be preset. Therefore, the focal lengths of the first and second sensors are known. The control unit may determine the depth (i.e., z-coordinate) of the part of the object depicted by the first and second patches using the following equation (1).

[0027]

Number

[0028] To obtain useful depth information using the above formula (1), it is necessary to carefully select the focal lengths of the first sensor and the second sensor, as well as the patch size. Preferably, the focal length depends on the specific application being performed (e.g., the specific surgery or laparoscopy being performed). For example, the focal length of the first sensor may be 6 cm measured from the distal end of the shaft, and the focal length of the second sensor may be 9 cm measured from the distal end of the shaft. The focal length of the first sensor may be an empirical value determined from the type of surgery (e.g., in the case of laparoscopy, it can be set to 6 cm measured from the tip of the laparoscope (i.e., the distal end of the shaft)).

[0029] The sharpness of an image may be represented by the information density (e.g., entropy) of the image of a specific patch. The control unit may be configured to apply the above formula 1 to each pair of patches of the first image data and the second image data (i.e., each first patch and the corresponding second patch). As a result, the depth value of each pair of patches of the image data can be determined by the control unit. The control unit having depth information may be configured to create a depth map using the depth information and the x and y coordinates of each pair of patches. In other words, the control unit may be configured to create a 3D model based on the x, y, and z coordinates of each pair of patches.

[0030] Alternatively or additionally, the control unit may be configured to create a 3D model using a focus stacking method. A further method that may be used to create a depth map is so-called focus / defocus depth. Furthermore, the control unit may be configured to perform one or more of the above methods to create depth information (i.e., 3D image data) based on at least two 2D image data (i.e., first image data and second image data). Specifically, the control unit may be configured to combine at least two methods for creating a depth map. In particular, Equation 1 above may be applied to patches of image data that have a relatively high texture compared to other patches of the image data. Alternatively, the shape-from-lightning method may be applied to other areas of image data that have a relatively poor texture compared to other areas of the image data. As a result, by applying different methods to different parts of the image data, the performance of depth map generation can be significantly improved. For this reason, at least one embodiment of the present invention can convert the relative difference in focus of image data obtained by two sensors into 3D shape information. In other words, the 3D coordinates of all patches in the image data can be calculated / determined by comparing the focus / blur of each patch (i.e., a single pixel or array of pixels) generated by each sensor. Optionally, the control unit may be configured to further process the depth map by filtering the resulting depth map. More specifically, to remove erroneous depth information, the control unit may be configured to compare the depth information of adjacent depth maps. If a depth is excessively high or low compared to adjacent depths (i.e., exceeding a previous threshold), this depth information is likely to be erroneous, and the control unit may be configured to remove such depth information. As a result, the depth map output by the control unit may be more accurate.

[0031] Preferably, the control unit is further configured to generate depth information by comparing the entropy of at least one first patch of the first image data with the entropy of at least one second patch of the second image data. Image sharpness is synonymous with entropy. Entropy is an example of information contained within image data (i.e., an indicator of the sharpness of a patch in the image data).

[0032] In other words, the first image data and the second image data can be compared based on their entropy. Therefore, the control unit may be configured to determine the entropy of the first image data and the entropy of the second image data. Specifically, a higher entropy means that each image data is in focus. Conversely, a lower entropy means that each image data is out of focus (i.e., blurred). The entropy can be calculated using the following equation (2).

[0033]

number

[0034] The following outlines three optional features for improving depth estimation accuracy.

[0035] Preferably, the control unit may be configured to monitor the adjustment of the focus of at least one sensor. For example, the control unit can detect the operation of the focusing means. The focus may be adjusted by a human user and / or by a robot operating the imaging system. That is, when the focus of at least one sensor is adjusted, the control unit may monitor the amount of adjustment. In other words, the control unit may detect the shift (i.e., distance) of the focus of each sensor. The control unit may also detect which parts of the image data are sharply rendered based on the image data (e.g., using edge detection). Furthermore, the control unit may know how far the sharply rendered parts of an object are from, for example, the tip of the endoscope. By combining the location of the focus in the image data with the knowledge of which parts of the image data are sharply rendered, the control unit can improve depth estimation. That is, the control unit can obtain information on which parts of the image data are sharply rendered at the present time and where the focus of each sensor is located. As a result, the control unit may determine the depth of the sharply rendered parts of the image data. Monitoring may also be performed in real time. Therefore, continuous improvement of depth estimation is possible. Monitoring of focus adjustment can be achieved by providing optical encoders, electrical measurements, and / or other mechanical sensors. Focus adjustment can be efficiently measured by using at least one of the above means. Furthermore, measurement accuracy can be improved by using these measurement means in combination.

[0036] Preferably, the control unit may be configured to monitor the operation of the imaging system. That is, the spatial position of the imaging system may be monitored by the control unit. For example, if the imaging system is installed in an endoscope, the control unit may detect the spatial position of the endoscope. Preferably, the position is detected in real time. This information may then be used to improve the accuracy of depth estimation. The spatial position can be detected by monitoring optical trackers, electromagnetic trackers, and / or actuators. Actuators may be used to operate the imaging system by a robot. That is, if the robot moves the imaging system when operating it, the control unit can detect such movement and, as a result, determine the current spatial position of the imaging system. Furthermore, the imaging system may include acceleration sensors and / or rotation sensors. Thus, the movement of the imaging system can be accurately monitored. Based on the known spatial position of the imaging system and information on which parts of the image data are clearly depicted (see summary above), the accuracy of depth estimation can be further improved.

[0037] Preferably, the control unit may be configured to estimate depth information using the size of a known object in the image data. That is, the control unit may be configured to determine the distance to a known object (e.g., from the tip of the endoscope) by knowing the dimensions of the known object. The object may be a marking present in the cavity under examination. Furthermore, the marking may be provided on an instrument used additionally during the examination (e.g., during surgery). Furthermore, the object may be the instrument itself or another object present in the cavity under examination. Therefore, optical calibration may be performed to improve the accuracy of depth estimation.

[0038] Improving depth estimation accuracy can be useful in reducing erroneous depth estimation results. erroneous results can occur because depth estimation may be based on the contrast of parts of the first and second image data (see summary above). The three features defined earlier can be used, for example, to check whether the depth estimation is correct. The latter feature may be advantageous because it eliminates the need for additional structural means to perform optical calibration. In other words, optical calibration can be performed by the control unit without requiring additional sensors, etc. However, two or all of the above features may be implemented to improve depth estimation accuracy. Furthermore, the information obtained as described above can also be used to compensate for distortions in the imaging system (e.g., the endoscope that constitutes the imaging system). At least one of the above features can be implemented using computer vision techniques. Computer vision is considered an interdisciplinary field of science that deals with how computers can gain a high level of understanding from digital images or videos. From an engineering perspective, it can be sought to understand and automate tasks that the human visual system can perform. The challenges of computer vision include acquiring, processing, analyzing, and understanding digital images, as well as methods for extracting high-dimensional data from the real world (e.g., forms of decision-making) to generate numerical and symbolic information. Understanding in this context sometimes means transforming visual images (image data) into a description of the world that can give meaning to thought processes and elicit appropriate actions. This image understanding can be seen as decoupling symbolic information from image data using models built with the help of geometry, physics, statistics, and learning theory.

[0039] Preferably, the optical channel is rotatable so that the fields of view of the first and second sensors are variable. In particular, only the distal end of the optical channel may be configured to rotate. That is, only a portion of the optical channel can be rotated. The sensors may be fixed relative to the rotation of the optical channel. Alternatively, an optical device (e.g., a single lens or an array of lenses) provided at the distal end of the optical channel may be configured to change the field of view by rotating. Thus, the proximal end of the optical channel may be fixedly held within the imaging system, and the distal end may be rotated to face the object being imaged. The rotatable portion of the optical channel may protrude from the imaging system. The optical channel may have an initial position in which it extends straight without curvature. Furthermore, the optical channel may rotate about its axis in the initial position. Specifically, the optical channel may be configured to be inclined at about 20° to 40°, preferably about 30°, with respect to the axis of the optical path in the initial position. Furthermore, the optical path may have a conical shape after it exits the optical channel. The aforementioned cone can be tilted by rotating at least a portion of the optical channel around its central axis. Specifically, by rotating, the optical path can be deviated by 30° relative to the central axis of the optical channel. Furthermore, at least the distal end of the optical channel may be movable relative to multiple sensors. As a result, the imaging system can be used for multiple purposes, even in narrow spaces or cavities. That is, only a portion of the optical channel may be directed towards the object to be imaged in order to acquire first and second image data. Therefore, even objects that are difficult to access can be imaged by the imaging system. In addition, by rotating the field of view around the axis of the optical channel and tilting the field of view relative to the axis of the optical channel, it is possible to view the entire periphery of the cavity being examined. For example, the endoscope that constitutes the imaging system does not need to be moved within the cavity to grasp the overall image of the cavity.

[0040] Preferably, the first and second sensors are arranged at an angle to each other; that is, at least two of the sensors are at an angle to each other. For example, multiple sensors are at an angle to each other if their highly sensitive surfaces are at an angle to each other. The inclination provides the effect that the imaging system can be more compact compared to a configuration in which the sensors are on the same plane to each other. Preferably, the multiple sensors are housed in a camera head separate from the shaft element configured to be inserted into the cavity under inspection. Thus, the camera head can have compact dimensions by arranging the multiple sensors at an angle. That is, the extension of the camera head along the direction of the shaft may be limited in order to ensure a compact camera head. In particular, the multiple sensors may be at a substantially 90° angle to each other. Substantially 90° may mean an arrangement of multiple sensors that forms an angle of 90° plus or minus 10° between the multiple sensors. Thus, the imaging system can be efficiently manufactured considering an acceptable level of tolerance.

[0041] Preferably, the first and second sensors are configured to use RGB light spectra, green light spectra, and / or NIR light spectra to generate first and second image data. Therefore, depending on the specific application, light spectra containing a lot of information under certain conditions can be used. Specifically, the green light spectrum contains a lot of information under relatively poor lighting conditions, while the NIR light spectrum provides a lot of information under nearly dark lighting conditions. As a result, shape and information can be easily recognized or determined from image data under different lighting conditions. Furthermore, the first and second sensors can use different light spectra / channels. For example, the first sensor can use the green spectrum, and the second sensor can use the NIR spectrum. As a result, depth information can be determined even when lighting conditions change.

[0042] A further aspect of the present invention provides a laparoscope equipped with the above-described imaging system. That is, the laparoscope may be an endoscope including a camera head constituting the above-described imaging system. The camera head housing the imaging system may be connected to the laparoscope via a camera adapter. A focus ring may be positioned on the camera adapter of the laparoscope to bridge the camera head and the telescope of the laparoscope. The telescope may be the shaft of the laparoscope. An optical channel may be located within the shaft. Furthermore, the shaft may be flexible so that the optical channel can rotate at the distal end of the laparoscope (e.g., the telescope), or it may protrude from the distal end of the laparoscope. Furthermore, the laparoscope may include a light source. The light source may be adjustable to illuminate the area to which the distal end of the optical channel is directed. Thus, first and second image data can be adequately generated even in poor lighting conditions. The light source may preferably be an LED light source. As a result, a laparoscope may be provided that can acquire at least two focus-shifted image data using one camera head. As a result, laparoscopes can have a highly integrated design and be used for applications in a wide range of fields. Laparoscopes may also be configured to be operated by a robot. That is, laparoscopes may have an interface configured to receive operating commands from a robot and transmit information (e.g., image data) to the robot. Thus, laparoscopes can be used in at least partially automated surgical environments.

[0043] A further aspect of the present invention provides a method for imaging an object, which includes generating first image data of the object by imaging the object along a first optical path using a first sensor, and generating second image data of the object by imaging the object along a second optical path using a second sensor, wherein the first and second sensors are focus-shifted, and the first and second optical paths are guided at least in part through the same optical channel.

[0044] Preferably, this method further includes dividing the first image data and the second image data into patches, where the patches of the first image data and the patches of the second image data correspond to each other. In other words, the patches of the first image data cover exactly the same area as the corresponding patches of the second image data. The corresponding patches of the first image data and the corresponding patches of the second image data can be considered as a pair of patches.

[0045] Preferably, the method further includes comparing the first image data and the second image data with each other to generate depth information.

[0046] Preferably, multiple images are compared with each other by comparing at least one pair of patches.

[0047] Preferably, the first image data and the second image data are generated simultaneously.

[0048] The advantages and features described in relation to the apparatus are also applicable to the methods, and vice versa. Unless expressly described, individual embodiments or their individual aspects and features may be combined or interchanged with one another whenever such combination or substitution is meaningful and within the meaning of the present invention, without limiting or expanding the scope of the present invention as described. Advantages described in relation to one aspect of the present invention are also advantages of other aspects of the present invention, to the extent applicable. [Brief explanation of the drawing]

[0049] [Figure 1] Figure 1 is a schematic diagram of an imaging system according to an embodiment of the present invention. [Figure 2] Figure 2 is a schematic diagram of an imaging system according to an embodiment of the present invention. [Figure 3] Figure 3 is a schematic diagram showing a part of the imaging system according to an embodiment of the present invention. [Figure 4] Figure 4 is a perspective view of a laparoscope according to an embodiment of the present invention. [Modes for carrying out the invention]

[0050] Next, useful embodiments of the present invention will be described with reference to the accompanying drawings. Similar elements or features are indicated by the same reference numerals in the drawings.

[0051] Figure 1 is a schematic diagram showing an imaging system 1 according to an embodiment of the present invention. Figure 1 depicts a coordinate system having x, y, and z axes. The imaging system 1 is configured to image an object 3. The imaging system 1 includes a first sensor 10 and a second sensor 20 configured to generate first image data and second image data. Furthermore, the first sensor 10 and the second sensor 20 are focus-shifted. In this embodiment, both sensors 10 and 20 are CMOS sensors. The image data includes spatial information in the xy plane (i.e., two-dimensional information). Therefore, the imaging system 1 includes a first optical path 11 extending between the first sensor 10 and the object 3, and a second optical path 21 extending between the second sensor 20 and the object 3. Furthermore, the imaging system 1 includes an optical channel 2 having a distal end 5 facing the object 3 and a proximal end 6 facing the first sensor 10 and the second sensor 20. The optical channel 2 is an elongated transparent body extending along its central axis C, which in this embodiment extends along the z-axis. The first and second optical paths 11 and 21 extend from the first and second sensors 10 and 20, respectively, to a beam splitter prism 7 provided within the imaging system 1. At the beam splitter prism 7, the first optical path 11 and the second optical path 21 are merged / split by the beam splitter prism 7 and further extend to the proximal end 6 of the optical channel 2. That is, the beam splitter prism 7 is provided in the optical paths 11 and 21 between the sensors 10 and 20 and the optical channel 2. The first and second optical paths are then guided by optical lenses (not shown). In another embodiment, each optical path has an optical lens. Subsequently, both the first and second optical paths 11 and 21 are guided through the optical channel 2. From the distal end 5 of the optical channel, both optical paths are guided in the same way toward the object 3. As a result, the first sensor 10 and the second sensor 20 can image the same portion of an object having the same viewpoint (i.e., the same field of view). Furthermore, the imaging system 1 includes a focus system 4 configured to adjust the focus (e.g., focal length) of the first sensor 10 and the second sensor 20. In this embodiment, the focus system is a lens arrangement and is provided within the optical channel 2.In other words, the focus system 4 is arranged to adjust the focus of both sensors 10 and 20 simultaneously. That is, the optical channel may be made between the two parts where the focus system 4 is provided. In further embodiments not shown, a focus system is provided for each sensor so that the focus of each sensor can be adjusted individually. In this embodiment, the focus of the two sensors 10 and 20 is set prior to the acquisition of the first and second image data.

[0052] Furthermore, the first sensor 10 and the second sensor 20 are arranged in the imaging system 1 so as to be focus-shifted. That is, the first sensor 10 has a different focus (i.e., a different focus distance) compared to the second sensor 20. In other words, the focus point of the first sensor 10 is located at a different position compared to the focus point of the second sensor 20. In this embodiment, the shifted focus is achieved by providing an additional lens 8 in the second optical path 21. That is, an additional lens 8 is provided in addition to the optical system (not shown) that may be used to guide the first and second optical paths. As a result, the same part of the object 3 cannot be imaged sharply by both the first sensor 10 and the second sensor 20. In the embodiment depicted in Figure 1, the first part 31 of the object 3 is imaged sharply by the first sensor 10, but the first part 31 imaged by the second sensor 20 is blurred (i.e., not depicted sharply).

[0053] Figure 2 is a schematic diagram showing an imaging system 1 according to an embodiment of the present invention. The imaging system 1 depicted in Figure 2 is the same as the imaging system 1 depicted in Figure 1. The difference from Figure 1 is that a second portion 32 of the object 3, located at different positions along the z-axis, is depicted. Furthermore, the second portion 32 is imaged in a sharp manner by the second sensor 20, while the second portion 32 is imaged in a blurred manner by the first sensor 10. Note that the first portion 31 and the second portion 32 depicted in the figure are parts of the object 3 that are omitted from the drawing for the sake of simplicity of explanation. That is, a part of the object 3 being imaged is sharpened by the first sensor 10, and another part of the object 3 being imaged is sharpened by the second sensor 20. As a result, a dual-sensor optical imaging system is provided in which the image axes of both sensors (first optical path 11 and second optical path 21) are at least partially identical, while the sensors have a focus shift. Therefore, it is possible to acquire fully recorded image data. However, according to further embodiments not depicted in the drawings, the imaging system includes two or more sensors, each configured and arranged in a manner similar to the two sensors of the above embodiments.

[0054] Furthermore, the imaging system 1 includes a control unit configured to divide the first image data and the second image data into a plurality of patches. In this embodiment, each patch has a size of 20 × 20 pixels. Furthermore, the control unit determines the image information (i.e., entropy) of each patch. That is, the first image data is divided into the same patches as the second image data. As a result, pairs of patches may be determined. A pair of patches may consist of a first patch of the first image data and a corresponding second patch of the second image data. Next, the depth (i.e., z-coordinate) of each pair of patches is determined using the entropy of each patch and the known focal lengths of the first sensor 10 and the second sensor 20. This is done for each pair of patches.

[0055] In some embodiments, the focus system 4 is operated to shift the focus points of sensors 10 and 20 by a predetermined increment. In some embodiments, the focus system 4 is automatically controlled and operated by a control unit. In another embodiment, the focus system 4 may be manually operated by a user via a focusing means (e.g., a focus ring) 104 (see Figure 3). At each increment, image data is generated by the first sensor 10 and the second sensor 20. Image data may be generated simultaneously or sequentially by the first sensor 10 and the second sensor 20. In this embodiment, the control unit determines the depth of each pair of patches using the following equation (1).

[0056]

number

[0057] For the algorithm to function, careful selection of the focal lengths of the first sensor 10 and the second sensor 20, as well as the patch size, is necessary. Therefore, it should be noted that these elements should be carefully selected according to the specific configuration of the object being imaged. The focal lengths of each sensor 10 and 20 are known. Thus, the depth information for each patch can be determined using equation (1) above.

[0058] Figure 3 depicts a part of the imaging system 1 according to an embodiment of the present invention. Specifically, Figure 3 shows two sets of schematic diagrams. In each diagram, the optical channel 2 is schematically depicted. Furthermore, the first and second optical paths 11 and 21 are schematically depicted (i.e., the first optical path 11 and the second optical path 21 overlap). The optical paths extend from the distal end 6 of the optical channel, which has a conical shape (i.e., they diverge). In this embodiment, the conical shape has an acute angle α of 60°. Therefore, the optical channel 2 of this embodiment includes a lens at its distal end that is configured to diverge the first optical path 11 and the second optical path 21 so that they have a divergence angle of 60°.

[0059] In the first row of Figure 3, the optical channel 2 is depicted in its initial position. In the second row of Figure 3, the optical channel is rotated at least partially so that the optical paths 11 and 21 face different directions. In particular, according to this embodiment, the optical channel 2 is at least partially rotatable about the z-axis. Thus, the first and second optical paths 11 and 21 can be rotated about the z-axis. In this embodiment, the optical paths are inclined such that the central axis D of the optical path is at an angle of 30° with respect to the z-axis (see the second row of Figure 3). The left side of the second row shows the optical paths 11 and 21 facing downwards towards the optical channel 2, while the right side of the second row shows the optical paths 11 and 21 facing upwards towards the optical channel 2. That is, the optical paths 11 and 21 can be rotated around the entire circumference of the z-axis.

[0060] Figure 4 shows a laparoscope 100 according to an embodiment of the present invention. The laparoscope 100 includes an imaging system 1 within a housing 102. The laparoscope has a shaft 101 with a distal end 103. An optical channel 2 is provided within the shaft 101 of the laparoscope 100. Furthermore, the laparoscope 100 includes a light source (not shown) configured to emit light from the distal end 103 of the laparoscope toward an object 3 along a first optical path 11 and a second optical path 12.

[0061] The above discussion is intended merely to illustrate the System and should not be interpreted as limiting the appended claims to any particular embodiment or set of embodiments. Therefore, while the System has been described in particular detail with reference to exemplary embodiments, it should be understood that numerous modifications and alternative embodiments may be conceived by those skilled in the art without departing from the broader, intended spirit and scope of the System as defined in the subsequent claims. Accordingly, the specification and drawings should be considered illustrative and not intended to limit the scope of the appended claims. [Explanation of Symbols]

[0062] 1. Imaging System 2 optical channels 3. Object 4 Focus System 5. Distal end of the optical channel 6. Proximal end of the optical channel 7 Beam Splitter Prism 8 lenses 10. First Sensor 11 1st optical path 20. Second Sensor 21 Second optical path 31 The first part of the object 32 The second part of the object 100 Laparoscopy 101 Shaft 102 Housing 103 Distal end of laparoscope 104 Focusing means α Angle of the optical path C optical channel axis D Axis of the optical path xx axis yy axis zz axis

Claims

1. An optical channel (2) configured to transmit light, A first sensor (10) is configured to generate first image data by imaging an object (3) along a first optical path (11), The system includes a second sensor (20) configured to generate second image data by imaging the object (3) along a second optical path (21), The first sensor (10) and the second sensor (20) have their focal points shifted. The first optical path (11) and the second optical path (21) are guided, at least in part, through the optical channel (2). The first sensor (10) and the second sensor (20) are arranged so as to be inclined relative to each other. The system further includes a control unit configured to generate depth information of the object based on the first image data and the second image data, The control unit is further configured to generate depth information by comparing the entropy of at least one first patch in the first image data with the entropy of at least one second patch in the second image data, wherein the position of the at least one first patch in the first image data corresponds to the position of the at least one second patch in the second image data. Imaging system (1).

2. The imaging system (1) according to claim 1, wherein the first optical path (11) and the second optical path (21) have different lengths.

3. The imaging system (1) according to claim 1 or 2, wherein the first sensor (10) and the second sensor (20) are further configured to simultaneously image the object (3).

4. The imaging system (1) according to any one of claims 1 to 3, further comprising a focus system (4) disposed in the first optical path (11) and / or the second optical path (21) and configured to change the focus of the first sensor (10) and / or the second sensor (20).

5. The imaging system (1) according to claim 4, further comprising a focusing means (104) configured to control the focusing system (4) so ​​that the focus of the first sensor (10) and / or the second sensor (20) can be adjusted.

6. The imaging system (1) according to any one of claims 1 to 5, wherein the first image data and the second image data represent the same region of the object (3).

7. The imaging system (1) according to claim 1, wherein the at least one first patch has the same size as the at least one second patch, i.e., a size of 20 × 20 pixels.

8. The imaging system (1) according to any one of claims 1 to 7, wherein the optical channel (2) is rotatable so that the fields of view of the first sensor (10) and the second sensor (20) are variable.

9. A laparoscope (100) comprising an imaging system (1) according to any one of claims 1 to 8.

10. The first sensor (10) is used to image the object along the first optical path (11) to generate first image data of the object, The second sensor (20) is used to image the object along the second optical path (21) to generate a second image data of the object, Based on the first image data and the second image data, depth information of the object is generated, The method includes generating depth information by comparing the entropy of at least one first patch of the first image data with the entropy of at least one second patch of the second image data, The position of the at least one first patch in the first image data corresponds to the position of the at least one second patch in the second image data, The first sensor (10) and the second sensor (20) have their focus shifted, The first optical path (11) and the second optical path (21) are guided, at least in part, through the same optical channel (2). A method for imaging an object, wherein the first sensor (10) and the second sensor (20) are arranged so as to be inclined relative to each other.

11. The method according to claim 10, further comprising the step of comparing the first image data and the second image data with each other in order to generate depth information.

12. The method according to claim 10 or 11, wherein the first image data and the second image data are generated simultaneously.

Images