Multi-viewpoint video capture device

Abstract

This multi-viewpoint video filming device for surgical operations avoids shielding of illumination by the head and body of an operator and reflection of the head in a video, by disposing, in a space between the top of the head of the operator and an affected area, surgery filming illumination equipment attached to a housing having a finite-length curved shape, such as a hollow ring shape or an arc shape, the housing being devised such that a plurality of cameras and a plurality of illumination instruments are not obstacles to the visual field and work of the operator. Said multi-viewpoint video filming device enables surgery assistance / recording of surgery under direct vision through multi-viewpoint video by being provided with a function for inferring contexts of the plurality of cameras and adding a video information processing function for selecting a camera video in which there is little reflection of the hand of the operator or the surgical instrument in the image.

Description

[Technical field]

[0001] The present invention relates to a multi-viewpoint video capture device for use in surgery, dissection, and the like. [Background technology]

[0002] The basis of surgical procedures is open surgery, which is performed in a large surgical field under laparotomy or thoracic surgery, and is a reliable surgical method that can respond to various unforeseen circumstances even in highly difficult surgeries. However, surgeons currently perform open surgery based only on information they see with their own eyes, and this method has hardly been improved in the history of modern surgery, which has continued for over 100 years.

[0003] If we compare direct vision surgery to flying an airplane, modern surgeons are in a similar situation to pilots who are still flying long distance flights under visual control without any support. In that sense, it could be said that the field of surgery is one that has been left behind by IT technology.

[0004] In response to this, laparoscopic surgery, in which an endoscope is inserted into the patient's abdomen or chest and the endoscopic image is displayed on a large monitor, has rapidly become popular in recent years. In laparoscopic surgery, a single clear enlarged image can be shared by all participants, which contributes to improving safety. Furthermore, since the image capturing the center of the surgical site can be easily recorded and archived, it is also greatly contributing to the creation of surgical teaching materials for retrospective learning with young surgeons and students and for surgical education.

[0005] In recent years, the scope of laparoscopic surgery has expanded dramatically, and many highly difficult surgeries are now being performed using it, but there is a risk of problems occurring when using awkward instruments while viewing partially enlarged images of the surgical site. For this reason, open surgery continues to play a central role in surgical procedures. Therefore, there is a strong demand for the development of technology that can effectively provide IT information during open surgery, which is the basis of surgical procedures.

[0006] Traditionally, video recording in direct surgery has been done with a single video camera placed above the surgeon's head. However, when the surgeon tries to look through the camera, it is often blocked by the head, shoulders, or other parts of the body, making it difficult to capture useful images. To solve this problem, it was necessary to have at least one video staff member on hand to frequently adjust the camera position to capture images in the gaps between the heads of multiple surgeons.

[0007] However, even with such efforts, it was not possible to completely avoid the obstruction of the image caused by the surgeon peering into the camera. In addition, in large hospitals with relatively ample human resources, it is possible to assign such personnel for special direct surgery intended for recording, but even so, it is not possible to casually record all routine surgeries. Furthermore, in city hospitals with few personnel, it is almost impossible to effectively record surgical images of direct surgery.

[0008] In open surgery, if caregivers and observers outside the surgical field (anesthesiologists, nurses, medical students, etc.) could also see the surgical field from the surgeon's point of view, it would be possible to enjoy the benefits of sharing the surgical field, as in laparoscopic surgery, but until now there has not been a imaging system that makes this possible. As a countermeasure, many attempts have been made to record the surgeon's line of sight by attaching a small camera to the surgeon's head or glasses. However, in actual clinical practice, the image is very blurred due to the surgeon's irregular head movements, making it impossible to capture images that are suitable for observation.

[0009] Furthermore, while the surgeon will often be concentrating and looking only at a very narrow surgical field, it is effective in increasing the safety of the surgery for the surgeon to perform surgery while referring to an image of the surgical field seen from a wide field of view or an image of the surgical field seen from the opposite side of the surgeon's perspective. However, up until now, surgeons have not been able to see images of the surgical field from any viewpoint other than their own. In addition to white light, far-infrared light can be used to highlight cancer lesions and lymph nodes injected with ICG fluorescent dye, and to visualize blood vessels, and this will be an important function of surgical support devices in the near future.

[0010] Patent Document 1 describes a camera system that, when photographing medical procedures such as surgery, avoids obstruction of the camera's field of view caused by the therapist's head, the surgeon's hands, surgical equipment, etc. when looking into the camera, and displays medical procedures and the condition of the affected area being treated on the screen with a high probability. The technology described in Patent Document 1 places a camera mechanism with a shadowless lamp equipped with multiple light sources above the surgeon's head. This camera mechanism with a shadowless lamp has one central camera and multiple surrounding cameras, and is configured to simultaneously capture images of the surgical site, which is the target of illumination, from multiple viewpoints.

[0011] Furthermore, in the technology described in Patent Document 1, a monitor mechanism is provided, which selects the necessary image from multiple images captured by the central camera and multiple surrounding cameras and displays it on a display device. Moreover, Patent Document 2 discloses a technique for capturing an image of blood vessels using infrared light and a cold mirror or a hot mirror.

[0012] In recent years, technology has developed that captures a scene synchronously with multiple cameras and generates images (free viewpoint images) that reproduce the scenery as it would be seen from any viewpoint (virtual viewpoint). This makes it possible, for example, to view highlight scenes of a soccer or basketball game from a variety of angles. [Prior art documents] [Patent documents]

[0013] [Patent Document 1] JP 2019-42199 A [Patent Document 2] JP 2017-68810 A [Patent Document 3] JP 2019-3320 A Summary of the Invention [Problem to be solved by the invention]

[0014] Applying multi-view imaging technology to direct surgery poses a completely different challenge than multi-view imaging technology, which is intended for watching sports, etc. The challenge is that the positional relationship between the subject, camera, and observer is significantly different in the shooting environment of a stadium and that of an operating room. In a stadium, there are no observers, i.e. viewers, on the straight line between the subject, i.e. the player, and the camera, and there are no obstructions.

[0015] On the other hand, in direct surgery, the surgeon is always between the camera installed above the patient's head and the surgical site, which is the subject of the operation. The surgeon's head and shoulders act as the first obstruction, and the surgeon's hands and surgical instruments working directly above the surgical field act as the second obstruction, making it impossible to obtain synchronized images of the subject from multiple cameras.

[0016] Patent Document 1 is an example of applying multi-viewpoint video to surgery without fully considering the above-mentioned occlusion problem in direct surgery. The camera mechanism with shadowless light described therein is very large and is placed above the surgeon's head, so many of the installed cameras are blocked by the surgeon's head, making it difficult to effectively capture the surgical site. In the end, as in the case of capturing images of direct surgery with a single conventional camera, it is difficult to obtain meaningful images unless a dedicated staff member is assigned to capture the images and the camera position is continually adjusted. Furthermore, it has been difficult to obtain usable images from a sufficient number of multiple cameras to obtain multi-viewpoint images, since many of the multiple cameras are obscured by the surgeon's head, shoulders, etc. and become ineffective.

[0017] Meanwhile, an essential element in performing surgery is the lighting device that illuminates the surgical field. Shadowless lamps equipped with multiple lights are designed in the hope that they will literally cast no shadows. However, in reality, lighting from a large shadowless lamp such as that described in Patent Document 1 is often blocked by the surgeon's head or shoulders, and an effective amount of light does not reach the surgical field. As a result, the surgeon or an assistant must frequently adjust the position of the lighting device during surgery.

[0018] As described above, with conventional equipment, including the overhead multiple camera system, it was difficult to effectively display and record video images of the surgical field without a dedicated camera staff. In addition, IT technology called multi-view imaging, which has made great strides in recent years in the field of sports spectating, could not be applied to the surgical field.

[0019] As a method for generating free viewpoint video, there is a technology related to a method for estimating geometric information such as the position and shape of a subject, as described in Patent Document 3. However, to generate free viewpoint video that is meaningful for surgical operations, geometric information alone is insufficient, and in addition, it is necessary to obtain semantic context information and generate a presentation video based on it.

[0020] Therefore, even if the first problem of the surgeon's head and body blocking the image or lighting is solved, it is impossible to obtain effective multi-view images because there is always a second obstruction, the surgeon's hands or surgical instruments, directly above the subject. In other words, in order to apply multi-view image technology to surgical sites, it is necessary to have a function that judges the situation of the shooting scene, that is, estimates context information, and automatically selects camera images that are not obstructed by the surgeon's hands or surgical instruments, based on the context information.

[0021] Furthermore, when this device is used for real-time surgical navigation, if it can present the surgeon with images from other viewpoints that he or she cannot notice from his or her own perspective, this will be an effective technology for improving surgical safety. To achieve this, it is necessary to have a function that can automatically recognize the limited narrow area within a wide shooting range where the surgeon is gazing. On top of that, it is also necessary to have a function that can determine and present to the surgeon effective images other than the one the surgeon is currently gazing at.

[0022] To provide a multi-viewpoint video capture device that can display and record images of the surgical field without being obstructed by the surgeon's head or body, and without being hindered by the surgeon's hands or surgical instruments, without the need for a dedicated camera staff for filming. In other words, the present invention solves the first problem of images and lighting being blocked by the surgeon's head and body, which could not be avoided with existing devices, by innovating the shape and positioning of the equipment, and further solves the second problem of obstruction caused by the surgeon's hands and surgical instruments using image information processing technology, with the aim of applying multi-viewpoint video capture technology to the field of surgery. On the other hand, the multi-viewpoint video shooting device of the present invention must be designed in such a way that it does not obstruct the surgeon's line of sight or impede his or her work. [Means for solving the problem]

[0023] In order to solve the above problems, the multi-viewpoint video capture device of the present invention comprises a capture device attached to a housing having a finite length curved shape such as a ring-shaped circular or arc-shaped housing, with multiple cameras aimed at a subject on which the surgeon is working, and a fixing device that positions the capture device in a position between the top of the surgeon's head and the subject.

[0024] A preferred form of the present invention comprises a camera calibration processing unit that estimates camera parameters including the positions, attitudes, and focal lengths of multiple cameras attached to an imaging device based on imaging information of a subject captured by the multiple cameras, and a presented viewpoint gaze point determination processing unit that detects a gaze point, which is the area of ​​the subject that the surgeon is paying attention to, in multi-view image data, estimates three-dimensional coordinate information of the gaze point by referring to the camera parameters, and selects one or more cameras that will best capture the gaze point based on the three-dimensional coordinate information. Furthermore, by arranging a number of lights between the multiple cameras attached to the housing, a lighting environment that is not blocked by the surgeon's head or shoulders is provided. Effect of the Invention

[0025] According to the present invention, it is possible to continuously capture clear images of the surgical field under a lighting environment that is not obstructed by the head or body of the surgeon, without the need for dedicated camera staff. Furthermore, by providing a function to effectively capture the center of the surgical field that the surgeon is gazing at from multiple viewpoints, it is also equipped with a surgical support function that provides the surgeon with detailed information on the surgical field from viewpoints other than the surgeon's in real time, which contributes to improving the safety of open surgery. Furthermore, recording this clear surgical field image allows the storage of a highly reliable visual record of the surgery.

[0026] This is extremely valuable as a medical record, and is expected to contribute greatly to surgical education. Furthermore, in surgeries performed at hospitals in areas with a shortage of medical staff, clear images of the surgical field can be obtained without a camera crew, so it is expected that this can be used to improve the safety of remote medical care, for example by transmitting these images in real time to experienced surgeons in remote locations for advice.

[0027] In addition, as described below, variant example 3 eliminates the need for a doctor to hold a camera during laparoscopic surgery, not only improving the efficiency of human resources but also contributing to improved safety by providing wide, multi-perspective images instead of the conventional narrow field of view.

[0028] The multiple illumination light sources installed in the multi-viewpoint video capture device of the present invention are installed at a position lower than the top of the surgeon's head, which is a feature of this device, so that the surgeon's head and body are not obstructed at all, and a shadow-free environment can be provided for the operating field. Furthermore, by equipping light sources with different wavelengths such as infrared light and ultraviolet light, it is possible to provide a lot of information that the surgeon cannot recognize with the naked eye. Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0029] [Figure 1] 1A and 1B are schematic diagrams showing a state in which a multi-viewpoint video shooting device according to an embodiment of the present invention is used, and a top view of a state in which a shooting lighting device is used. [Diagram 2] 1A and 1B are a transparent side view, a bottom view, and a transparent oblique view of a photographic lighting device. [Diagram 3] FIG. 13 is a simulation diagram of a hollow ring type multi-viewpoint image capturing device being used in an operating room. [Figure 4] FIG. 1 is a six-view diagram of a hollow ring type multi-viewpoint image capture device. [Diagram 5] FIG. 13 is a simulation diagram seen from a surgeon's viewpoint when using a hollow ring type multi-viewpoint image capture device. [Figure 6] 1A to 1C are diagrams showing examples of installation variations (polygonal, arc-shaped) of the multi-viewpoint video imaging device. [Figure 7] 1A to 1C are diagrams showing examples of installation variations (linear, V-shaped) of the multi-viewpoint video imaging device. [Figure 8] 1A to 1C are diagrams showing examples of installation variations (arch type, combined type) of the multi-viewpoint video shooting device. [Figure 9] FIG. 13 is a diagram showing a variation without lighting, an example of the installation of photographic equipment. [Figure 10] FIG. 13 shows a variation without a video capture device, an example of a lighting fixture. [Figure 11] 1 is a block diagram showing a hardware configuration of a multi-viewpoint video shooting device according to an embodiment of the present invention. [Figure 12] FIG. 2 is a block diagram showing software functions of the image lighting processing device. [Figure 13] 13 is a flowchart showing the flow of processing by a presentation viewpoint / gazing point determination processing unit. [Figure 14] FIG. 11 is a schematic diagram showing a state in which a multi-viewpoint video shooting device according to a first modified example of the present invention is used. [Figure 15] FIG. 13 is a schematic diagram showing a state in which a multi-viewpoint video shooting device according to a second modified example of the present invention is used. [Figure 16] FIG. 13 is a schematic diagram showing an LED light and a camera of a photographic lighting device according to a third modified example of the present invention. [Figure 17] FIG. 4 is a schematic diagram showing the LED lighting and camera of a photographic lighting device according to a fourth variant of the present invention, a time chart showing the emission timing of the visible light LED and infrared LED, and a block diagram showing the software functions of an infrared image acquisition unit provided within the input / output control unit of an image lighting processing device. [Figure 18] FIG. 13 is a schematic diagram showing a variation of a suspension arm according to a fifth modified example of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] [Multi-viewpoint video capture device] FIG. 1A is a schematic diagram showing a state in which a multi-viewpoint video imaging device 101 according to an embodiment of the present invention is in use, and FIG. 1B is a top view showing a state in which an imaging lighting device 102 is in use. 1A, a multi-viewpoint video shooting device 101 has a shooting lighting fixture 102 and an image lighting processing device 103 connected thereto. In addition, a large-screen external monitor 104 is connected to the image lighting processing device 103.

[0031] A patient 106, who has been anesthetized and has had his abdomen cut open, lies on an operating table 105. A surgeon 107a operates on the patient 106, and a surgeon 107b assists the surgeon 107a in the operation. In other words, the surgeon 107a is the surgeon who performs the operation, and the surgeon 107b is the assistant. Hereinafter, the surgeon 107a and the assistant 107b will be collectively referred to as "surgeon 107". The width of the operating table 105 is approximately 50 cm. The incised abdomen, that is, the surgical field 106a, has an opening with a short side of approximately 20 cm and a depth of approximately 10 cm.

[0032] 1A and 1B, a hollow ring-shaped photography lighting device 102 is installed at a position about 40 cm above the center of the surgical field 106a. In the example of FIG. 1, the photography lighting device 102 is a circular or elliptical ring-shaped housing, and multiple cameras (see FIG. 2B) and multiple lights (see FIG. 2B) are installed facing the surgical field 106a so as not to obstruct the field of view of the surgeons 107a and 107b when they view the surgical field 106a.

[0033] The surgeon 107 looks into the surgical field 106a from the hollow part of the photographing and lighting equipment 102 and performs surgical operations by inserting his / her hands into a space of about 40 cm below the photographing and lighting equipment 102. In order to secure a space for performing this surgical operation, the photographing and lighting equipment 102 is suspended from a structure above the surgical field (e.g., the ceiling) by, for example, a fixing equipment 102b, and is installed between the top of the surgeon's head and the surgical field 106a.

[0034] The surgeon 107 can view the surgical field 106a as if peering through the hollow part of the photographing and lighting equipment 102, but can also use the space between the photographing and lighting equipment 102 and the surgical field 106a to perform surgery and assist in the surgery while referring to the image of the surgical field 106a displayed on the external monitor 104.

[0035] In conventional open surgery, the area 60 cm above the surgical site is widely recognized in the surgical field as a sacred area that cannot be entered, and it is an area where new medical equipment is hesitant to be developed and placed. In other words, since monitoring of surgery in open surgery is entirely based on the visual observation of the surgical field 106a by the operating surgeons 107a and 107b with their naked eyes, placing any object 60 cm above the surgical site is recognized as obstructing the view of the surgeon 107.

[0036] The imaging lighting device 102 used in the multi-viewpoint image capturing device 101 according to the embodiment of the present invention is an instrument that is placed within 60 cm directly above the surgical site, which is an area that was previously taboo. However, the surgeon 107 can visually confirm the surgical field as if peering into the ring-shaped device installed in midair, and since a sufficient working space is secured between the device and the affected area, the surgeon 107 can perform the surgical operation smoothly. It is also possible to continue capturing clear images of the surgical field under a lighting environment that is not obstructed by the head or shoulders of the surgeon 107.

[0037] The multiple cameras provided on the photographing lighting equipment 102 are arranged to surround the surgical field 106a, so that the surgical field 106a can be photographed from various directions during surgery. Conventionally, in order to capture images of the surgical field 106a from various directions, the camera was moved to the required direction each time. In the case of the photographing lighting equipment 102 constituting a part of the multi-viewpoint image capturing device 101 according to the embodiment of the present invention, a function to physically move each camera is not required. By electronically switching the images of the multiple cameras arranged in advance to surround the surgical field 106a as required, the surgeon 107 can easily obtain images from the shooting direction required.

[0038] The image captured by the camera of the photographing lighting device 102 can be viewed on an external monitor 104 installed near the operating table 105. The surgeon 107 can view the surgical field 106a with the naked eye and also on the large-screen external monitor 104. Furthermore, as the name suggests, the multi-viewpoint video capture device 101 can form an image from any viewpoint based on images obtained from multiple cameras. Therefore, by switching the cameras and moving the viewpoint, it is possible to view the subject, the surgical field 106a, in three dimensions.

[0039] [Overall image of photography lighting equipment 102] 2A is a transparent side view of the photographic lighting device 102, FIG. 2B is a bottom view of the photographic lighting device 102 seen from below, and FIG. 2C is a transparent perspective view of the photographic lighting device 102. FIG. The photographic lighting device 102 has a hollow circular or elliptical ring-shaped housing 102a and a fixing device 102b. The fixing device 102b is provided on the photographic lighting device 102 in order to position and fix the housing 102a of the photographic lighting device 102 at a position between a surgical field 106a, which is the subject, and the top of the head of a surgeon 107, who is the operator.

[0040] 2A, 2B, and 2C, the size of the housing 102a is, for example, a circular shape of 30 to 40 cm. Since the housing 102a has a hollow shape, the surgeon 107 can directly see the surgical field 106a with the naked eye from the hollow part of the housing 102a.

[0041] The thinner the housing 102a, the better so as not to obstruct the field of view of the surgeon 107. As imaging elements have become smaller in recent years, the housing 102a may be formed from a thin pipe and the imaging element may be housed in the pipe. Alternatively, an imaging element of, for example, about 1 cm may be fastened and fixed to the outside of a pipe that is too thin to house the imaging element.

[0042] That is, in open surgery, the surgeon 107 can view the surgical field 106a not only directly with the naked eye but also through the external monitor 104. The existence of multiple means for observing the surgical field 106a provides the surgeon 107 with a sense of psychological security and contributes to improved safety. In order not to obstruct the view of the surgeon 107, the thinner the ring-shaped housing 102a, the better. Also, in order not to obstruct the operation of the surgeon or the assistant surgeon, it is essential that the fixing device 102b is not interposed between the imaging lighting device 102 and the surgical field 106a.

[0043] 2A and 2B, for example, 36 cameras 201 and 36 lights 202 are arranged alternately in the housing 102a. These cameras 201 and lights 202 are arranged facing the surgical field 106a as shown in FIG. 2C. When the shape of the housing 102a is a perfect circle and there are 36 cameras 201, each camera 201 captures an image surrounding the surgical field 106a within a range of 10°.

[0044] That is, the cameras 201 are arranged in the housing 102a at a predetermined interval, and all of the cameras 201 are arranged facing the surgical field 106a. Therefore, the cameras 201 are arranged in the housing 102a at different shooting angles with respect to the surgical field 106a, which is the subject of the camera. However, if the number of cameras 201 is large, the cameras 201 will be arranged side by side without any gaps between each other. Even in such a case, each camera 201 is arranged in the housing 102a at a different shooting angle with respect to the surgical field 106a, which is the subject.

[0045] As described above, the photographing lighting equipment 102 is placed within 60 cm directly above the surgical field 106a, which is the subject. Therefore, the cameras 201 placed in the housing 102a are located extremely close to the subject, and therefore the photographing angles of all the cameras 201 are necessarily different.

[0046] The number of cameras 201 and lights 202 may be determined arbitrarily, but ideally, in order to obtain high-resolution multi-viewpoint images, it is preferable to arrange 36 or more cameras 201. When the angle between the cameras 201 is about 10°, the transition between images appears smooth to the surgeon 107 watching the images captured by the cameras 201. When the angle between the cameras 201 is about 15° or more, the transition between images becomes clearly noticeable to the surgeon 107 watching the images captured by the cameras 201, and the transition may feel unnatural.

[0047] It is desirable to use an ultra-compact image sensor capable of capturing highly detailed color images, such as those used in recent years in smartphones, for the camera 201. Each camera 201 is equipped with an interface, such as a USB interface, and is connected to the image lighting processing device 103 shown in FIG. 1 via a USB hub 203.

[0048] It is desirable to use, for example, a high-brightness LED that emits the three primary colors with high color reproducibility as the light source used for the illumination 202. Since LEDs generally have directionality, it is desirable to appropriately disperse the light toward the surgical field 106a, which is the subject, by sealing the LED in the housing 102a with an acrylic or polyvinyl chloride plate that has been opaqued by a smoke finish or sandblasting process. The light source used for the lighting 202 is not limited to an LED. For example, a krypton bulb may be used, or an LED and a krypton bulb may be mixed to improve color reproducibility.

[0049] 3 is a simulation diagram of the state in which the basic ring-shaped photographing lighting device 102 is used in an operating room. A surgeon 107 observes the surgical field by peering into the ring formed by the housing 102a of the photographing lighting device 102, and performs surgery using a working space of about 40 cm between the device and the surgical field.

[0050] 4A is a top view of the photographic lighting device 102, FIG. 4B is a bottom view of the photographic lighting device 102, FIG. 4C is a front view of the photographic lighting device 102, FIG. 4D is a rear view of the photographic lighting device 102, and FIGS. As can be seen from Figures 4A, 4B, 4D, 4E and 4F, the fixing device 102b is provided with a handle 401 that allows the surgeon 107 to place the imaging lighting device 102 in an appropriate position.

[0051] Figure 5 is a simulation of the surgical field as seen from the viewpoint of a surgeon 107. Although the device is partially visible at the edge of the field of view, there is little obstruction of the surgical field, allowing the surgeon to perform surgical operations comfortably.

[0052] 6, 7 and 8 show examples of various modified shapes of the device. FIG. 6A is a simulation diagram of an operating room using a photographing lighting device 601 according to a modified example of the first shape, and FIG. 6B is a top view of the photographing lighting device 601 according to the modified example of the first shape. The photographic lighting device 601 has a rectangular housing 601a and a fixing device 601b. The fixing device 601b is provided with a handle 401, similar to the photographic lighting device 102 described above.

[0053] FIG. 6C is a simulation diagram of an operating room using a photographing lighting device 602 according to an example of a variation of the second shape, and FIG. 6D is a top view of the photographing lighting device 602 according to an example of a variation of the second shape. The photographic lighting device 602 includes a first housing 602a having an arc shape, a first fixing device 602b, a second housing 602c having an arc shape, and a second fixing device 602d. The first fixing device 602b and the second fixing device 602d are each provided with a handle 401, similar to the photographic lighting device 102 described above.

[0054] FIG. 7A is a simulation diagram of an operating room using a photographing lighting device 701 according to an example of a modified third shape, and FIG. 7B is a top view of the photographing lighting device 701 according to an example of a modified third shape. The photographic lighting device 701 includes a first housing 701a having a linear shape, a first fixing device 701b, a second housing 701c having a linear shape, and a second fixing device 701d. The first fixing device 701b and the second fixing device 701d are each provided with a handle 401, similar to the photographic lighting device 102 described above.

[0055] FIG. 7C is a simulation diagram of an operating room using a photographing lighting device 702 according to an example of a modification of the fourth shape, and FIG. 7D is a top view of the photographing lighting device 702 according to an example of a modification of the fourth shape. The photographic lighting device 702 has a V-shaped housing 702a and a fixing device 702b. The fixing device 702b is provided with a handle 401, similar to the photographic lighting device 102 described above.

[0056] Figure 8A is a simulation diagram of an operating room using a photographing lighting device 801 relating to an example of a modified fifth shape, Figure 8B is a top view of the photographing lighting device 801 relating to an example of a modified fifth shape, and Figure 8C is a view of the photographing lighting device 801 relating to the example of a modified fifth shape from the side. The photographic lighting device 801 has a vertically arc-shaped housing 801a and a fixing device 801b. The fixing device 801b is provided with a handle 401, similar to the photographic lighting device 102 described above.

[0057] Figure 8D is a simulation diagram of an operating room using a photographing lighting device 802 relating to an example of a sixth shape deformation form, Figure 8E is a top view of a photographing lighting device 802 relating to an example of a sixth shape deformation form, and Figure 8F is a view of a photographing lighting device 802 relating to an example of a sixth shape deformation form from the side. The photographic lighting device 802 has a housing 802a having an arc portion in the vertical direction and further arc ends at both ends of the arc, and a fixing device 802b. The fixing device 802b is provided with a handle 401, similar to the photographic lighting device 102 described above.

[0058] Polygonal shapes such as the rectangle shown in Figures 6A and 6B allow for the arrangement of a sufficiently large number of camera+light combinations, just like the basic ring shape, making it possible to create multi-viewpoint images. The C-type x 2 in Figures 6C and 6D and the I-type x 2 in Figures 7A and 7B are designed to eliminate the devices in front of the surgeon and assistant in order to further reduce the visual impairment caused by the 360° ring type, and to ensure better visual field and workability. However, this format comes at the cost of improved visual field, as the number of cameras and lights that can be arranged is reduced, and it cannot be denied that the quality of the multi-view images may be reduced.

[0059] Figures 7C and 7D show V-shaped devices placed on one side. While this provides excellent visibility and operability for the surgeon and assistant, it is expected that the quality of the multi-view imaging device will decline because the number of camera + light combinations is halved. On the other hand, Figures 8A, 8B, and 8C show an example of a modified configuration in which multiple cameras and lights are vertically arranged in an arch shape in a narrow area between the surgeon and assistant. It is possible to arrange many cameras and lights vertically while maintaining the visibility and workability of the surgeon and assistant.

[0060] The modified examples of Figures 8D, 8E, and 8F show a combination of the two C-types shown in Figure 6D and the arch-types shown in Figures 8B and 8C. It is possible to arrange a sufficient number of cameras and lights to compose a multi-viewpoint image while maintaining good visibility and workability for the surgeon and assistant.

[0061] FIG. 9A is a top view of the photographic fixture 901 as a variation without illumination, and FIG. 9B is a schematic diagram showing the photographic fixture 901 in use. The photographic equipment 901 is obtained by removing the lighting 202 from the photographic lighting equipment 102 and incorporating only the multiple cameras 201 in a housing 901a.

[0062] The photographing lighting device 102 according to the embodiment of the present invention is basically structured such that both multiple cameras and multiple lights are arranged alternately, but as shown in Figures 9A and 9B, it is also possible to design and use a variation without lighting as a photographing device 901 with multiple cameras. In this case, shadowless lighting that is installed in existing operating rooms can be used, which is advantageous in terms of the cost required for introducing the equipment.

[0063] FIG. 10A is a top view of lighting fixture 1001 as a variation without a camera, and FIG. 10B is a schematic diagram showing lighting fixture 1001 in use. The lighting fixture 1001 is obtained by removing the camera 201 from the photographing lighting fixture 102 and incorporating only a plurality of lights 202 in a housing 1001a.

[0064] The photographing lighting device 102 according to the embodiment of the present invention is based on a structure in which multiple cameras and multiple lights are arranged alternately, but variations in lighting only are also envisioned, as shown in Figures 10A and 10B. Conventional shadowless lights installed above the head of a surgeon 107 block light from the surgeon's 107's head and body, making it necessary to frequently adjust the lighting position, but this device, installed between the top of the surgeon's head and the patient, is not blocked by the surgeon's head or body at all, making it a completely shadowless lighting device with no shadows.

[0065] In order to minimize the obstruction of the operator's line of sight, the photographing lighting device 102 of the present invention is formed in various shapes of finite-length straight lines or finite-length curves as shown in Figures 3, 4, 5, 6, 7, and 8. The thickness of the housing 102a is made as thin as possible in accordance with the sizes of the multiple cameras 201 and lighting devices 202 stored therein so as not to obstruct the operator's line of sight.

[0066] Here, a curve is a concept including a straight line in the world of mathematics, and a straight line is a special example of a curve. Therefore, it can be said that the photographic lighting device 102 in the embodiment of the present invention, including the linear photographic lighting device 102, is formed of a housing made of a wire member of finite length. In other words, the shape of the photographic lighting device 102 of the present invention is not limited to the examples shown in Figures 3, 4, 5, 6, 7, and 8, but includes all housing shapes that achieve the purpose of not interfering with the field of view and work of the surgeon. The multiple cameras and multiple lights housed in this finite curved housing are all positioned facing the surgical field 106a, which is the subject of the image.

[0067] As described above, the imaging lighting device 102 used in the multi-viewpoint image capturing device 101 of the present invention can have various shapes. The housing 102a is desirably made of a lightweight and highly rigid material such as aluminum alloy, polycarbonate, carbon, etc. However, the camera 201 and the illumination 202 may be attached to a flexible pipe made of stainless steel or the like used for water pipes, etc., and configured to be freely deformable according to the shape of the surgical field 106a. If the material is one that can be freely deformed, such as a flexible pipe, it can be deformed into the various housing shapes described above.

[0068] [Multi-viewpoint video capture device 101: Hardware configuration] FIG. 11 is a block diagram showing the hardware configuration of a multi-viewpoint video imaging device 101 according to an embodiment of the present invention. As shown in FIG. 11, multi-viewpoint video shooting device 101 is made up of camera array section 1101, lighting array section 1102, and video lighting processing device 103 connected thereto. The camera array unit 1101 and the lighting array unit 1102 are built into the photographing lighting device 102 . Furthermore, a large-screen external monitor 104 is connected to the image lighting processing device 103 via a network 1103 .

[0069] The camera array unit 1101 is a group of multiple cameras 201 incorporated in the imaging lighting equipment 102, as described in Figures 1 and 2. These cameras 201 are connected to a serial interface 1128, such as a USB interface, that the image lighting processing device 103 has.

[0070] The lighting array unit 1102 is an assembly of multiple lights 202 incorporated in the imaging lighting device 102, as described in Fig. 1 and Fig. 2. These lights 202 are selectively controlled to emit light by a light 202 designated by the video lighting processing device 103 through a multiplexer 1105 controlled by a microcomputer 1104. The lights are driven to emit light by a current flowing from a power supply voltage node through a current limiting resistor R1106 and the multiplexer 1105. Note that the switching control of the multiplexer 1105 is configured to select not only one light, but also multiple lights simultaneously, including turning on all the lights at once, as necessary.

[0071] A microcomputer 1104 is connected between the multiplexer 1105 and the image lighting processing device 103, and the microcomputer 1104 controls the multiplexer 1105 according to commands from the image lighting processing device 103. The microcomputer 1104 is connected to a serial interface 1128 such as a USB interface that the image lighting processing device 103 has.

[0072] The microcomputer 1104 includes a CPU 1108, a ROM 1109, a RAM 1110, and a serial interface 1111 (abbreviated as “SI / F” in FIG. 11) such as a USB interface, all of which are connected to a bus 1107. The multiplexer 1105 is connected to the bus 1107 and is controlled by the microcomputer 1104.

[0073] The image lighting processing device 103 , which is a well-known computer, includes a CPU 1122 , a ROM 1123 , a RAM 1124 , a display unit 1125 , an operation unit 1126 , and a non-volatile storage 1127 , all of which are connected to a bus 1121 . The CPU 1122 reads out software programs that realize the functions of each unit of the image lighting processing device 103 from the non-volatile storage 1127 or the ROM 1123 and executes them.

[0074] The RAM 1124 temporarily writes images of all cameras 201 received from the camera array unit 1101 and variables and the like generated during the arithmetic processing performed in the image lighting processing device 103. The CPU 1122 executes programs recorded in the non-volatile storage 1127 or the ROM 1123 to realize various functions of the image lighting processing device 103.

[0075] In addition to the above, a serial interface 1128 such as a USB and a network interface card (NIC) 1129 are connected to the bus 1121 as communication interfaces. The external monitor 104 is connected to the NIC 1129 via a network 1103. The external monitor 104 includes a well-known computer, and the computer runs a network OS and functions as the external monitor 104 of the image lighting processing device 103.

[0076] The display unit 1125 of the image lighting processing device 103 may be used as a monitor as is, but since there are generally restrictions on the length of the connection cable between the display unit 1125 and the image lighting processing device 103, it is preferable to connect it to the external monitor 104 via a network 1103. Furthermore, by constructing the image lighting processing device 103 as a network video server, multiple external monitors 104 can be easily connected, making it possible to simultaneously display the same desired image on different monitors. Therefore, even if the external monitor 104 breaks down due to some accident, it is possible to prevent accidents during surgery by continuing to use the spare external monitor 104 as is.

[0077] [Image lighting processing device 103: software function] FIG. 12 is a block diagram showing the software functions of the image lighting processing device 103. The image lighting processing device 103 includes an input / output control unit 1201 , a camera calibration processing unit 1202 , a presentation point / gazing point determination processing unit 1204 , an image processing unit 1205 , an illumination determination processing unit 1206 , and a display unit 1125 .

[0078] The camera array unit 1101 is made up of a number of cameras 1 to N embedded in the photographing lighting equipment 102 and arranged to surround the periphery of the surgical field 106a, and photographs the surgical field 106a from various directions during surgery.

[0079] The lighting array unit 1102 is embedded in the photographing lighting fixture 102 together with the camera 201 and is arranged to surround the periphery of the surgical field 106a. Normally, the lighting array unit 1102 is composed of a large number of LED lighting units 1 to N, the number of which is equal to the number of cameras 201 in the camera array unit 1101. Then, the lighting units 202 adjacent to the camera 201 selected by the presentation viewpoint / gazing point determination processing unit 1204 are controlled to emit light. If necessary, all the lighting units are controlled to emit light at the same time.

[0080] Input / output control unit 1201 receives video data obtained from camera array unit 1101, and also provides lighting array unit 1102 with control information for switching between lights 1 to N.

[0081] Input / output control unit 1201 and camera array unit 1101 are connected via a bidirectional serial interface 1128 . All cameras 201 in camera array unit 1101 transmit video data to input / output control unit 1201 .

[0082] The input / output control unit 1201 and the lighting array unit 1102 are connected via a USB interface. When the input / output control unit 1201 receives a lighting command from a lighting determination processing unit 1206 (described later), it transmits a light emission command to the lighting 202 specified by the lighting determination processing unit 1206 in accordance with the lighting command. In the lighting array unit 1102 , the microcomputer 1104 receives a light emission command from the input / output control unit 1201 and controls the multiplexer 1105 to control the light emission of the designated lighting 202 .

[0083] A serial interface connects input / output control unit 1201 and camera calibration processing unit 1202. Camera calibration processing unit 1202 estimates camera parameters such as the positions, orientations, and focal lengths of cameras 1 to N constituting camera array unit 1101 based on video information. The cameras 201 embedded in the photographing lighting equipment 102 are configured with an angle of view that allows a common area to be observed in the images of adjacent cameras 201. Then, camera parameters such as the position, orientation, and focal length of each camera 201 relative to the surgical field 106a, which is the subject, are estimated from the correspondence relationship of the common observation area.

[0084] In a surgical operation, for example, if the surgeon is right-handed, it is common to hold surgical instruments such as an electric scalpel or scissors in the right hand and use tweezers or one's own fingers to support the operation with the left hand. Therefore, the subject area that the surgeon 107 wants to focus on as the operation target is within a range of about 0 to 5 cm from the two points described below. (1) The tips of surgical instruments such as scalpels, tweezers, and scissors (2) The tips of the surgeon's fingers, which are not holding the surgical instrument The procedure for setting partial image data and its observation position will be described below for each article.

[0085] (1) Surgical instruments such as scalpels, tweezers, scissors, etc. Just before surgery, surgical instruments such as scalpels, tweezers, and scissors are photographed in advance by camera array unit 1101. Video data obtained from camera array unit 1101 is recorded in RAM 1124 of input / output control unit 1201 as multi-viewpoint video data. Next, a staff member other than the surgeon 107, such as a nurse, selects the tip of a surgical instrument such as a scalpel that is shown in the multi-viewpoint video data with a mouse or the like of the operation unit 1126. When selecting a surgical instrument such as a scalpel, the staff member such as a nurse also specifies two-dimensional coordinates that are the observation position on the screen. For example, a range of about 0 to 5 cm from the tip of a surgical instrument such as a scalpel or tweezers is specified as observation position coordinate information.

[0086] Partial image data and observation position coordinate information of surgical tools such as scalpels, tweezers, and scissors cut out from the multi-viewpoint video data are stored in the RAM 1124 or non-volatile storage 1127 as gaze point object video and gaze point coordinates 1207.

[0087] (2) In the case of the surgeon's finger Prior to surgery, the surgeon photographs his / her own fingers several times using the camera array unit 1101 without holding a surgical tool such as a scalpel. During the photographing, the surgeon photographs while performing various procedures. The image data obtained from the camera array unit 1101 is recorded as multi-viewpoint image data in the RAM 1124 of the input / output control unit 1201. Next, a staff member other than the surgeon 107, such as a nurse, selects the tip of the surgeon's finger that appears in the multi-viewpoint image data using a mouse or the like of the operation unit 1126. When selecting the surgeon's hand, the staff member, such as a nurse, also specifies two-dimensional coordinates that are the observation position on the screen. For example, if the surgeon is right-handed, a range of about 0 to 5 cm from the tip of the index finger of the left hand is specified as observation position coordinate information.

[0088] The cut-out partial image data of the surgeon's hand and the observation position coordinate information are stored in the RAM 1124 or non-volatile storage 1127 as a gaze point article image and gaze point coordinates 1207. The reason for preparing multiple pieces of multi-viewpoint video data of the surgeon's fingers is that since fingers can freely change their shape, in order to increase the probability of image matching, it is necessary to prepare multiple pieces of multi-viewpoint video data as the gaze point object image and gaze point coordinates 1207. The gaze point refers to a three-dimensional point estimated by applying triangulation based on camera parameters to a representative point (two-dimensional point) of a partial area of ​​a subject that the surgeon is focusing on in the multi-view image data obtained from the camera array unit 1101.

[0089] A serial interface is connected between the input / output control unit 1201 and the presentation viewpoint gaze point determination processing unit 1204. The presentation viewpoint gaze point determination processing unit 1204 refers to the gaze point article image and gaze point coordinates 1207, detects the location of the subject that the surgeon 107 wants to see from among the images of all the cameras 201 received from the input / output control unit 1201, and estimates the two-dimensional position coordinates.

[0090] Then, the presented viewpoint gaze point determination processing unit 1204 further determines a presented viewpoint that is a viewpoint that can most appropriately capture the gaze point. In this presented viewpoint determination processing by the presented viewpoint gaze point determination processing unit 1204, a camera 201 that captures images from a direction suitable for observation by the surgeon 107 is selected from the cameras 1 to N constituting the camera array unit 1101, taking into consideration the reflection of the hands and surgical instruments of the surgeon 107 in the captured image.

[0091] The camera calibration processing unit 1202 and the presented viewpoint / gazing point determination processing unit 1204 are connected by a serial interface. The presented viewpoint / gazing point determination processing unit 1204 calculates the location of the subject that the assistant 107b of the surgeon 107a wants to see, that is, the three-dimensional coordinates of the gaze point, based on the stereo vision, from the camera parameters such as the position, attitude, and focal length of the camera 201 received from the camera calibration processing unit 1202 and the estimated two-dimensional position coordinates. Once the three-dimensional coordinate information of the gaze point has been determined, the presentation viewpoint gaze point determination processing unit 1204 then specifies the camera 201 that clearly captures the gaze point. The presentation viewpoint / gazing point determination processing performed by the presentation viewpoint / gazing point determination processing unit 1204 has been described above.

[0092] The illumination determination processing unit 1206 basically performs the process of determining the presentation point and gaze point by the presentation point and gaze point determination processing unit 1204. Information is generated for controlling on the lights 202 arranged on both sides of the camera 201 that is identified as being able to clearly capture the gaze point. If no specific camera 201 is designated, all lights are controlled to be on.

[0093] The image processing unit 1205 is an interface for outputting the camera 201 that clearly captures the gaze point identified by the presented viewpoint / gaze point determination processing unit 1204 to the display unit 1125, the external monitor 104, etc. The image processing unit 1205 not only supplies multi-viewpoint images to the display unit 1125 installed in the operating room, but also provides the multi-viewpoint images to the external monitor 104a via the network 1103. The external monitor 104a can simultaneously display images from multiple cameras 201. For example, one of the cameras that is closest to the line of sight of the surgeon 107a and captures a clear image that is not obstructed by the hands of the surgeon or assistant or surgical instruments can be enlarged and displayed as a main image and shared by all participating members. In addition, for example, the external monitor 104b can present the surgeon 107a with an image from an angle different from his line of sight, and the external monitor 104c can present image information other than the direct view image to the surgeon, for example, by presenting image information other than white light captured by near-infrared rays to the external monitor 104c. Although not specifically shown, in order for the viewer or surgeon 107 to view the multi-viewpoint video with intuitive or simple operations, a function for manually selecting the camera 201 from the operation unit may be added to the presented viewpoint / gazing point determination processing unit 1204 and / or the video processing unit 1205.

[0094] [Processing flow of presentation viewpoint / gazing point determination processing unit 1204] FIG. 13 is a flowchart showing the flow of processing by the presentation point / gazing point determination processing unit 1204. When processing begins (S1301), the presentation viewpoint / gazing point determination processing unit 1204 acquires three-dimensional coordinate information of the position of the camera 201, the person area volume (mainly the position of the surgeon's 107's hand), the gaze point position, and the instrument area position based on the multi-view video and camera parameters received from the input / output control unit 1201 and the camera calibration processing unit 1202 (S1302).

[0095] Next, the presented point of view / gazing point determination processing unit 1204 determines whether or not the hand of the surgeon 107 is between the camera position and the position of the gazing point, based on the three-dimensional coordinate information of the gazing point (S1303). If it is determined in step S1303 that the hand of the surgeon 107 is between the camera position and the gaze point position (YES in S1303), a camera is searched for that does not have the hand of the surgeon 107 between the camera position and the gaze point position.

[0096] If it is determined in step S1303 that the surgeon's 107's hand is not between the camera position and the gaze point position (NO in S1303), the presented point of view gaze point determination processing unit 1204 then determines whether or not a surgical instrument is between the camera position and the gaze point position based on the three-dimensional coordinate information of the gaze point (S1304).

[0097] If the presentation viewpoint gaze point determination processing unit 1204 determines in step S1304 that a surgical instrument is present between the camera position and the gaze point position (YES in S1304), a camera that does not have a surgical instrument between the camera position and the gaze point position is searched for. If the presentation point of view / gazing point determination processing unit 1204 determines in step S1304 that there is no surgical instrument between the camera position and the point of view position (NO in S1304), it outputs the address information of the camera used among cameras 1 to N and the zoom value (S1306) and terminates the series of processes (S1307).

[0098] The multi-viewpoint video capture device 101 according to the embodiment of the present invention described above presents the surgeon 107 with not only direct viewing with the naked eye, but also live enlarged information from a viewpoint other than the naked eye, in surgery that has previously been performed only with direct viewing with the naked eye. The photographing lighting device 102 equipped with a camera 201 and a light 202 is positioned between the surgeon 107 and the patient 106, and unlike conventional surgical lighting devices, the light from the lighting device is not blocked by the heads of the surgeons 107a and 107b. Therefore, the surgical field 106a and the area around the gaze point are always brightly illuminated by the light 202, significantly improving visibility.

[0099] Furthermore, since the surgical field 106a and the area around the focus point enlarged and displayed by the multi-viewpoint image capture device 101 are multi-viewpoint images, the surgical field 106a and the area around the focus point can be viewed three-dimensionally by moving the viewpoint. In particular, the presented viewpoint gaze point determination processing unit 1204 identifies the three-dimensional coordinate information of the gaze point in the multi-viewpoint image being shot from the three-dimensional coordinate information of the gaze point previously stored in the gaze point object image and the gaze point coordinate 1207, thereby enabling appropriate switching of the viewpoint to be achieved, centered on the object to be focused on (e.g., the tip of a scalpel). As a result, the object of interest (for example, the tip of a scalpel) is always observed at the same location on the screen, making it easier to observe the object of interest. Due to the above-mentioned features, the multi-viewpoint video imaging device 101 according to the embodiment of the present invention can significantly improve the reliability and safety of surgery.

[0100] Even if there is no presented viewpoint gaze point determination processor 1204, illumination determination processor 1206, gaze point article image and gaze point coordinates 1207, it is possible to move the presented viewpoint by manual operation via the operation unit 1126. However, in this case, since the three-dimensional coordinate information of the gaze point is unknown, the viewpoint will be switched to center on an object other than the object to be focused on (e.g., the tip of a scalpel). As a result, the observation position of the object to be focused on (e.g., the tip of a scalpel) moves on the screen as the viewpoint moves.

[0101] [First Variation: A photography lighting device 102 in which a ring is placed directly on a surgical field 106a of a patient 106] FIG. 14 is a schematic diagram showing a state in which a multi-viewpoint video imaging device 1401 according to a first modified example of the present invention is used. A multi-viewpoint video camera 1401 shown in FIG. 14 differs from the multi-viewpoint video camera 101 shown in FIG. 1 in that the camera lighting device 1402 is miniaturized and placed directly on the surgical field 106a of the patient 106.

[0102] 1, a hollow ring on which a camera array unit 1101 and a lighting array unit 1102 are arranged is attached to a frame disposed between the patient 106 and the surgeon 107, for example, at a position about 30 cm away from the patient 106. This allows the surgeon 107 to perform surgery while visually observing the surgical field 106a. In the example of Figure 1, the head of the surgeon 107 is in the shadow of the lighting, making it possible to eliminate imperfections in the photography, but the problem is that the frame to which the hollow ring is attached may obstruct the surgeon's 107 view.

[0103] Therefore, in a multi-viewpoint video capture device 1401 according to a first modified example of the present invention shown in Fig. 14, a hollow ring in which a camera array unit 1101 and a lighting array unit 1102 are arranged is installed so as to cover the periphery of a surgical field 106a of a patient 106. In this case, the surgeon 107 can perform surgery while visually observing. With this surgical procedure, the procedure can be performed in the same condition as a normal surgical operation, so that the surgeon 107 is less hindered in his work compared to the photographing and lighting device 102 in Fig. 1. However, since the photographing and lighting device 1402 comes into contact with the patient 106, there is a possibility that the hands and surgical tools of the surgeons 107a and 107b will be cast in the shadows of the lighting device.

[0104] [Second Modification: A photography lighting device 102 in which a ring is directly inserted into the surgical field 106a of a patient 106] FIG. 15 is a schematic diagram showing a state in which a multi-viewpoint video imaging device 1501 according to a second modified example of the present invention is in use. The difference between the multi-perspective video camera 1501 shown in Figure 15 and the multi-perspective video camera 101 shown in Figure 1 and the multi-perspective video camera 1401 shown in Figure 14 is that the filming lighting device 1502 is even smaller than the filming lighting device 1402 of the first variant example shown in Figure 14 and is inserted directly inside the surgical field 106a of the patient 106.

[0105] 15 is an example in which a hollow ring in which a camera array unit 1101 and a lighting array unit 1102 are arranged is inserted into the abdominal cavity of a patient 106 and set up so as to look down on the surgical site from the upper part of the abdominal cavity. According to this surgical procedure, surgeons 107a and 107b can perform surgery while watching the captured image, and furthermore, the camera 201 can capture the entire abdominal cavity. However, the hollow ring in which the camera 201 and the lighting array unit 1102 are arranged must be inserted and removed through an insertion port provided in the abdomen of the patient 106 .

[0106] For example, it is necessary to insert the camera array unit 1101 and the lighting array unit 1102 arranged in a line into the abdominal cavity through a port and to deform them into a circle inside the abdominal cavity. After the surgery, when imaging is completed, a mechanism is required to deform them into a string again and take them out of the port.

[0107] [Third Modification: Imaging lighting device 1601 for simultaneously imaging blood vessels in the surgical field 106a of the patient 106 using an infrared LED 1605 and a cold mirror 1606 or a hot mirror] If an image of blood vessels is displayed on the external monitor 104 together with a visible light image during surgery, medical accidents that could result in injury to blood vessels can be prevented, and further improvements in safety can be expected.

[0108] FIG. 16 is a schematic diagram showing a photographic lighting device 1601 according to a third modified example of the present invention, and an LED light 1602 and a camera unit 1603 incorporated in the photographic lighting device 1601. The LED lighting 1602 has a visible light LED 1604 and an infrared light LED 1605. The visible light LED 1604 and the infrared light LED 1605 emit light simultaneously and continuously. The camera unit 1603 has a cold mirror 1606 installed at an angle of 45° on the optical axis, a first camera 1007 installed at a position past the cold mirror 1606 on the optical axis, and a second camera 1008 installed at a position perpendicular to the cold mirror 1606 on the optical axis.

[0109] The cold mirror 1606 transmits near-infrared light and reflects visible light, so that the first camera 1007 captures an infrared image and the second camera 1008 captures a visible light image. Even if a hot mirror that transmits visible light and reflects near-infrared light is installed instead of the cold mirror 1606, the first camera 1007 can capture visible light images and the second camera 1008 can capture infrared images. In either case, since visible light images and infrared images can be captured simultaneously from the incoming light, it becomes possible to simultaneously display the visible light image and infrared image on the display unit 1125 and the external monitor 104.

[0110] [Fourth Modification: A multi-viewpoint image capturing device that captures blood vessels in a surgical field 106a of a patient 106 in a time-division manner using an infrared LED 1605] In the third modified example described above, it is necessary to incorporate a cold mirror 1606 or a hot mirror and two cameras 201 in the camera unit 1603. Such special products are not widely available on the market, and manufacturing them from ready-made products would be costly. However, if time resolution is sacrificed, it is possible to capture visible light images and infrared images almost simultaneously without using a camera with such a special structure.

[0111] Fig. 17A is a schematic diagram showing an LED lamp 1602 of an imaging lighting fixture and a camera 201 according to a fourth modified example of the present invention, and Fig. 17B is a time chart showing the emission timing of a visible light LED 1604 and an infrared LED 1605. Fig. 17C is a block diagram showing the software functions of an infrared image acquisition unit 1702 provided in an input / output control unit 1201 of an image lighting processing device 103.

[0112] 17A, camera 201 is a general image pickup device and does not have a special structure like the third modified example. Instead, camera 201 outputs a frame timing pulse. The frame timing pulse is input to light emission timing control unit 1701.

[0113] The light emission timing control unit 1701 synchronizes the light emission timing of the visible light LED 1604 and the infrared LED 1605 with a frame timing pulse included in the video data output by the camera 201, and controls the light emission of the visible light LED 1604 and the infrared LED 1605 at the timing shown in FIG. 17B. That is, the light emission timing control unit 1701 causes only the visible light LED 1604 to emit light at the first frame timing t1711, causes only the infrared LED 1605 to emit light at the second frame timing t1712, and turns off both the visible light LED 1604 and the infrared LED 1605 at the third frame timing t1713. This operation is repeated.

[0114] Then, the infrared LED-on captured video data obtained at the second frame timing and the infrared LED-off captured video data obtained at the third frame timing are input to an infrared image acquisition unit 1702 (see Figure 17C) formed within the input / output control unit 1201 of the image lighting processing device 103. Also, as shown in FIG. 17C, the infrared image acquisition unit 1702 has an infrared LED on shot image buffer 1703 that holds infrared LED on shot image data, an infrared LED off shot image buffer 1704 that holds infrared LED off shot image data, and an adder 1705.

[0115] The infrared LED-on captured video data output from the infrared LED-on captured video buffer 1703 and the infrared LED-off captured video data output from the infrared LED-off captured video buffer 1704 are input to an adder 1705. The adder 1705 subtracts the infrared LED off captured image data from the infrared LED on captured image data for each pixel. This subtracts the infrared LED off captured image data, i.e., noise components due to visible light, contained in the infrared LED on captured image data.

[0116] With the above configuration, although the frame rate is reduced to one-third compared to the third modified example, it is possible to obtain infrared images using a general-purpose camera 201 as is. Therefore, as with the third modified example, it is possible to prevent medical accidents that cause damage to blood vessels during surgery, and it is expected to improve safety.

[0117] Furthermore, by increasing the types of lighting that irradiate light of only a specific wavelength spectrum, rather than just infrared light, and performing the processes shown in Figures 17A, 17B, and 17C, it becomes possible to obtain image data of the specific wavelength spectrum.

[0118] A general personal computer can be used as it is as the image lighting processing device 103. Therefore, it becomes possible to simultaneously display on the display unit 1125 and the external monitor 104 not only the multi-view image of the surgical field 106a by visible light and the multi-view image of blood vessels by infrared light, but also information output by an external device.

[0119] [Fifth Modification: Variation of Fixture 102b] FIG. 18 is a schematic diagram showing a variation of a fixing device 102b according to a fifth modified example of the present invention. The fixing device 102b shown in FIG. 1 is provided to keep the imaging lighting device 102 suspended above the surgical field 106a. In order not to interfere with the operation of the surgeon 107, the fixing device 102b has a structure that allows it to be suspended from the ceiling of the operating room (not shown).

[0120] If we ignore the technical idea of ​​a form that does not interfere with the operation of the surgeon 107, it is also possible to provide legs 1801a and 1801b that extend directly below the photographing lighting device 102 and an extension arm 1802 that extends directly to the side of the photographing lighting device 102, as shown in Figure 18. Furthermore, it is also possible to provide an extension arm 1803 that extends diagonally above the photographing lighting device 102. In this way, various variations are possible for the form of the fixing device for maintaining the photographing lighting device 102 in a state where it is lifted above the surgical field 106a.

[0121] Conventionally, there are three instruments that a surgeon 107 needs in a direct surgery: a shadowless lamp, a camera for photographing the surgical site, and an information display device. These have traditionally existed separately. The multi-viewpoint video capture device 101 according to the present invention aims to provide a surgeon 107 with a wide variety of information essential for surgery in an integrated manner, like an airplane cockpit.

[0122] For example, not only can useful video information be shared in real time with multiple medical professionals, but preoperative simulation information such as preoperative CT images, MRI images, and 3D reconstruction images, as well as navigation information such as appropriate vascular course in key areas using infrared images, can also be displayed in an appropriate layout on the external monitor 104 as the surgery progresses. In addition, vital information of the patient 106, such as blood pressure, pulse, electrocardiogram, and electroencephalogram, acquired in real time, medical safety information such as the surgical procedure and surgical staff, and medical cost information such as the equipment used may be displayed as shared information.

[0123] In recent years, there has been a growing societal demand for medical safety, and it is expected that the multi-viewpoint video imaging device 101 according to the embodiment of the present invention will play a major role in improving the safety of surgery. In contrast to surgeries that have been performed only by direct vision with the naked eye up until now, by displaying various information to the surgeon 107, such as live magnified information, infrared information, preoperative radiographic image information, 3D simulation information, and even medical safety information, it will be possible to significantly improve the safety of surgery.

[0124] In addition, by properly archiving and recording the live expanded information, it can be useful for the education of students and young surgeons, and can also be stored as a record that can withstand the disclosure of medical information.

[0125] Furthermore, while a shadowless lamp is an essential piece of equipment in all operating rooms, the illuminated camera device used in the embodiment of the present invention can serve as a more compact shadowless lamp. As a result, by using the illuminated camera device used in the present invention, large-scale shadowless lamp equipment becomes unnecessary in the operating room, and it may become a core piece of equipment in the approximately 20,000 operating rooms that are said to exist throughout Japan.

[0126] As described above, the multi-viewpoint video imaging device 101 according to the embodiment of the present invention can be said to be a completely new surgery support system, and it can even be said that it has the potential to completely change the landscape of operating rooms around the world.

[0127] As mentioned above, this multi-viewpoint video capture device is intended to be used primarily in surgical procedures on living organisms; however, the device's features of providing a clear lighting environment that is not obstructed by the surgeon's head or body, and unobstructed video capture of the surgical field, can also be effectively applied in fields such as pathological autopsies, forensic autopsies, surgical autopsies, and educational autopsies on cadavers, and even in veterinary surgeries on animals.

[0128] Although an embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and includes other modifications and applications without departing from the gist of the present invention described in the claims.

[0129] For example, the above-described embodiments have described in detail and specifically the configurations of the devices and systems in order to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and furthermore, it is possible to add the configuration of another embodiment to the configuration of one embodiment. Also, it is possible to add, delete, or replace a part of the configuration of each embodiment with the configuration of another embodiment. [Explanation of symbols]

[0130] 101...multiple viewpoint image capture device, 102...camera lighting device, 102a...housing, 102b...fixing device, 103...image lighting processing device, 104...external monitor, 105...operating table, 106...patient, 106a...surgical field, 107...surgeon, 201...camera, 202...lighting, 203...USB hub, 401...handle, 601...camera lighting device, 601a...housing, 601b...fixing device, 602...camera lighting device, 602a...first housing, 602b...first fixing device, 602c...second housing, 602d...second fixing device, 701...camera lighting device, 701a...first housing, 7 01b...first fixed device, 701c...second housing, 701d...second fixed device, 702...photography lighting device, 702a...housing, 702b...fixed device, 801...photography lighting device, 801a...housing, 801b...fixed device, 802...photography lighting device, 802a...housing, 802b...fixed device, 901...photography device, 901a...housing, 1001...lighting device, 1001a...housing, 1007...first camera, 1008...second camera, 1101...camera array unit, 1102...lighting array unit, 1103...network, 1104...microcomputer, 1105...multiplexer , R1106...current limiting resistor, 1107...bus, 1108...CPU, 1109...ROM, 1110...RAM, 1111...serial interface, 1121...bus, 1122...CPU, 1123...ROM, 1124...RAM, 1125...display unit, 1126...operation unit, 1127...non-volatile storage, 1128...serial interface, 1129...NIC, 1201...input / output control unit, 1202...camera calibration processing unit, 1204...presentation viewpoint / gazing point determination processing unit, 1205...image processing unit, 1206...illumination determination processing unit, 1207...gaze Point coordinates, 1401...multiple viewpoint video shooting device, 1402...shooting lighting device, 1501...multiple viewpoint video shooting device, 1502...shooting lighting device, 1601...shooting lighting device, 1602...LED lighting, 1603...camera unit, 1604...visible light LED, 1605...infrared LED, 1606...cold mirror, 1701...light emission timing control unit, 1702...infrared image acquisition unit, 1703...infrared LED on shooting image buffer, 1704...infrared LED off shooting image buffer, 1705...adder, 1801a...legs, 1802, 1803...extension arms

Claims

1. A photographing device in which a plurality of cameras are attached to a ring-shaped circular or arc-shaped finite length curved housing so that the cameras are directed toward a subject on which an operator is to perform a work; A fixing device for placing the photographing device at a position between the top of the head of the surgeon and the subject; a camera calibration processing unit that estimates camera parameters including positions, attitudes, and focal lengths of the plurality of cameras attached to the photographing tool based on photographing information of the subject photographed by the plurality of cameras; a presentation viewpoint gaze point determination processing unit that detects a gaze point, which is an area of ​​the subject that the surgeon focuses on, in the multi-viewpoint image data, estimates three-dimensional coordinate information of the gaze point by referring to the camera parameters, and selects one or more cameras that capture images from a direction suitable for observation by the surgeon or an observer outside the surgical field based on the three-dimensional coordinate information; A multi-viewpoint video shooting device comprising:

2. (delete)

3. The photographic equipment includes a plurality of lights; A plurality of lights are disposed between the plurality of cameras attached to the housing; The multi-viewpoint video shooting device according to claim 1 .

Images