Information processing device and method

Abstract

To accurately display the position and shape of the object being treated during surgery. [Solution] The information processing device according to the embodiment comprises a generation unit, an acquisition unit, and a display control unit. The generation unit extracts processing targets from preoperative image data depicting the processing targets of the patient, and generates path field image data showing the processing targets in the surgical route planned before surgery from among the extracted processing targets. The acquisition unit acquires surgical field image data of multiple time-series frames that show the surgical field and depict the processing targets during the patient's surgery. The display control unit extracts a reference processing target from one frame of surgical field image data, performs at least one of deformation and movement on the processing target shown by the path field image data to match the extracted processing target, and displays the deformed and moved processing target superimposed on the one frame of surgical field image data on the display unit.

Description

Technical Field

[0001] The embodiments disclosed in this specification and the drawings relate to an information processing apparatus and method.

Background Art

[0002] There is an application that extracts cerebral blood vessels from medical images such as CT (Computed Tomography) images and separates and displays the arteries and veins of the extracted cerebral blood vessels in different modes.

[0003] Also, in a craniotomy, the surgeon grasps the anatomical information of the patient's brain from preoperative images such as CT images and the displays of the arteries and veins obtained by the above application, and formulates a surgical plan. During the operation, the surgeon proceeds with the procedure based on the surgical plan.

[0004] However, when the surgeon actually starts a craniotomy, the shapes and positions of the arteries and veins of the actual cerebral blood vessels may differ from the shapes and positions of the arteries and veins of the cerebral blood vessels displayed by the above application. This is because, for example, due to the craniotomy, the pressure inside the brain changes and various structures inside the brain deform. Therefore, the shapes and positions of the arteries and veins of the cerebral blood vessels displayed preoperatively by the above application may not be useful information for the surgeon during the actual craniotomy.

[0005] In addition, regarding other applications that extract other processing targets from medical images and display the extracted processing targets preoperatively, similar problems may occur during actual surgeries.

[0006] Therefore, it is desirable to accurately display the position and shape of the processing target during the operation.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

[0008] One of the problems that the embodiments disclosed in this specification and drawings aim to solve is to accurately display the position and shape of the object being processed during the procedure. However, the problems that the embodiments disclosed in this specification and drawings aim to solve are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described later can also be positioned as other problems. [Means for solving the problem]

[0009] The information processing device according to this embodiment comprises a generation unit, an acquisition unit, and a display control unit. The generation unit extracts processing targets from preoperative image data depicting the processing targets of the patient, and generates path field image data showing the processing targets in the surgical route planned preoperatively from among the extracted processing targets. The acquisition unit acquires surgical field image data of multiple time-series frames showing the surgical field and the processing targets during the patient's surgery. The display control unit extracts a reference processing target from one frame of surgical field image data, performs at least one of deformation and movement on the processing target shown by the path field image data to match the extracted processing target, and displays the deformed and moved processing target superimposed on the one frame of surgical field image data on the display unit. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 shows an example of the configuration of a surgical field image display system according to the first embodiment. [Figure 2] Figure 2 is a diagram illustrating an example of processing performed by the information processing device according to the first embodiment. [Figure 3] Figure 3 shows the resulting image data according to the first embodiment. [Figure 4]Figure 4 shows a single frame of surgical field image data according to the first embodiment. [Figure 5] Figure 5 shows an example of display image data according to the first embodiment. [Figure 6] Figure 6 is a diagram illustrating an example of processing performed by the information processing device according to the second embodiment. [Figure 7] Figure 7 is a diagram illustrating path field image data according to the second embodiment. [Figure 8] Figure 8 is an enlarged view of a portion of the path visual field image data in the second embodiment, in which at least one of deformation and / or movement has been performed on the cerebral blood vessels. [Figure 9] Figure 9 is a diagram illustrating an example of the processing performed by the surgical field image display system according to Modification 1. [Figure 10] Figure 10 is a diagram illustrating an example of the processing performed by the surgical field image display system according to Modification 2. [Figure 11] Figure 11 is a diagram illustrating an example of the processing performed by the surgical field image display system according to Modification 3. [Figure 12] Figure 12 is a diagram illustrating an example of the processing performed by the surgical field image display system according to Modification 4. [Figure 13] Figure 13 is a diagram illustrating an example of the processing performed by the surgical field image display system according to Modification 5. [Figure 14] Figure 14 is a diagram illustrating an example of the processing performed by the surgical field image display system according to Modification 6. [Modes for carrying out the invention]

[0011] Hereinafter, embodiments and modifications of an information processing apparatus and method will be described in detail while referring to the drawings. Note that the information processing apparatus and method according to the present application are not limited to the embodiments and modifications shown below. Also, embodiments can be combined with other embodiments, modifications, or prior art as long as there is no contradiction in the processing content. Similarly, modifications can be combined with embodiments, other modifications, or prior art as long as there is no contradiction in the processing content.

[0012] (First Embodiment) FIG. 1 is a diagram showing an example of the configuration of an intraoperative image display system 1 according to an embodiment. The intraoperative image display system 1 is, for example, a system installed in a medical facility such as a hospital or a clinic and assisting a user such as a surgeon in performing a surgery. Therefore, the intraoperative image display system 1 is also referred to as, for example, a surgical assistance system.

[0013] As shown in FIG. 1, the intraoperative image display system 1 according to the present embodiment includes a medical device 10, an intraoperative camera 20, and an information processing apparatus 30. Here, the medical device 10 and the information processing apparatus 30 are connected to each other so as to be communicable, either wired or wirelessly, as indicated by the double-headed arrows in FIG. 1. Similarly, the intraoperative camera 20 and the information processing apparatus 30 are connected to each other so as to be communicable.

[0014] The medical device 10 is a device used in surgery. The medical device 10 is, for example, inserted into a patient's body during surgery. The medical device 10 is provided with a small camera, which captures the inside of the body during surgery and transmits the image data obtained by the capture to the information processing apparatus 30 in real time. That is, the medical device 10 transmits image data (moving image data) of a plurality of frames in time series to the information processing apparatus 30. Also, each time the medical device 10 generates one frame of image data by capturing the inside of the patient's body, the generated one frame of image data is transmitted to the information processing apparatus 30. Hereinafter, in the present embodiment, the case where a surgeon performs a craniotomy on a patient using the medical device 10 will be described.

[0015] The surgical field camera 20 is provided at a position where it is possible to photograph the surgical field of a patient during a craniotomy (in this embodiment, the part where the protective tissue such as the skull covering the brain is opened), photographs the surgical field during the craniotomy, and transmits the image data obtained by the photographing to the information processing device 30 in real time. That is, the surgical field camera 20 transmits image data of a plurality of frames in time series to the information processing device 30. Also, each time the surgical field camera 20 generates image data of one frame, the generated image data of one frame is transmitted to the information processing device 30.

[0016] The information processing device 30 is realized by a computer device such as a server or a workstation. Also, in the following description, the case where one information processing device 30 executes various processes described below will be described, but a plurality of information processing devices 30 may execute the various processes described below in a distributed manner.

[0017] As shown in FIG. 1, the information processing device 30 includes a communication interface 31, an input interface 32, a display 33, a memory 34, and a processing circuit 35.

[0018] The communication interface 31 controls the transmission and communication of various information and various data transmitted and received between the information processing device 30 and other devices (the medical device 10 and the surgical field camera 20) connected to the information processing device 30 by wire or wirelessly. The communication interface 31 is connected to the processing circuit 35. The communication interface 31 receives information and data transmitted by other devices. In this case, the communication interface 31 transmits the received information and data to the processing circuit 35. Also, the communication interface 31 receives information and data transmitted by the processing circuit 35. In this case, the communication interface 31 transmits the received information and data to other devices. For example, the communication interface 31 is realized by a network card, a network adapter, a NIC (Network Interface Controller), or the like.

[0019] The input interface 32 receives various instructions and input operations for various information from users, including surgeons and other physicians. The input interface 32 is connected to the processing circuit 35. The input interface 32 converts the operations received from the user into electrical signals and transmits them to the processing circuit 35. For example, the input interface 32 can be implemented by a trackball, switch buttons, mouse, keyboard, touchpad that accepts operations by touching the operating surface, touchscreen that integrates a display screen and a touchpad, a non-contact input interface using an optical sensor, and an audio input interface. In this specification, the input interface 32 is not limited to those equipped with physical operating components such as a mouse or keyboard. For example, an electrical signal processing circuit that receives electrical signals corresponding to input operations from an external input device provided separately from the information processing device 30 and transmits these electrical signals to the processing circuit 35 is also included as an example of the input interface 32.

[0020] The display 33 outputs various information and data. The display 33 is connected to the processing circuit 35. The display 33 is controlled by the processing circuit 35 and displays the various information and data transmitted by the processing circuit 35. For example, the display 33 may be implemented by various displays such as a liquid crystal display, a CRT (Cathode Ray Tube) display, or a touch panel. The display 33 is an example of a display unit.

[0021] Memory 34 stores various data and programs. Memory 34 is connected to processing circuit 35. Memory 34 stores data transmitted by processing circuit 35 under the control of processing circuit 35. Data stored in memory 34 is read out by processing circuit 35. For example, memory 34 can be implemented using semiconductor memory elements such as RAM (Random Access Memory) or flash memory, or a hard disk or optical disc. Memory 34 is, for example, an example of a storage unit.

[0022] The processing circuit 35 controls the entire information processing device 30. For example, the processing circuit 35 executes various processes in response to instructions received from the user via the input interface 32.

[0023] As shown in Figure 1, the processing circuit 35 includes an arteriovenous separation image generation function 351, a pathway field of view generation function 352, a blood vessel extraction and tracking function 353, a positioning and blood vessel projection function 354, and a display control function 355.

[0024] The processing circuit 35 is implemented, for example, by a processor. In this case, each of the processing functions described above is stored in memory 34 in the form of a program (information processing program) that can be executed by the computer. The processing circuit 35 then reads each program stored in memory 34 and executes each program, thereby realizing each processing function corresponding to each program. In other words, the processing circuit 35, with each program read, has the processing functions shown in Figure 1.

[0025] The processing circuit 35 may be composed of a combination of multiple independent processors, with each processor executing its own program to realize each processing function. Furthermore, each processing function of the processing circuit 35 may be implemented by distributing or integrating it across one or more processing circuits. Also, each processing function of the processing circuit 35 may be implemented by a mixture of hardware such as circuits and software. While this example describes a case where each program corresponding to each processing function is stored in a single memory 34, each program may be distributed and stored across multiple memory circuits. For example, each program corresponding to each processing function may be distributed and stored across multiple memory circuits, and the processing circuit 35 may read and execute each program from each memory circuit.

[0026] As described below, the surgical field image display system 1 according to this embodiment is configured to accurately display the position and shape of arteries and veins of the cerebral blood vessels during surgery.

[0027] Figure 2 is a diagram illustrating an example of processing performed by the information processing device 30 according to the first embodiment. Figure 2 shows an example of the processing flow, and each of the processes in steps S101 to S103 shown on the right side of Figure 2 is performed preoperatively (before craniotomy). In addition, each of the processes in steps S201 to S204 shown on the left side of Figure 2 is performed intraoperatively (during craniotomy).

[0028] For example, before surgery, the arteriovenous isolation image generation function 351 first acquires preoperative images of the patient obtained by taking images in advance using medical imaging diagnostic equipment such as an X-ray CT scanner or an MRI (Magnetic Resonance Imaging) scanner (step S101). The preoperative images referred to here are medical images such as CT images or MR images in which the cerebral blood vessels are depicted, which are the target of processing for the patient.

[0029] The arteriovenous separation image generation function 351 then executes application 401 to extract the patient's cerebral blood vessels from the preoperative image and generates image data (result image data) 41 in which the arteries and veins of the cerebral blood vessels are depicted in different ways (step S102). Figure 3 shows the result image data 41 according to the first embodiment. As shown in Figure 3, the arteries 41a and veins 41b are depicted in different ways in the result image data 41, thus separating the arteries 41a and veins 41b in the result image data 41. For this reason, the process in step S102 is also called "arteriovenous separation". The following explanation will use the case where the result image data 41 shown in Figure 3 is generated in step S102 as an example. In step S102, the display control function 355 displays the result image based on the result image data 41 on the display 33.

[0030] Application 401 is an application with two functions. The first function is to extract cerebral blood vessels from an image, generate resulting image data in which arteries and veins of the cerebral blood vessels are depicted in different ways, and display a resulting image based on the generated resulting image data on the display 33. The second function is to generate path field image data, which shows the cerebral blood vessels present in the surgical path, as viewed by the surgeon from the direction in which the medical device 10 was inserted, when the surgeon inserts the medical device 10 along the surgical path, as extracted by the first function. In step S102, the first function described above is executed.

[0031] Then, the surgeon performing the craniotomy confirms the position and shape of the arteries 41a and veins 41b from the result images based on the result image data 41 displayed on the display 33, grasps the anatomical information of the patient's brain, and plans the craniotomy. In the surgical plan, the surgeon also plans the route (surgical route) through which the medical device 10 will be inserted during the craniotomy. This surgical route is, for example, the route from the opening of the tissue protecting the brain, avoiding the arteries 41a and veins 41b of the brain, to the site of the brain lesion (e.g., a tumor), through which the medical device 10 reaches.

[0032] The surgeon then operates the input interface 32 to input the planned surgical route into the processing circuit 35 of the information processing device 30.

[0033] Then, the route field generation function 352 executes the application 401 using the input surgical route to perform the second function described above (step S103). That is, in step S103, the route field generation function 352 generates route field image data and stores the generated route field image data in memory 34. The route field image data stored in memory 34 is used for intraoperative processing described later.

[0034] As described above, in the preoperative processing steps S101 to S103, the arteriovenous separation image generation function 351 extracts the target areas (cerebral blood vessels) from preoperative image data depicting the patient's target areas, and the route field generation function 352 generates route field image data showing the target areas in the surgical route planned preoperatively from among the extracted target areas. The arteriovenous separation image generation function 351 and the route field generation function 352 are, for example, examples of generation units.

[0035] Next, we will explain each of the steps S201 to S204 during the surgery.

[0036] The vascular extraction and tracking function 353 acquires multiple time-series image data (dynamic image data) transmitted in real time from the medical device 10 or the surgical field camera 20 as surgical field image data (step S201). Figure 4 is a diagram showing one frame of surgical field image data 42 according to the first embodiment. As shown in Figure 4, the surgical field is depicted in the surgical field image data 42. Thus, in step S201, the vascular extraction and tracking function 353 acquires multiple time-series frames of surgical field image data that show the surgical field and depict the object to be processed during the patient's surgery. In the first embodiment, the vascular extraction and tracking function 353 that performs the processing in step S201 is, for example, an example of an acquisition unit.

[0037] Then, in step S202, the blood vessel extraction and tracking function 353 performs the following processing. For example, in step S202, the blood vessel extraction and tracking function 353 first takes one frame of surgical field image data from among multiple frames of time-series surgical field image data transmitted in real time as the reference frame's surgical field image data, and extracts cerebral blood vessels, including arteries and veins, from the reference frame's surgical field image data as reference cerebral blood vessels (reference cerebral blood vessels). Then, in step S202, the blood vessel extraction and tracking function 353 obtains cerebral blood vessels by tracking the reference cerebral blood vessels in the surgical field image data of each frame after the reference frame.

[0038] For example, among surgical field image data from multiple frames, the surgical field image data from the first frame may be used as the reference frame's surgical field image data. In this case, the surgical field image data from the second frame onward will be the surgical field image data from frames later than the aforementioned reference frame.

[0039] Furthermore, the blood vessel extraction and tracking function 353 performs the above-described processing in real time. For example, the blood vessel extraction and tracking function 353 performs the above-described processing each time it acquires one frame of surgical field image data from the medical device 10 or the surgical field camera 20.

[0040] Then, the alignment and vascular projection function 354 performs the following processing in step S203. First, the processing in step S203 for the surgical field image data of the reference frame will be explained. For example, in step S203, the alignment and vascular projection function 354 first aligns the reference cerebral vessels extracted from the surgical field image data of the reference frame with the cerebral vessels in the preoperatively planned surgical route indicated by the route field image data stored in memory 34. During this alignment, at least one of deformation and movement of the cerebral vessels indicated by the route field image data is performed without deforming or moving the reference cerebral vessels. Note that movement includes translation and rotation. Then, in step S203, the alignment and vascular projection function 354 generates display image data by superimposing (projecting) the cerebral vessels that have been deformed and moved onto the surgical field image data of the reference frame.

[0041] Next, the processing in step S203 for surgical field image data of frames after the reference frame will be explained. In outline, the alignment / vascular projection function 354 uses the cerebral blood vessels obtained in the surgical field image data of frames after the reference frame to perform the same processing as the processing performed using the reference cerebral blood vessels described above. Specifically, for example, in step S203, the alignment / vascular projection function 354 first aligns the cerebral blood vessels obtained from the surgical field image data with the cerebral blood vessels indicated by the path field image data stored in memory 34. During this alignment, the cerebral blood vessels obtained from the surgical field image data are not deformed or moved, while at least one of the deformation and movement of the cerebral blood vessels indicated by the path field image data is performed. Then, in step S203, the alignment / vascular projection function 354 generates display image data by superimposing the deformed and moved cerebral blood vessels onto the surgical field image data.

[0042] The alignment and vascular projection function 354 performs the above-described processing in real time and generates image data for display in real time. For example, the alignment and vascular projection function 354 performs the above-described processing each time that cerebral blood vessels are extracted or obtained from a single frame of surgical field image data by the vascular extraction and tracking function 353 and generates image data for display.

[0043] Figure 5 shows an example of display image data 44 according to the first embodiment. As shown in Figure 5, the display image data 44 for one frame is image data in which arteries 43a and veins 43b of the cerebral blood vessels are superimposed on surgical field image data 42.

[0044] The display control function 355 then displays an image based on the display image data 44 on the display 33 in real time. For example, each time a frame of display image data 44 is generated by the alignment / vascular projection function 354, the display control function 355 displays an image based on the generated image data 44 on the display 33.

[0045] As described above, in step S202, the blood vessel extraction and tracking function 353 extracts or obtains a processing target from one frame of surgical field image data. In step S203, the alignment and blood vessel projection function 354 performs at least one of deformation and movement on the processing target indicated by the path field image data in accordance with the extracted or obtained processing target, and generates display image data 44 by superimposing the processed processing target, which has undergone at least one of deformation and movement, onto the one frame of surgical field image data. In step S204, the display control function 355 displays an image based on the display image data 44 on the display 33. The blood vessel extraction and tracking function 353 that performs the processing in step S202, the alignment and blood vessel projection function 354 that performs the processing in step S203, and the display control function 355 that performs the processing in step S204 are, for example, examples of display control units.

[0046] The surgical field image display system 1 according to the first embodiment has been described above. In craniotomy, intracellular pressure changes, and the shapes of various organs change significantly. This is true not only for craniotomy but also for laparotomy and thoracotomy. As a result, the shape and position of the patient's cerebral arteries and veins obtained preoperatively may change significantly during surgery. However, the surgical field image display system 1 according to this embodiment deforms and moves at least one of the cerebral arteries and veins obtained preoperatively to match the shape of the cerebral arteries and veins during craniotomy. The surgical field image display system 1 then overlays and displays the deformed and moved cerebral arteries and veins on a moving image based on moving image data from the medical device 10 or surgical field camera 20. This makes it possible to accurately display the position and shape of the cerebral arteries and veins in accordance with their actual position and shape during surgery. As a result, it is possible to effectively support the surgeon's operation.

[0047] Furthermore, if the shape and position of the arteries and veins of the patient's cerebral blood vessels, obtained preoperatively, change significantly during surgery, the progress of the surgery will vary greatly depending on the surgeon's surgical experience and spatial recognition ability. On the other hand, according to this embodiment, the position and shape of the arteries and veins of the cerebral blood vessels can be displayed accurately in accordance with their actual position and shape during surgery, allowing the surgeon to proceed with the surgery appropriately regardless of their surgical experience or spatial recognition ability. For this reason, according to this embodiment, even young surgeons with relatively little surgical experience can proceed with the surgery appropriately. Therefore, the surgical field image display system 1 according to this embodiment may be used as a means of training such young surgeons to gain surgical experience.

[0048] (Second embodiment) Next, the surgical field image display system 1 according to the second embodiment will be described. In the description of the second embodiment, the configurations that differ from the first embodiment will be mainly described, and the description of configurations similar to the first embodiment may be omitted. Figure 6 is a diagram illustrating an example of processing performed by the information processing device 30 according to the second embodiment.

[0049] The application 402 according to the second embodiment shown in Figure 6 differs from the application 401 according to the first embodiment shown in Figure 2 in that, in addition to the two functions described above, it has two further functions: a third function and a fourth function. The third function of application 402 is to perform the same processing as in step S202 described in the first embodiment. The fourth function of application 402 is to perform the processing in step S205. The processing in step S205 will be described later.

[0050] Figure 6 shows an example of the processing flow. Steps S101 to S103, shown on the right side of Figure 6, are the same as steps S101 to S103 in the first embodiment and are performed preoperatively (before craniotomy). Steps S201, S202, S205, and S206, shown on the left side of Figure 6, are performed intraoperatively (during craniotomy).

[0051] The processes in steps S101 to S103 and S201 of the second embodiment are the same as those in steps S101 to S103 and S201 of the first embodiment. Therefore, a description of the processes in steps S101 to S103 and S201 of the second embodiment will be omitted. In the second embodiment, the blood vessel extraction and tracking function 353 that performs the process in step S201 is, for example, an example of the first acquisition unit.

[0052] As shown in Figure 6, during surgery, following step S201, in step S202, the vascular extraction and tracking function 353 executes the same process as in step S202 according to the first embodiment by executing application 402. For example, in step S202, the vascular extraction and tracking function 353 extracts cerebral blood vessels, including arteries and veins, from the surgical field image data of the reference frame as reference cerebral blood vessels (reference cerebral blood vessels). Then, in step S202, the vascular extraction and tracking function 353 obtains cerebral blood vessels by tracking the reference cerebral blood vessels in the surgical field image data of each frame after the reference frame.

[0053] Then, following step S202, the alignment and vascular projection function 354 performs the following processing in step S205. For example, in step S205, the alignment and vascular projection function 354 aligns the reference cerebral vessels extracted from the surgical field image data of the reference frame with the cerebral vessels obtained from the surgical field image data of other frames other than the reference frame.

[0054] Here, among the surgical field image data of multiple frames, the surgical field image data of the first frame may be used as the surgical field image data of the reference frame. In this case, the surgical field image data of the second frame and subsequent frames will be the surgical field image data of frames other than the reference frame mentioned above. Hereafter, the surgical field image data of frames other than the reference frame will be referred to as the surgical field image data of the Nth (N is an integer of 2 or more)th frame.

[0055] In step S205, during the alignment process, the alignment / vascular projection function 354 performs at least one of the deformation and / or movement of the cerebral vessels obtained from the surgical field image data of the Nth frame, without deforming or moving the reference cerebral vessels. The alignment / vascular projection function 354 then obtains at least one of the expansion / contraction rate and the amount of movement of the cerebral vessels that have undergone at least one of the deformation and / or movement.

[0056] The alignment and vascular projection function 354 performs the above-described processing in real time to acquire at least one of the expansion / contraction rate and the amount of movement of cerebral blood vessels that have undergone deformation and / or movement in real time. For example, each time cerebral blood vessels are obtained from the surgical field image data of the Nth frame by the vascular extraction and tracking function 353, the alignment and vascular projection function 354 performs the above-described processing to acquire at least one of the expansion / contraction rate and the amount of movement of cerebral blood vessels that have undergone deformation and / or movement. In this way, the alignment and vascular projection function 354 acquires at least one of the expansion / contraction rate and the amount of movement of cerebral blood vessels from the surgical field image data of the Nth frame. The alignment and vascular projection function 354 according to the second embodiment is, for example, an example of a second acquisition unit.

[0057] Then, in step S206, the display control function 355 executes the process described below. For example, the display control function 355 acquires the path field image data stored in the memory 34. Figure 7 is a diagram illustrating the path field image data according to the second embodiment. The path field image data is image data that shows the cerebral blood vessels present in the surgical path, among the cerebral blood vessels included in the result image data 41 generated by the first function of the application 402, as seen by the surgeon from the direction in which the medical device 10 was inserted along the surgical path (the direction indicated by the arrow in Figure 7).

[0058] The display control function 355 then performs deformation and / or movement on the cerebral blood vessels shown by the path field image data, using at least one of the scaling ratio and displacement amount of the cerebral blood vessels obtained from the surgical field image data of the Nth frame. Figure 8 is an enlarged view of a portion 47 of the path field image data in the second embodiment, in which deformation and / or movement have been performed on the cerebral blood vessels. Figure 8 shows a portion 47 of the cerebral blood vessels that have been deformed and / or moved.

[0059] The display control function 355 then causes the display 33 to display a path field image on the display 33 for each frame from the second frame onward, based on path field image data in which at least one of deformation and movement has been performed on the cerebral blood vessels.

[0060] The display control function 355 displays a path field image on the display 33 in real time based on the path field image data. For example, each time that the alignment / vascular projection function 354 acquires at least one of the enlargement / reduction rate and displacement amount of the cerebral blood vessels from the Nth frame of surgical field image data, the display control function 355 performs the above-described process and displays a path field image on the display 33 based on the path field image data in which at least one of deformation and displacement has been applied to the cerebral blood vessels.

[0061] As described above, in step S202, the blood vessel extraction and tracking function 353 extracts or obtains a processing target from one frame of surgical field image data. In step S205, the alignment and blood vessel projection function 354 obtains at least one of the scaling ratio and movement amount of the processing target between frames from the surgical field image data. In step S206, the display control function 355 performs at least one of deformation and movement on the processing target indicated by the path field image data based on at least one of the scaling ratio and movement amount of the processing target, and displays the path field image data on the display 33 after at least one of the deformation and movement has been performed on the processing target. The display control function 355 that performs the processing in step S206 is, for example, an example of a display control unit.

[0062] The surgical field image display system 1 according to the second embodiment has been described above. The surgical field image display system 1 according to the second embodiment performs at least one of the following on the arteries and veins of the patient's cerebral blood vessels obtained preoperatively: deformation and / or movement to match the shape of the arteries and veins of the cerebral blood vessels during craniotomy. The surgical field image display system 1 then displays the arteries and veins of the cerebral blood vessels that have been deformed and / or moved. This makes it possible to accurately display the position and shape of the arteries and veins of the cerebral blood vessels in accordance with their actual position and shape during surgery. As a result, it is possible to effectively support the surgeon during the operation.

[0063] Furthermore, according to the second embodiment, similar to the first embodiment, the surgeon can be allowed to perform the surgery appropriately regardless of the surgeon's surgical experience or spatial perception ability, and even a relatively young surgeon with little surgical experience can perform the surgery appropriately. In addition, the surgical field image display system 1 according to the second embodiment may be used as an educational tool to help young surgeons gain surgical experience, similar to the surgical field image display system 1 according to the first embodiment.

[0064] (Various variations) Next, various modifications will be described. In the following descriptions of the various modifications, we will mainly explain the differences from the embodiments described above, and descriptions of similar configurations may be omitted.

[0065] (Variation 1) Figure 9 is a diagram illustrating an example of the processing performed by the surgical field image display system 1 according to Modification 1. In Modification 1, the display control function 355 calculates the position of the medical device 10 based on the surgical field image data each time a frame of surgical field image data is acquired by the blood vessel extraction and tracking function 353. Then, each time the display control function 355 calculates the position of the medical device 10, it displays an image on the display 33, as shown in Figure 9, in which the medical device 10 is superimposed at the calculated position on the CT image 51 depicting the patient's lesion site (e.g., tumor) 50. This displays the distance between the target lesion site 50 and the medical device 10, that is, the depth of penetration of the medical device 10 relative to the target, so that the surgeon can easily understand how far the medical device 10 will reach the target during surgery.

[0066] (Modification 2) Figure 10 is a diagram illustrating an example of the processing performed by the surgical field image display system 1 according to Modification 2. In Modification 2, the display control function 355 displays the surgical field image on the display 33 each time a frame of surgical field image data is acquired by the blood vessel extraction and tracking function 353, based on the surgical field image data.

[0067] The display control function 355 then calculates the position and shape of the medical device 10 based on the surgical field image data each time a frame of surgical field image data is acquired by the blood vessel extraction and tracking function 353.

[0068] The display control function 355 then displays an image on the display 33 in which the medical device 10 is superimposed on the patient model 52, obtained by placing the medical device 10, which has a calculated shape, at a calculated position on the patient model 52, as shown in Figure 10. This allows the surgeon to easily understand which part of the patient's body the surgical field image displayed on the display 33 represents.

[0069] (Variation 3) Figure 11 is a diagram illustrating an example of the processing performed by the surgical field image display system 1 according to Modification 3. In Modification 3, the display control function 355 displays an image on the display 33 in real time based on image data 44 in which arteries 43a and veins 43b of cerebral blood vessels are superimposed on surgical field image data 42, similar to the first embodiment.

[0070] Here, as shown in Figure 11, a specific vein 43c among the multiple veins 43b is assumed to be a relatively important blood vessel and one that should not be damaged. In this case, the display control function 355 calculates the position of the medical device 10 based on the surgical field image data each time a frame of surgical field image data is acquired by the blood vessel extraction and tracking function 353. Then, each time the position of the medical device 10 is calculated, the display control function 355 determines whether the calculated position of the medical device 10 is within a predetermined range from the position of the specific vein 43c.

[0071] If the location of the medical device 10 is calculated to be within a predetermined range from the location of a specific vein 43c, the display control function 355 displays, for example, a warning message "A medical device is approaching an important blood vessel." on the display 33. Alternatively, if the information processing device 30 is equipped with a speaker, and the location of the medical device 10 is calculated to be within a predetermined range from the location of a specific vein 43c, the display control function 355 may output a warning sound to the speaker. In other words, in Modification 3, the display 33 or speaker outputs a warning if the medical device 10 is located within a predetermined range of a specific processing target among the processing targets. The display 33 and speaker are, for example, examples of output units.

[0072] (Modification 4) Figure 12 is a diagram illustrating an example of processing performed by the surgical field image display system 1 according to Modification 4. In the first embodiment described above, the case in which application 401 extracts cerebral blood vessels (cerebral arteries and veins) as the processing target was described, but in Modification 4, application 401 may extract cranial nerves 54 as the processing target instead of cerebral blood vessels. In Modification 4, the surgical field image display system 1 may use cranial nerves 54 instead of cerebral blood vessels and perform the same processing as in the first embodiment described above. As a result, the display 33 displays in real time an image based on image data 55 in which cranial nerves 54 are superimposed on the surgical field image data 42 shown in Figure 12, instead of the image based on image data 44 shown in Figure 5.

[0073] (Variation 5) Figure 13 is a diagram illustrating an example of the processing performed by the surgical field image display system 1 according to Modification 5. In Modification 5, the application 401 may extract multiple parenchyma (brain parenchyma) 57a, 57b as the processing target instead of multiple cerebral blood vessels. Then, in Modification 5, the surgical field image display system 1 may use multiple parenchyma 57a, 57b instead of multiple cerebral blood vessels and perform the same processing as in the first embodiment described above. As a result, as shown in Figure 13, the display 33 displays in real time an image based on image data 58 in which multiple parenchyma 57a, 57b with different display modes are superimposed on the surgical field image data 42.

[0074] (Experimental variation 6) Figure 14 is a diagram illustrating an example of the processing performed by the surgical field image display system 1 according to Modification 6. Modification 6 differs from the application 401 in the first embodiment in that the application 401 further calculates the direction of blood flow for each blood vessel (artery and vein). In Modification 6, as shown in Figure 14, the display 33 displays an image based on image data 60 in which arteries 43a and veins 43b of different configurations are superimposed on surgical field image data 42, an arrow 43a1 indicating the direction of blood flow in artery 43a is superimposed corresponding to artery 43a, and an arrow 43b1 indicating the direction of blood flow in vein 43b is superimposed corresponding to vein 43b, in real time.

[0075] (Other variations) Furthermore, application 401 may calculate the blood flow rate for each blood vessel. In this case, an image in which the blood flow rate for each blood vessel is superimposed numerically on the image shown in Figure 5 will be displayed on the display 33 in real time. Additionally, application 401 may extract multiple vascular supply regions. In this case, an image in which multiple vascular supply regions with different display modes are superimposed on the image shown in Figure 5 will be displayed on the display 33 in real time.

[0076] Furthermore, although the first embodiment described the case of performing craniotomy on a patient, the surgical field image display system 1 can also be used in the case of thoracotomy and laparotomy. For example, if the target of processing is the liver in laparotomy, an image based on image data in which multiple liver regions with different display modes are superimposed on surgical field image data 42 depicting the patient's liver will be displayed on the display 33 in real time. Also, for example, if the target of processing is the lungs in thoracotomy, an image based on image data in which multiple lung lobes with different display modes are superimposed on surgical field image data 42 depicting the patient's lungs will be displayed on the display 33 in real time.

[0077] In the description of the embodiments above, the term "processor" refers to circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). Here, instead of storing the program in memory 34, the processor may be configured to directly incorporate the program into its circuitry. In this case, the processor realizes its function by reading and executing the program incorporated into the circuitry. Furthermore, each processor in each embodiment is not limited to being configured as a single circuit; multiple independent circuits may be combined to form a single processor, and its function may be realized in this way.

[0078] Here, the program executed by the processor is provided pre-installed in ROM (Read Only Memory) or memory circuits. Alternatively, this program may be provided as a file in an installable or executable format on a computer-readable, non-transient storage medium such as a CD (Compact Disk)-ROM, FD (Flexible Disk), CD-R (Recordable), or DVD (Digital Versatile Disk). Furthermore, this program may be stored on a computer connected to a network such as the Internet and provided or distributed by downloading it via the network. For example, this program consists of modules containing the processing functions described above. In actual hardware, the CPU reads the program from a storage medium such as ROM and executes it, loading each module onto the main memory and generating it in the main memory.

[0079] According to at least one embodiment or modification described above, the position and shape of the object to be processed can be displayed with high accuracy during the procedure.

[0080] While several embodiments have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims and their equivalents. [Explanation of Symbols]

[0081] 30 Information Processing Devices 351 Arteriovenous separation image generation function 352 Pathway Visual Field Generation Function 353. Blood vessel extraction and tracking function 354 Alignment and blood vessel projection function 355 Display control function

Claims

1. A generation unit extracts the target of processing from preoperative image data depicting the target of processing for the patient, and generates path field image data showing the target of processing in the surgical route planned preoperatively from among the extracted target of processing. During surgery on the aforementioned patient, an acquisition unit is provided that displays the surgical field and acquires surgical field image data of multiple frames in a time series in which the object to be processed is depicted. A display control unit that extracts a reference processing target from one frame of surgical field image data, performs at least one of deformation and movement on the processing target indicated by the path field image data to match the extracted processing target, and displays the processed target, which has undergone at least one of the deformation and movement, superimposed on the one frame of surgical field image data on the display unit. An information processing device equipped with the following features.

2. The display control unit further performs at least one of the deformation and movement of the processing target indicated by the path field image data to match the processing target obtained by tracking the reference processing target in the surgical field image data of another frame, and displays the processing target, which has undergone at least one of the deformation and movement, superimposed on the surgical field image data of the other frame on the display unit. The information processing apparatus according to claim 1.

3. A generation unit extracts the target of processing from preoperative image data depicting the target of processing for the patient, and generates path field image data showing the target of processing in the surgical route planned preoperatively from among the extracted target of processing. During surgery on the aforementioned patient, a first acquisition unit acquires surgical field image data of multiple time-series frames showing the surgical field and depicting the target of processing, A second acquisition unit that acquires at least one of the scaling ratio and movement amount of the processing target between frames from the surgical field image data, A display control unit that performs at least one of the following on the processing target, the scaling ratio and the amount of movement of the processing target, on the processing target indicated by the path field image data, and displays the path field image data on the display unit after at least one of the transformation and movement of the processing target; An information processing device equipped with the following features.

4. The display control unit, From the surgical field image data, the position of the medical device used in the surgery is calculated. The information processing apparatus according to claim 1 or 3, which displays on the display unit an image in which the medical device is superimposed on the medical image in which the lesion site of the patient is depicted, in which the medical device is superimposed on the medical device.

5. The display control unit, From the surgical field image data, the position and shape of the medical device used in the surgery are calculated. The information processing apparatus according to claim 1 or 3, which displays on the display unit an image in which the medical device is superimposed on the patient model, obtained by placing the medical device of a calculated shape at the calculated position on the patient model.

6. The information processing apparatus according to claim 1 or 3, further comprising an output unit that outputs a warning when a medical device is located within a predetermined range of a specific processing target among the processing targets.

7. The processing targets are extracted from preoperative image data depicting the patient's processing targets, and from the extracted processing targets, path field image data is generated showing the processing targets in the surgical route planned preoperatively. During the surgery on the aforementioned patient, surgical field image data is acquired for multiple frames in a time series in which the surgical field is shown and the object to be processed is depicted. A reference processing target is extracted from one frame of surgical field image data, at least one of the processing target indicated by the path field image data is performed on the processing target to match the extracted processing target, and the processing target, which has undergone at least one of the deformation and movement, is superimposed on the one frame of surgical field image data and displayed on the display unit. method.

8. The processing targets are extracted from preoperative image data depicting the patient's processing targets, and from the extracted processing targets, path field image data is generated showing the processing targets in the surgical route planned preoperatively. During surgery on the aforementioned patient, surgical field image data is acquired for multiple time-series frames showing the surgical field and depicting the target of processing. From the surgical field image data, obtain at least one of the scaling ratio and movement amount of the object to be processed between frames. Based on the scaling ratio and the amount of movement of the processing target, at least one of the processing target and the path field image data is subjected to deformation and movement, and the path field image data on which at least one of the deformation and movement has been performed is displayed on the display unit. method.

Images