Computer program and image playback method

Abstract

The present invention provides a computer program and video playback method that facilitate understanding of events during surgery. The computer program causes a computer to perform the following processing: recording surgical area images obtained by capturing the surgical area during endoscopic surgery in a time series; determining whether bleeding exceeding a predetermined amount occurs based on the surgical area images; and, if bleeding exceeding the predetermined amount is determined, playing a portion of the recorded surgical area images encompassing the time range from a point before bleeding begins to a point after bleeding occurs, using a set playback method.

Description

Technical Field

[0001] The present invention relates to a computer program and an image playing method. Background Art

[0002] Patent Document 1 proposes a solution for playing a portion of a surgical motion picture.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2011-36370. Summary of the Invention

[0006] Problems to be solved by the invention

[0007] In Patent Document 1, although it is possible to confirm a necessary portion by starting playback from a characteristic point, simply playing back a scene where an unexpected event such as bleeding has occurred often makes it difficult to grasp the portion that caused the unexpected event.

[0008] The present invention has been made in view of such a background, and an object of the present invention is to provide a technique for making events during surgery easier to understand.

[0009] Solutions for solving problems

[0010] A computer program of the present invention causes a computer to perform the following processing: recording surgical area images obtained by photographing the surgical area during endoscopic surgery in a time series; determining whether bleeding exceeding a specified amount is present based on the surgical area images; and, if bleeding exceeding the specified amount is determined to be present, playing a portion of the recorded surgical area images that includes a time range from a point before bleeding begins to a point after bleeding begins, in a set playback mode.

[0011] Effects of the Invention

[0012] According to the present invention, events during surgery can be easily understood. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a conceptual diagram illustrating the outline of the surgical system according to the present embodiment.

[0014] Figure 2 This is a block diagram schematically illustrating the surgical system according to this embodiment.

[0015] Figure 3 It is a block diagram illustrating a hardware configuration example of the control unit 30 .

[0016] Figure 4 It is a block diagram illustrating an example of the software configuration of the control unit 30 .

[0017] Figure 5 This is a block diagram illustrating a schematic configuration of the image M acquired by the endoscopic camera 13 .

[0018] Figure 6 It is a diagram that illustrates the occurrence of an event.

[0019] Figure 7 This is a diagram illustrating frame F1 when an event occurs.

[0020] Figure 8 1 is a diagram showing an example of a frame F1 at the time of event occurrence.

[0021] Figure 9 2 is a diagram showing an example of the display 21 displaying a small window image W.

[0022] Figure 10 It is a diagram for explaining the processing performed in the surgical system according to this embodiment.

[0023] Figure 11 It is a diagram showing a software configuration example of the control unit 30 according to the second embodiment.

[0024] Figure 12 It is a diagram showing the processing flow of the second embodiment.

[0025] Figure 13 It is a diagram showing a software configuration example of the control unit 30 according to the third embodiment.

[0026] Figure 14 It is a diagram illustrating the processing flow of the control unit 30 according to the third embodiment.

[0027] Figure 15 It is a diagram showing an example of the software configuration of the control unit 30 according to the fourth embodiment.

[0028] Figure 16 It is a diagram for explaining the processing flow of the control unit 30 according to the fourth embodiment.

[0029] Figure 17 It is a diagram showing the overall configuration of a surgical system 1 according to a fifth embodiment.

[0030] Figure 18 It is a diagram for explaining the processing flow of the control unit 30 according to the sixth embodiment.

[0031] Figure 19 This is a diagram illustrating an example of a playback method.

[0032] Figure 20 It is a diagram for explaining the processing flow of the control unit 30 according to the seventh embodiment.

[0033] Figure 21This figure shows an example of display of thumbnails of bleeding scenes.

[0034] Figure 22 It is a diagram for explaining the processing flow of the control unit 30 according to the eighth embodiment.

[0035] Figure 23 This is a diagram showing an example of a graph display showing a time-series change in bleeding volume.

[0036] Figure 24 It is a diagram for explaining the processing flow of the control unit 30 according to the tenth embodiment. DETAILED DESCRIPTION

[0037] <Overview of the Invention>

[0038] The present invention is described below by listing the contents of the embodiments. The present invention includes, for example, the following structures.

[0039] [Project 1]

[0040] A motion picture playback system, comprising:

[0041] a moving image acquisition unit that acquires a moving image captured of a surgical field during endoscopic surgery;

[0042] an event detection unit that analyzes the moving image and detects a predetermined event in the operating area;

[0043] a partial moving image extraction unit that extracts a partial moving image during a period including a detection time point of the event from the moving image;

[0044] The event playback unit plays the partial moving image in each of a plurality of playback modes.

[0045] [Project 2]

[0046] The motion picture playback system described in item 1 is characterized in that:

[0047] The playing mode includes at least two of normal playing, slow playing, repeated playing and reverse playing.

[0048] [Item 3]

[0049] The motion picture playback system described in item 1 is characterized in that:

[0050] The event playback unit selects the types of the plurality of playback methods based on at least one of a type of the organ to be operated on, a surgical method related to the treatment, and a parameter related to the event.

[0051] [Item 4]

[0052] The motion picture playback system described in item 1 is characterized in that:

[0053] The event playback unit selects at least any one of the playback speed related to the playback method, the time from the detection time point to the start time point of the period, the time from the detection time point to the end time point of the period, and the playback direction based on at least any one of the category of the organ serving as the surgical object, the surgical method involved in the treatment, and the parameters involved in the event.

[0054] [Item 5]

[0055] The motion picture playback system described in item 1 is characterized in that:

[0056] The event is bleeding from the surgical subject,

[0057] The event detection unit also detects the amount of bleeding,

[0058] The playing mode at least includes the slow playing,

[0059] The event playback unit determines a playback speed of the slow playback based on at least the bleeding amount.

[0060] [Item 6]

[0061] The surgical support system according to item 1 is characterized in that:

[0062] For a second period of the first period that is shorter than the first period and includes the detection time point, the event playback unit plays the partial moving image at a slower playback speed than during other periods.

[0063] [Item 7]

[0064] The surgical support system according to item 1 is characterized in that:

[0065] It also includes a setting unit, which sets the types of the multiple playback modes, and at least any one of the playback speed, the time from the detection time point to the start time point of the period, the time from the detection time point to the end time point of the period, and the playback direction related to the playback mode.

[0066] [Item 8]

[0067] The surgical support system according to item 7 is characterized in that:

[0068] After the event playback unit starts playing, the setting unit changes at least any one of the types of the multiple playback modes, the playback speed of the playback mode, the time from the detection time point to the start time point of the period, the time from the detection time point to the end time point of the period, and the playback direction.

[0069] System Overview

[0070] The following describes a surgical system according to one embodiment of the present invention. The surgical support system of this embodiment can be a system that supports endoscopic surgery, and in particular, can be a system that outputs a moving image captured of the surgical field and, in the event of an event such as localized bleeding, plays back a moving image containing the scene of the event.

[0071] <First embodiment>

[0072] Figure 1 1 is a conceptual diagram illustrating the outline of a surgical system according to the present embodiment. As shown in the figure, the surgical system 1 includes a surgical unit 10 , an operating unit 20 , and a control unit 30 for controlling the surgical unit 10 and the operating unit 20 .

[0073] Figure 2 This is a block diagram schematically illustrating the surgical system according to this embodiment.

[0074] The operating unit 10 performs surgical treatment on a patient 100 as an operating target. In this embodiment, the operating unit 10 includes an instrument unit 11 as a treatment unit, a sensor unit 12 as a treatment unit, and an endoscopic camera 13 as an image acquisition unit.

[0075] The apparatus portion 11 is composed of a movable arm and a surgical tool installed at the movable arm front end. As a surgical tool, for example, a scalpel (electric knife etc.), scissors, forceps, needle holder, tweezers etc. can be expected, but various surgical tools in addition thereto can also be installed according to purposes.

[0076] The sensor unit 12 may employ various sensors such as a pressure sensor, a gyro sensor, an acceleration sensor, and a temperature sensor for detecting the state of the device unit 11 .

[0077] The structure of the endoscopic camera 13 will be described later.

[0078] The operating unit 20 receives operations from the operating surgeon 110, who is the operator of the surgical unit 10. In this embodiment, the operating unit 20 includes a display 21 as a display unit, a controller 22, and a speaker 23. The operating unit 20 may also include a microphone for receiving voice instructions from the operating surgeon 110.

[0079] The display 21 provides various information to the operating surgeon 110. In this embodiment, the display 21 adopts HMD (Head Mount Display) technology to realize VR (Virtual Reality), and utilizes the parallax of the operating surgeon's 110 eyes to form a stereoscopic vision of the area to be recognized.

[0080] In this embodiment, the controller 22 is implemented by an input device such as a joystick or a foot pedal, and the speaker 23 provides various information to the operating surgeon 110 through sound or the like.

[0081] <Hardware of Control Unit 30>

[0082] Figure 3 3 is a block diagram illustrating an example of the hardware configuration of the control unit 30. As shown in the figure, the control unit 30 mainly includes a processor 31, a memory 32, a storage device 33, a transceiver 34, and an input / output unit 35, which are electrically connected to each other via a bus 36.

[0083] The processor 31 is a computing device that controls the operation of the control unit 30 , controls data transmission and reception between various elements, and performs processing required for executing application programs.

[0084] In the present embodiment, the processor 31 is, for example, a CPU (Central Processing Unit), and executes various processes by running an application program stored in a storage device 33 described later and opened in the memory 32 .

[0085] The memory 32 includes a main storage device composed of a volatile storage device such as a DRAM (Dynamic Random Access Memory) and an auxiliary storage device composed of a nonvolatile storage device such as a flash memory and an HDD (Hard Disc Drive).

[0086] The memory 32 is used as a work area of ​​the processor 31 and stores a BIOS (Basic Input / Output System) that is executed when the control unit 30 is activated and various setting information.

[0087] The storage device 33 stores application programs and data used for various processes. In this embodiment, the storage device 33 stores an image processing program that performs various processes. The image processing program is provided, for example, on a non-transitory recording medium RM on which the program is readable. The processor 31 reads the desired program from the recording medium RM using a reader (not shown) and stores the read program in the storage device 33. This image processing program will be described in detail later.

[0088] The transmission and reception unit 34 connects the control unit 30 to the Internet. The transmission and reception unit 34 may include a short-range communication interface such as Bluetooth (registered trademark) or BLE (Bluetooth Low Energy).

[0089] In this embodiment, the control unit 30 is connected to the surgical unit 10 and the operating unit 20 via the transmitting and receiving unit 34 .

[0090] Information input devices such as a keyboard and a mouse and output devices such as a display can be connected to the input / output unit 35 as needed.

[0091] The bus 36 transmits, for example, address signals, data signals, and various control signals among the connected processor 31 , memory 32 , storage device 33 , transmission / reception unit 34 , and input / output unit 35 .

[0092] <Software of the control unit 30>

[0093] Figure 4 3 is a block diagram illustrating an example of the software configuration of the control unit 30. As shown in the figure, the control unit 30 includes an operation input unit 311, an occurrence time point determination unit 312, a partial moving image extraction unit 313, a playback unit 314, a video recording unit 331, a parameter storage unit 332, and a setting storage unit 333.

[0094] The image recording unit 331 records the image M acquired by the endoscopic camera 13 and displayed in real time on the display 21 as a recorded image M1 .

[0095] Figure 5 This is a block diagram schematically illustrating the structure of an image M acquired by the endoscopic camera 13. As shown in the figure, the endoscopic camera 13 acquires an image M by capturing a surgical procedure O performed on a patient 100 by the instrument unit 11. The image M acquired by the endoscopic camera 13 consists of a plurality of frames F. After acquisition, the image M is sequentially displayed on the display 21 in real time. Furthermore, the image M displayed on the display 21 is recorded as a recorded image M1 in the image recording unit 331.

[0096] The surgical system 1 of this embodiment having such a structure opens multiple small holes of about 3 mm to 10 mm in size on the body of the patient 100 to reach the abdominal cavity or thoracic cavity of the patient 100. Instead of cutting the body open, the endoscopic camera 13 and the surgical tools of the instrument part 11 are inserted through the above-mentioned holes to perform minimally invasive endoscopic surgery under the endoscope.

[0097] The operation input unit 311 receives various operations input by the operating surgeon 110 through the operation unit 20 .

[0098] The occurrence time point determination unit 312 determines the time point at which an event occurred (hereinafter referred to as the event occurrence time point) from the recorded image M1. For example, the occurrence time point determination unit 312 performs image analysis on each frame F constituting the recorded image M1 to extract image features that indicate an event (e.g., the red color of bleeding). This allows the determination of whether an event occurred in the frame and determines the time point of the earliest frame F during a series of events in which the same event occurred as the event occurrence time point.

[0099] The occurrence time point identification unit 312 can be implemented, for example, by an artificial intelligence module that autonomously identifies frames related to an event through machine learning. Various algorithms such as neural networks, random forests, and SVM (support vector machines) can be used as appropriate for machine learning.

[0100] Figure 6 It is a diagram that illustrates the occurrence of an event. Figure 7 This is a diagram illustrating frame F1 when an event occurs.

[0101] like Figure 6 As shown, when the surgical tools 11a and 11b of the instrument section 11 of the surgical unit 10 are used to perform surgical treatment on the affected area T of the patient 100, an unexpected situation (event) occurs to the patient 100, such as localized bleeding, and bleeding B is displayed on the image M displayed in real time on the monitor 21. At this time, as shown in FIG. Figure 7 As shown, the occurrence time point identification unit 312 can identify, from the frames F constituting the recorded image M1, the frame F1 in which the occurrence of bleeding B is recorded. In the frame F1 in which the event occurred, the early stage of bleeding B is recorded, which is the stage immediately after the bleeding B has begun to flow out but before the bleeding B has spread, that is, the state immediately after the bleeding B has occurred.

[0102] The partial moving image extraction unit 313 extracts a partial moving image (frames included in the period) during a period including the event occurrence time point. Figure 7In the example shown in FIG. 1 , the partial moving image extraction unit 313 can extract each frame from the start frame to the end frame as a partial moving image, using frame FS at a time point i1 earlier than frame F1 at event time t0 as the start frame and frame FE at a time point i2 later than frame F1 at event time t0 as the end frame. In other words, the partial moving image extraction unit 313 can extract the partial moving image from the recorded video M1 for the period from the start time point t0-i1 to the end time point t0+i2.

[0103] The parameter storage unit 332 stores various parameters such as times i1 and i2. Figure 4 In the example shown in FIG. 1 , parameter storage unit 332 stores parameters such as the lead time, lag time, playback speed, and number of playbacks. For example, the lead time is time i1, and the lag time is time i2. The playback speed is the playback speed used when playback unit 314, described later, plays a portion of a moving image in slow motion. The number of playbacks is the number of repetitions used when playback unit 314, described later, plays a portion of a moving image repeatedly.

[0104] The playback unit 314 plays a portion of the moving image. In this embodiment, the playback unit 314 plays the portion of the moving image in multiple playback modes. The multiple playback modes include at least two of normal playback, slow playback, repeat playback, and reverse playback. In other words, the playback unit 314 can play the scene containing the time point of the event multiple times while changing the playback mode. Various parameters related to the playback mode (e.g., the playback speed of slow playback, the number of repeat playback times, etc.) are stored in the parameter storage unit 332.

[0105] The playback unit 314 can display a portion of the image as a small window image on the display 21 that the operating doctor 110 is viewing. Figure 8 1 is a diagram showing an example of a frame F1 at the time of event occurrence. Figure 8 An example of identifying the time point at which bleeding B occurs is shown in FIG. Figure 9 2 is a diagram showing an example of the display 21 displaying a small window image W. Figure 9 In the example, the frame F1 at the time point when the event occurs is displayed as a small window image W.

[0106] Furthermore, the playback unit 314 can play back the partial moving image in a combination of different playback methods according to the type of operation of the surgical tool of the instrument unit 11. The playback method is stored in the setting storage unit 333 described later.

[0107] Figure 4In the example, the storage unit 333 stores multiple playback modes (lists) corresponding to the operation categories. Examples of operation categories include excision, peeling, traction, suturing, and suction. These operations can be determined, for example, through image analysis using machine learning. Alternatively, the operation can be determined based on the type of surgical tool installed on the instrument unit 11, the operation input signal from the operating surgeon 110 received by the operation input unit 311, and so on.

[0108] <Processing>

[0109] Figure 10 1 and 2 are diagrams for explaining the processing performed in the surgical system according to this embodiment.

[0110] The occurrence time point identification unit 312 identifies the time point (frame) at which the event occurred from the recorded image M1 (S401) and identifies the type of operation of the surgical tool of the instrument unit 11 (S402). For example, the occurrence time point identification unit 312 can identify the type of operation by analyzing the image.

[0111] The partial motion image extraction unit 313 reads out the parameter and playback mode list from the parameter storage unit 332 and the setting storage unit 333 (S403), calculates the period to be extracted (t0-i1~t0+i2) based on the advance time (i1) and lag time (i2) read out as parameters, and the determined event occurrence time point t0, and extracts the partial motion image of the calculated period from the recorded image M1 (S404).

[0112] The playback unit 314 takes out the next playback mode from the playback mode list (S405). If there is a next playback mode (S406: No), it plays the part of the moving image in the next playback mode (S407). Here, the playback mode parameters (playing speed of slow playback, number of repeat playbacks, etc.) use the parameters read out above. The playback unit 314 can use the part of the image as Figure 9 The small window image W shown is outputted superimposed on the real-time image from the endoscope camera 13 . Alternatively, a sub-display may be provided separately from the display 21 to output a portion of the image on the sub-display.

[0113] As described above, the surgical system according to this embodiment can output a partial moving image of the period including the time point of the event occurrence, i.e., output the scene of the event occurrence, when an event (e.g., bleeding, etc.) is detected in the space to be operated on (e.g., the abdominal cavity, etc.). Figure 6 As in the example, when bleeding B occurs and spreads, the surgeon 110 performs the following Figure 9Shown in the small window image W, confirm the part of the motion picture during the early time point (event occurrence time point) that comprises hemorrhage B and can easily determine the bleeding site where hemorrhage B has for example occurred.Can also determine the reason that hemorrhage B occurs, thereby can adopt appropriate disposal.Its reason is as follows: under the situation of for example hemorrhage, suppose that hemorrhage is to clamp the internal tissue with the excision instrument and implement excision so that what occurs after the tissue destruction, the action before clamping and the time in the excision process can be grasped with usual speed, but the process that hemorrhage begins to spread from the bleeding point after the tissue destruction is the phenomenon that occurs in the very short time, therefore, by slowing down the playback speed of this scene, can be easily understood.In addition, by playing in reverse, the blood that has spread is gathered to a little, and the position of grasping the bleeding point also becomes easy.In addition, by slowing down the playback speed of a part like this or playing in reverse, determining the site of occurrence and the cause of occurrence of events such as hemorrhage can become easy.

[0114] Furthermore, the surgical system of this embodiment enables viewing of a portion of a moving image during a period encompassing the time point of an event, such as bleeding B, using multiple playback modes. Therefore, compared to simply viewing a past scene once, multiple displays, as in this embodiment, make it easier and more accurate to grasp the location and cause of the event. Furthermore, by playing the same scene in different playback modes, as in this embodiment, the same event can be viewed from different perspectives, making it easier and more accurate to grasp the event's location (e.g., bleeding site) and the primary cause (e.g., cause of the bleeding) of the event.

[0115] <Second embodiment>

[0116] Figure 11 3 is a diagram showing an example of the software configuration of the control unit 30 according to the second embodiment. In the first embodiment, the playback speed for slow playback (or fast forward playback) is set as a parameter in the parameter storage unit 332. However, in the second embodiment, the playback speed can be changed according to the amount of bleed.

[0117] The control unit 30 of the second embodiment is different from the control unit 30 of the first embodiment (see Figure 4), further comprising a bleeding volume detection unit 315. The bleeding volume detection unit 315 analyzes the images of each frame F of the recorded image M1 and estimates the amount of blood contained in the image. The bleeding volume detection unit 315 can, for example, calculate the amount of bleeding (or the bleeding area, i.e., the proportion of the surgical target area covered by blood) based on the number of pixels of the bleeding color contained in the image. In addition, the playback unit 314 of the second embodiment changes the playback speed according to the amount of bleeding. Specifically, the playback unit 314 can make the playback speed slower as the amount of bleeding (or bleeding area) increases. In addition, the playback unit 314 can make the playback speed slower as the amount of bleeding (or bleeding area) changes over time (e.g., the increase in bleeding volume per unit time).

[0118] Figure 12 1 is a diagram showing the processing flow of the second embodiment. Figure 10 ). In the second embodiment, the bleeding amount detection unit 315 detects the amount of bleeding (or bleeding speed) from a portion of the moving image (S421), and the playback unit 314 can adjust the playback speed to be slower as the amount of bleeding (or bleeding speed) increases (or increases) (S422). Then, in step S407, the portion of the moving image is played at the adjusted playback speed. Thus, in the case of heavy or rapid bleeding, the scene at the time of bleeding (the time of the event) can be played more slowly, making it easier and more accurate to determine the bleeding site and cause.

[0119] Furthermore, the playback unit 314 may change the playback speed only when the playback mode is slow playback, or may change the playback speed for all playback modes.

[0120] <Third embodiment>

[0121] Figure 13 3 is a diagram showing an example of the software configuration of the control unit 30 according to the third embodiment. In the first embodiment, the playback speed of a portion of the moving image is constant. However, in the third embodiment, the playback speed can be slowed down for a shorter period before and after the event occurs in the portion of the moving image.

[0122] The control unit 30 of the second embodiment is different from the control unit 30 of the first embodiment (see Figure 4 ) compared, as parameters stored in the parameter storage unit 332, two sets of advance time, lag time and playback speed are stored. Figure 13In the example shown in FIG1 , the parameter storage unit 332 stores a first group and a second group. The first group includes advance time 1 (i1(1)), lag time 1 (i2(1)), and playback speed 1, while the second group includes advance time 2 (i1(2)), lag time 2 (i2(2)), and playback speed 2. The duration of the second group is shorter than the duration of the first group, and both the duration of the first group and the duration of the second group include the event start time point (frame F1). In addition, the playback speed 2 of the second group is slower than the playback speed 2 of the first group.

[0123] Figure 14 3 is a diagram illustrating a process flow of the control unit 30 of the third embodiment. Figure 10 ) difference. In the third embodiment, steps S441 to S443 are executed instead of the playback processing of step S407. In step S441, the period from the start time point t0-i1(1) to the start time point t0-i1(2) of the second group is played at the playback speed 1 of the first group, and the start time point t0-i1(1) is the same as the start time point t0-i1 of the first embodiment (S441), the period from the start time point t0-i1(2) of the second group to the end time point t0+i2(2) of the second group is played at the playback speed 2 of the second group (S442), and the period from the end time point t0+i2(2) of the second group to the end time point t0+i2(1) of the first group is played at the playback speed 1 of the first group (S443). In this way, the period of the second group, which includes the time point of event occurrence and is shorter than the period of the first group, can be played at a slower speed.

[0124] In the third embodiment, similarly to the second embodiment, the bleeding amount detection unit 315 may be provided, and playback speed 2 may be set to a speed corresponding to the bleeding amount or bleeding speed, and playback may be performed for the second group of periods.

[0125] Furthermore, the playback speed 1 of the first group can be set to a speed faster than the normal speed (real time) to fast-forward the portion other than the scene including the time point of the event occurrence.

[0126] <Fourth embodiment>

[0127] Figure 15This figure shows an example of the software structure of the control unit 30 according to the fourth embodiment. In the fourth embodiment, various parameters related to the playback mode (advance time, lag time, playback direction, playback speed, number of playbacks, etc.) can be set and changed. In the fourth embodiment, the parameter storage unit 332' stores various parameters corresponding to the playback mode ID representing the playback mode. In addition, the control unit 30 includes a setting unit 316, which receives input of the parameters stored in the parameter storage unit 332'. The setting unit 316 can register the parameters in advance and can also change the parameters during surgery or during motion image playback.

[0128] Figure 16 1 is a diagram illustrating a processing flow of the control unit 30 of the fourth embodiment. Figure 10 ) In the fourth embodiment, the setting unit 316 receives input of parameters for each playback mode in advance and registers them in the parameter storage unit 332' (S461). Then, in step S403, the playback mode list is read (S403'). Before step S407 during playback, the playback unit 314 reads the parameters corresponding to the playback mode from the parameter storage unit 332' (S462) and plays the portion of the moving image using the read parameters (S407').

[0129] This allows for free setting of playback speed, playback duration, and the like according to the playback method.

[0130] <Fifth embodiment>

[0131] Figure 17 This diagram shows the overall configuration of a surgical system 1 according to a fifth embodiment. In the fifth embodiment, the operating surgeon 110 directly operates surgical tools on a patient 100, rather than using a remote control. This can be either a laparotomy or a laparoscopic procedure. In such a scenario, the control unit 30 of each of the aforementioned embodiments can be used to replay scenes containing the time points at which events occurred multiple times while changing the playback mode.

[0132] <Sixth embodiment>

[0133] The sixth embodiment describes a configuration for playing a portion of a moving image within a predetermined time range including the bleeding start time when bleeding of a predetermined amount or more is detected.

[0134] Figure 18 1 is a diagram illustrating a processing flow of the control unit 30 of the sixth embodiment. Figure 10). When an event (bleeding in this embodiment) is detected in step S401, the bleeding amount detection unit 315 analyzes the image of frame F obtained by the endoscopic camera 13 and detects the amount of bleeding (S471). As described in Embodiment 2, the bleeding amount detection unit 315 can detect the amount of bleeding based on the number of pixels or the area of ​​the region having a color corresponding to bleeding. Whenever an image of frame F is input, the bleeding amount detection unit 315 detects the amount of bleeding and compares it with a preset threshold to determine whether there is bleeding exceeding a specified amount (S472). If there is no bleeding exceeding the specified amount (S472: No), the control unit 30 returns the process to step S401.

[0135] If it is determined that bleeding has occurred at a predetermined amount or more (S472: YES), the control unit 30 proceeds to step S402 and beyond. Specifically, the control unit 30 reads various parameters related to the playback mode (e.g., playback speed, number of repeat playbacks, etc.) and a playback mode list based on the type of operation (e.g., excision, peeling, traction, suturing, or suction), and plays a partial motion image extracted from the recorded video M1 according to the read parameters and playback mode list. The method for extracting the partial motion image is the same as in the first embodiment. Specifically, the partial motion image extraction unit 313 extracts, as partial motion images, frames F within the time range from the time point (t0-i1) before the onset of bleeding to the time point (t0+i2) after the onset of bleeding in the recorded video M1 recorded by the video recording unit 331. In this embodiment, t0 represents the time point at which bleeding begins. Times i1 and i2 can be preset or set based on the type of operation (e.g., excision, peeling, traction, suturing, or suction) or the scene being captured.

[0136] The playback unit 314 plays the partial motion image in at least one of normal playback, slow playback, frame-by-frame playback, repeat playback, and reverse playback. Figure 9 The small window image W shown is output superimposed on the real-time image (surgical field image) from the endoscopic camera 13. Alternatively, a sub-display may be provided separately from the display 21, and a portion of the moving image may be output to the sub-display. Furthermore, the playback unit 314 may repeatedly playback the portion of the moving image while switching between at least two of normal playback, slow playback, frame-by-frame playback, and reverse playback. Furthermore, the playback unit 314 may playback the scene including the bleeding start time at a lower speed than the scenes before and after the bleeding start time.

[0137] Figure 19This figure illustrates an example of a playback method. In a recorded video M1 containing a scene where bleeding exceeding a predetermined amount is detected, with time t = t0 being the bleeding start time, the partial motion image extraction unit 313 extracts frames F from time t = t0 - i1(1) to time t = t0 + i2(1) as partial motion images. i1(1) is set to, for example, 2 to 6 seconds depending on the type of operation and the scene. i2(1) is set to an appropriate time, for example, 2 seconds.

[0138] The playback unit 314 switches the playback mode in sequence, such as normal playback, partial slow playback, and partial slow reverse playback, to play part of the moving image. The playback unit 314 may also repeatedly perform normal playback, partial slow playback, and partial slow reverse playback after performing partial slow reverse playback.

[0139] During normal playback, for example, i1(1) is set to 6 seconds and i2(1) is set to 2 seconds. That is, the playback unit 314 plays a portion of the moving image at the normal playback speed (playback speed 1), which is 8 seconds in total, starting from 6 seconds before the start of bleeding to 2 seconds after the start of bleeding. Since the playback of the surgical area image captured before the start of bleeding is slightly longer, the surgeon can understand the operation that caused the bleeding.

[0140] In partial slow playback, for example, i1(1) is set to 3 seconds, i2(1) is set to 2 seconds, and i1(2) and i2(2) are each set to 0.2 seconds. That is, the playback unit 314 plays the 2.8-second partial motion image from 3 seconds before the bleeding start time to 0.2 seconds before at the normal playback speed (playback speed 1), and plays the 0.4-second partial motion image from 0.2 seconds before the bleeding start time to 0.2 seconds after at one-fifth the normal playback speed (playback speed 2). In addition, the playback speed of slow playback is not limited to one-fifth speed and can be set appropriately. After slow playback, the playback unit 314 plays the 1.8-second partial motion image from 0.2 seconds after the bleeding start time to 2 seconds after at the normal playback speed (playback speed 1). In partial slow playback, the moment of bleeding is played back in slow motion, so there is an advantage that the surgeon can easily identify the bleeding site. In addition, since the video is played slowly from the time of bleeding, the surgeon can predict that bleeding will occur thereafter. In addition, by adopting partial slow playback, the playing time can be prevented from becoming longer.

[0141] The time setting for partial slow reverse playback is the same as the time setting for partial slow playback. The playback unit 314 plays back the portion of the moving image from 0.2 seconds after the bleeding start time to 2 seconds after at a normal playback speed (playback speed 1), and plays back the portion of the moving image from 0.2 seconds before the bleeding start time to 0.2 seconds after at a normal playback speed of one-fifth (playback speed 2). In addition, the playback speed during slow reverse playback is not limited to one-fifth speed and can be set appropriately. After slow playback, the portion of the moving image from 3 seconds before the bleeding start time to 0.2 seconds before is played back at a normal playback speed (playback speed 1). During partial slow reverse playback, since the diffused bleeding is displayed in a manner that converges to one part in sequence, the surgeon can easily identify the bleeding part by grasping the convergence point.

[0142] Figure 9 , a structure for playing part of the moving images in the order of normal play, partial slow play, and partial slow reverse play is described. Such a play order and play mode can be the default setting of the control unit 30. In addition, the control unit 30 can also accept changes to the play order and play mode through the operating unit 20. The play order and play mode can also be set arbitrarily by the surgeon. For example, the surgeon can change the setting so that part of the moving images are played in the order of partial slow play and partial slow reverse play, or can change the setting so that part of the moving images are played in the order of normal play, partial slow play, reverse play, and partial slow reverse play. When the control unit 30 accepts the change in the setting of the play order and play mode, it plays part of the moving images according to the changed play order and play mode.

[0143] As described above, in the sixth embodiment, a bleeding scene is displayed as a moving image only when bleeding of a predetermined amount or more is detected. Therefore, only the bleeding scene that the surgeon needs to recognize can be displayed without interfering with the surgery.

[0144] In the sixth embodiment, the threshold for the amount of bleeding is preset, but the threshold may be set by a surgeon, such as a surgeon, via the operation input unit 311. Once the threshold is set, the bleeding amount detection unit 315 compares the amount of bleeding estimated from the image with the set threshold to determine whether bleeding exceeds the predetermined amount.

[0145] <Seventh embodiment>

[0146] In the seventh embodiment, bleeding exceeding a threshold value occurring during shooting is identified and the identified bleeding scene is displayed as a thumbnail on the display 21. Upon receiving selection of a thumbnail, the control unit 30 plays a moving image of the corresponding bleeding scene.

[0147] Figure 20 1 is a diagram illustrating a processing flow of the control unit 30 of the seventh embodiment. Figure 18 ). When bleeding exceeding a predetermined amount is detected (S481), the control unit 30, using the same process as the sixth embodiment, generates a thumbnail representing the bleeding scene and displays the generated thumbnail on the display 21, superimposed on the surgical field image (S482). A still image or a moving image representing the bleeding scene can be displayed in the thumbnail. Furthermore, as information identifying the bleeding scene, the control unit 30 stores, for example, information on the bleeding start time t0, in association with the thumbnail in the parameter storage unit 332 (S483).

[0148] Figure 21 This figure shows an example of display of thumbnails of bleeding scenes. Figure 21 An example is shown in which two thumbnails TH1 and TH2 are superimposed on an image of the surgical field and displayed on the display 21. Thumbnails TH1 and TH2 are preferably positioned away from the center of the surgical field image to avoid obstructing identification of the treated area. Therefore, the control unit 30 can, for example, position the generated thumbnails TH1 and TH2 at the edges of the surgical field image. Alternatively, the control unit 30 can determine the positions of the surgical tools 11a and 11b through image analysis and display the thumbnails TH1 and TH2 at positions away from the determined surgical tools 11a and 11b. Furthermore, the control unit 30 can also attach information to each thumbnail TH1 and TH2 indicating the time at which bleeding exceeding a specified amount was detected.

[0149] The control unit 30 determines whether a thumbnail selection has been accepted via the operation input unit 311 (S484). The operation input unit 311 may accept a thumbnail selection using an input device such as a joystick or foot pedal included in the controller unit 22, or may accept a voice instruction for selecting a thumbnail using a microphone. If no thumbnail selection has been accepted (S484: No), the control unit 30 returns the process to step S481.

[0150] Whenever bleeding exceeding a predetermined amount is detected, the control unit 30 adds a new thumbnail to the surgical field image. Furthermore, if the number of thumbnails increases significantly, it becomes difficult to identify the affected area T on the surgical field image. Therefore, the number of thumbnails displayed can be limited. For example, the control unit 30 can select and display a predetermined number of thumbnails in descending order of detection time. Alternatively, instead of displaying the thumbnails superimposed on the surgical field image, the surgical field image can be displayed on the display 21 while the thumbnails are displayed separately on the secondary display.

[0151] When a thumbnail selection operation is accepted (S484: YES), the partial moving image extraction unit 313 reads out the parameter and playback method list for the bleeding scene represented by the selected thumbnail from the parameter storage unit 332 and the setting storage unit 333 (S403). At this time, the partial moving image extraction unit 313 can read out the necessary parameter and playback method list using the bleeding start time point t0 associated with the thumbnail as a search key.

[0152] The processing after reading the parameters and the playback mode list is the same as that in the sixth embodiment. The partial motion image extraction unit 313 extracts the partial motion image, and the playback unit 314 plays the partial motion image according to the playback mode described in the playback mode list. The playback unit 314 plays the partial motion image according to at least one playback mode including normal playback, slow playback, frame-by-frame playback, repeat playback, and reverse playback. The playback unit 314 can use the partial motion image as Figure 9 The small window image W shown is superimposed on the real-time image (surgical field image) from the endoscope camera 13. Alternatively, a sub-display may be provided separately from the display 21, and a portion of the moving image may be output to the sub-display. The playback unit 314 may also repeatedly play the portion of the moving image while switching between at least two of normal playback, slow playback, frame-by-frame playback, and reverse playback. Furthermore, the playback unit 314 may play the scene including the bleeding start time at a slower speed than the scenes before and after the bleeding start time.

[0153] As described above, the surgical system 1 in the seventh embodiment plays the moving image of the bleeding scene only when the thumbnail is selected. Therefore, it is possible to provide only the bleeding scene desired by the operating surgeon or other surgeon as a moving image.

[0154] <Eighth embodiment>

[0155] In the eighth embodiment, a configuration will be described in which, when bleeding is detected in an edge region of the operating area region obtained by the endoscopic camera 13, a notification is given regarding the detection of bleeding.

[0156] Figure 22 1 is a diagram illustrating a processing flow of the control unit 30 according to the eighth embodiment. Figure 18 ) using the same process as the sixth embodiment. When bleeding exceeding a predetermined amount is detected (S491), the control unit 30 determines whether the bleeding occurs in the end region of the surgical field image (S492). The end region can be set as the area of ​​the surgical field image excluding the center (area of ​​interest). In one example, the end region is set as the upper, lower, left, and lower quarter regions of the surgical field image.

[0157] If bleeding is determined to have occurred in the edge region of the surgical field image (S492: Yes), the control unit 30 reports the occurrence of bleeding (S493). The control unit 30 displays, for example, text information or an icon indicating the occurrence of bleeding on the display 21. Alternatively, the control unit 30 may output an audio message indicating the occurrence of bleeding through the speaker 23.

[0158] As described above, in the eighth embodiment, when it is determined that bleeding has occurred in the edge region of the surgical field image, information indicating the occurrence of bleeding is reported, thereby enabling the surgeon to recognize the presence of the hardly noticeable bleeding.

[0159] <Ninth embodiment>

[0160] In the ninth embodiment, a configuration for displaying a graph showing a time-series change in the amount of bleeding will be described.

[0161] Figure 23 This figure shows an example of a graph showing the time-series change in bleeding volume. The bleeding volume detection unit 315 of the control unit 30 can detect the bleeding volume at each moment by analyzing the frames F at each moment. The control unit 30 can generate a graph showing the time-series change in bleeding volume and display it along with the surgical field image. Figure 23 An example of a graph GR showing the time-series changes in bleeding volume is shown below an image of the surgical area. The horizontal axis of the graph GR represents time, and the vertical axis represents bleeding volume. In this graph GR, time Ta indicates the time when bleeding starts, and time Tb indicates the time when bleeding stops. Instead of bleeding volume, the time-series changes in the number of pixels or area corresponding to bleeding can also be displayed as a graph.

[0162] In addition, the control unit 30 may also be as follows Figure 23 As shown, points on the graph GR where the bleeding volume exceeds a specified amount are indicated with a marker such as MK. To highlight portions exceeding the threshold, the control unit 30 may also emphasize the corresponding portion of the graph GR by bolding or changing its color. The control unit 30 may also display the playback interval of the partial moving image (t0-i1≦t≦t0+i2) along with the graph GR. Furthermore, the control unit 30 may also display the graph GR during playback of the partial moving image.

[0163] As described above, in the ninth embodiment, a graph showing the time-series change in the amount of bleeding is displayed, so that the operator can understand the state of bleeding that occurred in the past.

[0164] <Tenth embodiment>

[0165] In the tenth embodiment, a configuration is described in which the presence of bleeding exceeding a predetermined amount is determined when a resection tool is included in the surgical field image, and no determination is made when a hemostatic tool is included in the surgical field image.

[0166] Figure 24 10. The process flow of the control unit 30 of the tenth embodiment is described. Figure 10 ). If an event (hemorrhage in this embodiment) is detected in step S401, the image of frame F obtained by the endoscope camera 13 is analyzed to determine whether the surgical field image contains a resection tool (S501). The control unit 30 can determine whether the surgical field image contains a resection tool by using techniques such as template matching to determine whether a predetermined shape is included. If no resection tool is included (S501: No), the control unit 30 returns the process to step S401.

[0167] If the image frame F contains a cutting tool (S501: Yes), the control unit 30 analyzes the image frame F and detects the amount of bleeding (S502). As described in Embodiment 2, the bleeding amount detection unit 315 can detect the amount of bleeding based on the number of pixels or the area of ​​the region having a color corresponding to bleeding. Each time the image frame F is input, the bleeding amount detection unit 315 detects the amount of bleeding and compares it with a preset threshold to determine whether bleeding exceeds a predetermined amount (S503). If bleeding does not exceed the predetermined amount (S503: No), the control unit 30 returns the process to step S401.

[0168] If it is determined that bleeding has occurred at a predetermined amount or greater (S503: YES), the control unit 30 performs the processing subsequent to step S402 described in the first embodiment. Specifically, the control unit 30 reads various parameters related to the playback mode (e.g., playback speed, number of repeat playbacks, etc.) and a playback mode list based on the type of operation (e.g., excision, peeling, traction, suturing, or suction), and plays a partial moving image extracted from the recorded video M1 according to the read parameters and playback mode list. The method for extracting the partial moving image is the same as in the first embodiment. Specifically, the partial moving image extraction unit 313 extracts as partial moving images frames F within the time range from the time point (t0-i1) before the onset of bleeding to the time point (t0+i2) after the onset of bleeding in the recorded video M1 recorded in the video recording unit 331. In this embodiment, t0 represents the time point at which bleeding begins. Times i1 and i2 can be preset or set based on the type of operation (e.g., excision, peeling, traction, suturing, or suction) or the captured scene.

[0169] Furthermore, when playing the partial moving image, the control unit 30 analyzes the image of frame F obtained by the endoscopic camera 13 to determine whether the surgical field image contains a hemostatic tool (S504). The control unit 30 can determine whether the surgical field image contains a hemostatic tool by using techniques such as template matching to determine whether a predetermined shape is present. If the surgical field image does not contain a hemostatic tool (S504: No), the control unit 30 plays the partial moving image (S407). If the surgical field image contains a hemostatic tool (S504: Yes), the control unit 30 does not play the partial moving image and terminates the processing in this flowchart.

[0170] As described above, in the tenth embodiment, a partial moving image is played only when the amount of bleeding increases to a predetermined amount or more during excision of the surgical site using a resection tool. Furthermore, if a hemostatic tool appears in the surgical field image, it can be assumed that the hemostatic site has been identified, and thus the partial moving image can be controlled so as not to be played.

[0171] The above description of the present embodiment is intended to facilitate understanding of the present invention and is not intended to limit the present invention. The present invention may be modified and improved without departing from the spirit thereof, and the present invention includes equivalents thereof.

[0172] For example, in the above embodiment, the event is described as bleeding B in the vicinity of the affected area T of the patient 100 , but damage, mishandling, or other abnormalities may also be detected.

[0173] This surgical system can be used not only in surgery, but also in all treatments where the above-mentioned accidents may occur, such as diagnosis, examination, diagnostic puncture and specimen collection, image examination, prescription and injection (including blood transfusion), rehabilitation training, anesthesia, radiotherapy, use of medical equipment, etc. The present invention can be applied to any treatment that may involve accidents caused by medical treatment or suspected of being caused by medical treatment.

[0174] The number of surgical systems during surgery is not limited to one, but may be multiple. In this case, the operation of the operating unit and the working unit may be synchronized with each other or may function independently of each other (without interlocking the operation of the first and second surgical devices).

[0175] The surgical objects of this system are not limited to human bodies, but can also be animals, objects used as training objects, plants, etc.

[0176] In the above-mentioned embodiments, instead of detecting the amount of bleeding, the same effects can be achieved by detecting digestive juices or bile discharged due to damage to organs or digestive organs.

[0177] In the above embodiments, the event is assumed to be a bleeding accident, but for example, a specific operation performed with a surgical tool can also be detected as an event. For example, suturing / anastomosis with a stapler (suture staple) can also be detected as an event, and the suturing / anastomosis process can be played back. This allows confirmation of whether the treatment was completed correctly.

[0178] While this embodiment depicts real-time images during surgery on a patient 100, this method is also useful for viewing recorded images M1 of surgical procedures afterward. For example, the operating surgeon 110 can view recorded images M1 of surgeries performed by themselves or other operating surgeons to identify the cause and location of an incident for future reference.

[0179] Description of Reference Numerals

[0180] 1 Surgical system

[0181] 10 Surgical Unit

[0182] 11 Equipment Department (Disposal Department)

[0183] 13 Endoscope camera (image acquisition unit)

[0184] 20 operating units

[0185] 21 Display (display unit)

[0186] 30 control unit

[0187] 33 Storage Device

[0188] 311 Operation Input Unit

[0189] 312 Occurrence Time Determination Department

[0190] 313 Partial motion image extraction unit

[0191] 314 Playback Department

[0192] 331 Video Recording Department

[0193] 332 Parameter storage unit

[0194] 333 Setting Storage Unit

Claims

1. A storage medium storing a computer program, wherein the computer program is configured to cause a computer to perform the following processing: Recording the surgical area images obtained by shooting the surgical area of ​​endoscopic surgery in time series; Judging whether there is bleeding exceeding a prescribed amount based on the surgical area image; If bleeding exceeding a predetermined amount is detected multiple times, thumbnails representing the respective bleeding scenes are displayed; When the displayed thumbnail is selected, a partial image of the bleeding scene corresponding to the selected thumbnail in the recorded surgical field image is played in the set playback mode, covering a time range from a time point before the bleeding starts to a time point after the bleeding starts.

2. The storage medium according to claim 1, The playback mode includes at least one of normal playback, slow playback, frame-by-frame playback, repeated playback, and reverse playback.

3. The storage medium according to claim 1, The playing mode includes a mode of repeatedly playing while switching at least two of normal playing, slow playing, frame-by-frame playing and reverse playing.

4. The storage medium according to any one of claims 1 to 3, The playback method includes a method in which the playback speed of a scene including the bleeding start time point is played slower than the playback speed of scenes before and after the bleeding start time point.

5. The storage medium according to claim 1, wherein the computer program is configured to cause the computer to perform the following processing: The images of the operation area captured during the endoscopic surgery are displayed in one display area, and partial images within the time range are played in another display area.

6. The storage medium according to claim 1, wherein the computer program is configured to cause the computer to perform the following processing: When bleeding is detected in the end area of ​​the surgical field image, information to the effect that bleeding has been detected is reported.

7. The storage medium according to claim 1, wherein the computer program is configured to cause the computer to perform the following processing: Displays a graph showing the time series changes in bleeding volume.

8. A method for playing an image, comprising: performing the following steps on a computer: Recording the surgical area images obtained by shooting the surgical area of ​​endoscopic surgery in time series; Judging whether there is bleeding exceeding a prescribed amount based on the surgical area image; If bleeding exceeding a predetermined amount is detected multiple times, thumbnails representing the respective bleeding scenes are displayed; When the displayed thumbnail is selected, a partial image of the bleeding scene corresponding to the selected thumbnail in the recorded surgical field image is played in the set playback mode, covering a time range from a time point before the bleeding starts to a time point after the bleeding starts.

Images