Program, information processing device, information processing system, and information processing method

Abstract

Provided are a program, an information processing device, an information processing system, and an information processing method which can automatically and accurately record information about a medical apparatus in an operation room.　The program executes, in a computer, the processing of: acquiring an operation space image that is imaged by an imaging means, which images an operation space, and includes monitor screens of a plurality of display devices distributed in the operation space; extracting information pertaining to the monitor screen of each of the display devices on the basis of the acquired operation space images; and storing, in time series, the information pertaining to the extracted monitor screen of each of the display devices.

Description

Program, information processing device, information processing system, and information processing method

[0001] The present invention relates to a program, an information processing device, an information processing system, and an information processing method.

[0002] For health management, it is known that personal physical data (e.g., weight, body temperature, blood pressure, etc.) measured using a measuring device is stored in a server via a network and managed by the server. For example, an information processing device is disclosed that executes a process of storing numerical information obtained from an image displayed on a display unit of a physical data measuring device for each type of measurement item of the physical data measuring device (Patent Document 1).

[0003] Patent No. 5601250

[0004] However, in an operating room where various medical devices are used, it is difficult to automatically and accurately record information about the various medical devices using the information processing device disclosed in Patent Document 1.

[0005] In addition, multiple measuring devices are used in operating rooms, and the measuring devices are often connected by cables to transfer data. In this case, cable noise generated in the cables can cause problems with the data being misread by each measuring device. Furthermore, since multiple cables are placed in the operating room, there is a concern that surgeons may trip when moving around the room.

[0006] The present invention has been made in consideration of the above circumstances, and aims to provide a program, an information processing device, an information processing system, and an information processing method that can automatically and accurately record information about medical equipment in an operating room.

[0007] A program according to one aspect of the present disclosure causes a computer to execute a process of acquiring a surgical space image captured by an imaging device that captures the surgical space and including the monitor screens of multiple display devices distributed throughout the surgical space, extracting information relating to the monitor screens of each display device based on the acquired surgical space image, and storing the extracted information relating to the monitor screens of each display device in chronological order.

[0008] An information processing device according to one aspect of the present disclosure includes an acquisition unit that acquires a surgical space image captured by an imaging device that captures an image of the surgical space and including the monitor screens of multiple display devices distributed throughout the surgical space, an extraction unit that extracts information relating to the monitor screens of each display device based on the surgical space image acquired by the acquisition unit, and a memory unit that stores the information relating to the monitor screens of each display device extracted by the extraction unit in chronological order.

[0009] An information processing system according to one aspect of the present disclosure comprises an imaging device that captures an image of a surgical space and obtains a surgical space image including the monitor screens of a plurality of display devices distributed in the surgical space, and an information processing device that communicates with the imaging device, wherein the information processing device comprises an acquisition unit that acquires the surgical space image obtained by the imaging device, an extraction unit that extracts information relating to the monitor screens of each display device based on the surgical space image acquired by the acquisition unit, and a memory unit that stores the information relating to the monitor screens of each display device extracted by the extraction unit in chronological order.

[0010] An information processing method according to one aspect of the present disclosure is an information processing method executed by a computer, and includes the steps of acquiring a surgical space image captured by an imaging device that captures an image of the surgical space and including monitor screens of multiple display devices distributed throughout the surgical space, extracting information regarding the monitor screens of each display device based on the acquired surgical space image, and storing the extracted information regarding the monitor screens of each display device in chronological order.

[0011] According to the present disclosure, information about medical equipment in an operating room can be automatically and accurately recorded.

[0012] 1 is a schematic diagram for explaining an overview of an information processing system according to embodiment 1. FIG. 1 is a block diagram showing an example of the configuration of the information processing system according to embodiment 1. FIG. 2 is an explanatory diagram showing an example of the record layout of an image DB. FIG. 3 is an explanatory diagram showing an example of the record layout of a measurement information DB. FIG. 4 is a flowchart showing the procedure of information processing executed by a control unit of an information processing device according to embodiment 1. FIG. 5 is a block diagram showing an example of the configuration of an information processing system according to embodiment 2. FIG. 6 is an explanatory diagram of an example of selection processing executed by a control unit of an information processing device according to embodiment 2. FIG. 7 is a flowchart showing the procedure of information processing executed by a control unit of an information processing device according to embodiment 3. FIG. 8 is a block diagram showing an example of the configuration of an information processing system according to embodiment 4. FIG. 9 is a flowchart showing the procedure of information processing executed by a control unit of an information processing device according to embodiment 4. FIG. 10 is a schematic diagram of an example of information on an abnormality displayed on a display unit. FIG. 11 is a schematic diagram of an example of information on an abnormality displayed on a display unit. FIG. 12 is a lookup table for determining an abnormality in an extracorporeal circulation device. FIG. 13 is a flowchart showing the procedure of information processing executed by a control unit of an information processing device according to embodiment 5. FIG. 14 is a block diagram showing an example of the configuration of a storage unit of an information processing device according to embodiment 6. FIG. 15 is a schematic diagram showing an example of the configuration of a behavior recognition model. FIG. 16 is an explanatory diagram showing another example of the record layout of a measurement information DB. 10 is a flowchart showing the procedure of information processing executed by a control unit of an information processing device according to a sixth embodiment.

[0013] (Embodiment 1) Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a schematic diagram for explaining an overview of an information processing system according to embodiment 1. Fig. 2 is a block diagram showing an example of the configuration of the information processing system according to embodiment 1. The information processing system according to embodiment 1 includes an imaging device 10 that captures images of an operating space in an operating room 100 in a hospital, and an information processing device 50 that communicates with the imaging device 10.

[0014] As shown in Fig. 1, a plurality of measuring devices 20, 21, 22, etc. are installed in an operating room 100 to measure biological information such as a patient's pulse rate, arterial blood oxygen saturation, body temperature, etc. Although three measuring devices are shown in Fig. 1, the number of measuring devices is not limited to three and may be any number other than three. Each of the measuring devices 20, 21, 22, etc. is equipped with a monitor (display device) 201, 211, 221, etc. that displays the measurement information. The monitors 201, 211, 221, etc. are distributed throughout the operating room 100.

[0015] The imaging device 10 is, for example, a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and captures images of the surgical space in the operating room 100 at predetermined time intervals and transmits the captured images (hereinafter referred to as surgical space images) including the display screens of multiple monitors 201, 211, 221, etc. to the information processing device 50. The imaging device 10 may continuously capture images of the surgical space in the operating room 100 during the surgery to sequentially obtain one frame of the surgical space image. The imaging device 10 and the information processing device 50 are connected via a network N, but may also be directly connected by a wired cable, or may transmit and receive information wirelessly via, for example, Bluetooth (registered trademark).

[0016] The information processing device 50 includes a control unit 51 that controls the entire device, a communication unit 52, a storage unit 53, a large-capacity storage unit 54, a reading unit 55, a display unit 56, and an operation unit 57. Each component is connected via a bus B. In the first embodiment, the information processing device 50 is a personal computer, but the information processing device 50 may also be a server device connected to a network N. The information processing device 50 may be installed in the operating room 100, or may be installed in a location different from the operating room 100 (for example, a management room, etc.).

[0017] The control unit 51 includes an arithmetic processing device such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit), and performs various information processing, control processing, and the like related to the information processing device 50 by reading and executing a control program 1P stored in the storage unit 53. Note that although the control unit 51 is described in FIG. 2 as being a single processor, it may be a multi-processor.

[0018] The communication unit 52 is a communication module for performing communication-related processing, and transmits and receives information between the imaging device 10 and the information processing device 50 via the network N.

[0019] The storage unit 53 includes memory elements such as RAM (Random Access Memory) and ROM (Read Only Memory), and stores the program 1P or data required for the control unit 51 to execute processing. The storage unit 53 also temporarily stores data required for the control unit 51 to execute arithmetic processing.

[0020] The mass storage unit 54 includes a recording medium such as a hard disk drive (HDD) or a solid state drive (SSD), etc. The mass storage unit 54 includes an image DB 541 and a measurement information DB 542, which will be described later.

[0021] In this embodiment, the storage unit 53 and the large-capacity storage unit 54 may be configured as an integrated storage device. The large-capacity storage unit 54 may also be configured with a plurality of storage devices. Furthermore, the large-capacity storage unit 54 may also be an external storage device connected to the information processing device 50.

[0022] The reading unit 55 reads the portable storage medium 1a, which may be a CD (Compact Disc)-ROM or a DVD (Digital Versatile Disc)-ROM. The control unit 51 may read the program 1P from the portable storage medium 1a via the reading unit 55 and store it in the mass storage unit 54. The control unit 51 may also download the program 1P from another computer via the network N or the like and store it in the mass storage unit 54. Furthermore, the control unit 51 may read the program 1P from the semiconductor memory 1b.

[0023] The display unit 56 is a liquid crystal display, an organic EL (electroluminescence) display, or the like, and displays various information according to instructions from the control unit 51. The operation unit 57 accepts operations from the operator and notifies the control unit 51 of the accepted operations. The operation unit 57 accepts operations from the operator using an input device such as a mechanical button or a touch panel provided on the surface of the display unit 56. The operation unit 57 may also be an input device such as a mouse and a keyboard, and these input devices may be configured to be detachable from the information processing device 50.

[0024] In the information processing device 50 of embodiment 1, time-series surgical space images captured by the imaging device 10 are acquired via the communication unit 52, the surgical space images are acquired by the control unit 51, and information regarding the screens of each monitor 201, 211, 221, etc. is extracted based on the acquired surgical space images, and the extracted information regarding each monitor screen is stored in chronological order.

[0025] The control unit 51 extracts information about the monitor screen by, for example, executing a monitor screen detection process and a measurement information extraction process displayed on the monitor screen for the surgical space image acquired via the communication unit 52, as described below. Regarding the monitor screen detection process, the control unit 51 detects the monitor screen from the surgical space image using, for example, a learning model that has been trained in advance by machine learning. The learning model is trained in advance to accept, for example, surgical space image data as input and output the coordinates of the monitor screen area included in the surgical space image. The learning model is trained in advance by machine learning using, for example, training data that associates the surgical space image data with coordinates indicating the image area where the monitor screen is displayed included in the surgical space image. The control unit 51 inputs the surgical space image captured by the imaging device 10 into the learning model and obtains the coordinates of the monitor screen area output by the learning model. Regarding the monitor screen detection process, for example, the control unit 51 may use conventional object detection technology to detect the monitor screen from the surgical space image captured by the imaging device 10. The object detection technology may be, for example, pattern matching, A-KAZE (Accelerated KAZE), SIFT (Scale Invariant Feature Transform), etc., and the monitor screen is detected by extracting features using a local feature extraction method. Furthermore, a table associating the frame shape of each monitor 201, 211, 221, ... with the displayed measurement items is stored in advance in the storage unit 53, and the control unit 51 can identify each monitor 201, 211, 221, ... and specify its measurement items according to the frame shape of each monitor 201, 211, 221, ... shown in the surgical space image. Note that the identification of each monitor 201, 211, 221, ... is not limited to the above method, and may be performed, for example, by attaching an identification marker (such as a colored label) to each monitor 201, 211, 221, ... in advance and associating the color of the marker with the measurement item.

[0026] The control unit 51 stores the monitor screen area coordinates as monitor image data in the image DB 541 of the mass storage unit 54. The control unit 51 may extract an image area showing the monitor screen from the surgical space image and store data of the extracted image area in the image DB 541 of the mass storage unit 54 as monitor image data. FIG. 3 is an explanatory diagram showing an example of a record layout of the image DB 541. As shown in FIG. 3, the image DB 541 includes an imaging date and time column, a surgical space image column, and a monitor image column. The imaging date and time column stores imaging date and time information of the image. The surgical space image column stores the acquired surgical space image. The monitor image column stores the monitor screen area coordinates indicated by four values: x coordinate, y coordinate, width (w), and height (h). The monitor image column may also store the extracted monitor screen image.

[0027] The control unit 51 uses the detection results of the monitor screen to extract information about the monitor screen from the image within the area coordinates. The information about the monitor screen includes various measurement information displayed on the monitor screen, and may be measurement values, text data, or graph data, or may be medical images such as IVUS (Intravascular Ultrasound) images or angiograms. In this embodiment, measurement values ​​are used as an example of measurement information. Regarding the measurement value extraction process, the control unit 51 recognizes numbers on the monitor screen displayed in the surgical space image using, for example, a learning model that has been trained in advance by machine learning. The learning model receives monitor image data, including surgical space image data and monitor screen area coordinates, as input, and is trained in advance to output the probability that each number displayed on the monitor screen corresponds to a digit "0" to "9." Note that the monitor screen area coordinates used in the measurement value extraction process are not limited to coordinates output by the aforementioned learning model, but may also be coordinates input by the operator via the operation unit 57 while observing the surgical space image. Alternatively, data on the monitor screen image area extracted from the surgical space image may be input to the learning model. The learning model is machine-learned using training data that associates, for example, monitor image data with labels indicating the probability that each number depicted in the image within the area coordinates corresponds to a digit "0" to a digit "9." In the first embodiment, the control unit 51 inputs the monitor image data into the learning model and obtains the probability that each digit corresponds to a digit "0" to a digit "9" output from the learning model. From the probabilities, numbers corresponding to probabilities higher than a predetermined threshold (e.g., 80%) are identified to extract measurement values ​​on the monitor screen. The measurement value data extracted by the control unit 51 is stored in the measurement information DB 542 of the mass storage unit 54. This improves the accuracy of the measurement values. FIG. 4 is an explanatory diagram showing an example of a record layout of the measurement information DB 542. As shown in FIG. 4, the measurement information DB 542 includes an imaging date / time column and a measurement information column. The imaging date / time column stores the imaging date / time of the image. The measurement information column stores the recognized measurement values ​​for each monitor 201, 211, 221, etc.The measurement information DB 542 may include other columns such as a monitor screen column and an abnormality type column, as will be described later.

[0028] Furthermore, when the probability of each of the digits "0" to "9" output from the learning model for a certain digit is equal to or less than a predetermined threshold (for example, 50%), the control unit 51 determines that the classification rate for the digit is low and does not store the measurement value indicated by the digit in the measurement information DB 542. The control unit 51 may use the measurement information stored in the measurement information DB 542 to create a trend graph and display it on the display unit 56.

[0029] FIG. 5 is a flowchart showing the procedure of information processing executed by the control unit 51 of the information processing device 50 according to the first embodiment.

[0030] The control unit 51 acquires an image of the surgical space captured by the imaging device 10 (step S1), detects the monitor screen from the acquired image (step S2), and stores the data of the detected monitor image in the image DB 541 (step S3).

[0031] The control unit 51 extracts information about the monitor screen from the monitor image data stored in the image DB 541 (step S4), and stores the extracted information together with the image capture date and time in the measurement information DB 542 (step S5).

[0032] The control unit 51 determines whether there are any unprocessed monitor images, and if it determines that there are any unprocessed monitor images (step S6: YES), the control unit 51 returns the process to step S4. If it determines that there are no unprocessed monitor images (step S6: NO), the control unit 51 ends the process.

[0033] In the first embodiment, the surgical space in the operating room 100 is imaged, and information about the monitor screen is extracted and stored from the surgical space image, thereby automatically recording measurement information for each measuring device 20, 21, 22, etc. within the imaging area. Centralized management of information from multiple medical devices reduces the burden on medical personnel and avoids human error due to manual recording. Furthermore, since no wiring is required between each measuring device 20, 21, 22, etc. and the information processing device 50, the effort required for managing medical device wiring is eliminated, reducing various risks associated with complex wiring in medical environments (e.g., misidentification due to cable noise, tangled cables, and interference when moving measuring devices within the operating room or when the surgeon moves around). Furthermore, even if communication protocols are not consistent with each other, newly installed medical devices do not require extensive software modifications.

[0034] Furthermore, the recorded measurement information can be used to graph (trend graph, etc.) time-series data of patient behavior, for example. Because the recorded measurement information does not contain outliers due to cable noise or human error, highly accurate trend information can be obtained when creating a trend graph using the extracted measurement information. Furthermore, if the identified measurement information is unreliable, the measurement information is not recorded, so an accurate trend graph can be created without using unreliable identification results.

[0035] In addition to the monitor screen, the surgical space image may also show the blood reservoir and each circuit of the extracorporeal circulation device. By extracting information about the fluid level in the blood reservoir from the surgical space image using the information processing system according to the first embodiment, the fluid level in the extracorporeal circulation device can be easily monitored.

[0036] (Embodiment 2) An information processing system according to embodiment 2 differs from embodiment 1 in that multiple imaging devices are installed, and therefore the following mainly describes this difference. Since the other configurations and effects are the same as those of embodiment 1, the same reference numerals are used for corresponding parts and detailed descriptions are omitted.

[0037] Fig. 6 is a block diagram showing an example of the configuration of an information processing system according to embodiment 2. Image capturing devices 10, 11, 12, ... are installed in different locations in the operating room 100, capture images of the operating space in the operating room 100 from different angles, obtain multiple images of the operating space, and transmit them to the information processing device 50. Although Fig. 6 shows three image capturing devices, the number of image capturing devices is not limited to three and may be two, four, or more.

[0038] In the information processing device 50, the control unit 51 acquires multiple surgical space images via the communication unit 52, selects one image from the multiple surgical space images depending on the installation location of each imaging device 10, 11, 12, etc., and extracts information related to the monitor screen from the selected image.

[0039] The storage unit 53 stores a priority table (not shown) in which the priority of each imaging device is preset for each monitor 201, 211, 221, etc. based on the installation location of each imaging device 10, 11, 12, etc. When detecting the monitor screen, the control unit 51 uses the surgical space image captured by the imaging device with the highest priority based on the priority table for the monitor to be detected. Note that the setting of priorities is not limited to this, and priorities may be set for each surgical space image by comparing the accuracy of each surgical space image captured at the same time by each imaging device 10, 11, 12, etc.

[0040] If an obstruction is captured in the surgical space image captured by the highest-priority imaging device, making it impossible to detect the monitor screen or recognize the measurement information, the control unit 51 re-performs the process of detecting the monitor screen using the surgical space image captured by the next-highest-priority imaging device. The control unit 51 recognizes the obstruction using, for example, a technology called "YOLO," which detects objects such as people from images using a deep learning model. Figure 7 is an explanatory diagram of an example of a selection process performed by the control unit 51 of the information processing device 50 according to embodiment 2. Figure 7 shows the recognition status of the monitor 201 based on the surgical space images captured by each imaging device 10, 11, and 12 at each time point, with a circle indicating recognition possible and a cross indicating recognition impossible. As shown in Figure 7, the measurement information displayed on the monitor 201 cannot be recognized from the surgical space image captured by the imaging device 10 at time T2. The control unit 51 selects the surgical space image captured by the imaging device 11 and re-performs the process of detecting the monitor screen.

[0041] FIG. 8 is a flowchart showing the procedure of information processing executed by the control unit 51 of the information processing device 50 according to the second embodiment.

[0042] The control unit 51 acquires a plurality of surgical space images captured by the imaging devices 10, 11, 12, . . . (step S21), and stores the acquired images in the image DB 541 of the mass storage unit 54 (step S22).

[0043] The control unit 51 selects one image from the plurality of surgical space images based on the priority table (step S23), and detects the monitor screen from the selected surgical space image (step S24). The control unit 51 determines whether the monitor screen is obstructed by an obstruction (step S25). If it determines that the monitor screen is obstructed (step S25: YES), the control unit 51 selects another image from the plurality of surgical space images based on the priority table (step S26), and returns the process to step S24.

[0044] When the control unit 51 determines that the object is not blocked (step S25: NO), the control unit 51 stores data of the detected monitor image in the image DB 541 (step S27).

[0045] The control unit 51 extracts information about the monitor screen based on the monitor image data stored in the image DB 541 (step S28), and stores the extracted information together with the imaging time in the measurement information DB 542 of the large-capacity memory unit 54 (step S29).

[0046] The control unit 51 determines whether there are any unprocessed monitor images (step S30), and if it determines that there are any unprocessed monitor images (step S30: YES), the process returns to step S23. If it determines that there are no unprocessed monitor images (step S30: NO), the control unit 51 ends the process.

[0047] In the second embodiment, multiple imaging devices 10, 11, 12, etc. are installed. Therefore, if measurement information cannot be recognized from one surgical space image, the measurement information can be supplemented using other surgical space images. Therefore, medical device information can be recorded with high accuracy. Furthermore, by using surgical space images captured by highly reliable imaging devices for each monitor screen, medical device information can be recorded with high accuracy.

[0048] (Embodiment 3) The information processing system according to embodiment 3 differs from embodiment 1 in that the information processing device 50 recognizes an obstruction between each monitor screen and the imaging device 10, and therefore the following mainly describes this difference. The other configurations and effects are the same as those of embodiment 1, so the same reference numerals are used for corresponding parts and detailed descriptions are omitted.

[0049] The control unit 51 recognizes an obstruction between the monitor screen and the imaging device 10, and extracts information about the monitor screen excluding the area of ​​the recognized obstruction. Furthermore, when the control unit 51 recognizes the surgeon as an obstruction, it specifies the position of the recognized surgeon.

[0050] The control unit 51 recognizes occlusions using, for example, a technology called "YOLO," which uses a deep learning model to detect objects such as people from images. For example, if the display screen of the monitor 201 is blocked by the surgeon in a surgical space image captured by the imaging device 10 at a certain point in time, the control unit 51 identifies the surgeon's position as a point on a line between the imaging device 10 and the monitor 201. Note that even if a portion of the display screen of the monitor 201 is blocked by the surgeon, information may be extracted from an unblocked area. In this case, the control unit 51 identifies the surgeon's pixel area on the display screen using a segmentation-based machine learning model such as SegNet. The control unit 51 extracts an image of a pixel area other than the identified pixel area on the display screen. The control unit 51 recognizes measurement information for the area using a machine learning model that performs character recognition or pattern matching.

[0051] FIG. 9 is a flowchart showing the procedure of information processing executed by the control unit 51 of the information processing device 50 according to the third embodiment.

[0052] In the information processing device 50, the control unit 51 acquires an operation space image captured by the imaging device 10 (step S41), and determines whether the monitor screen in the acquired operation space image is obstructed (step S42). If the control unit 51 determines that the monitor screen is obstructed (step S42: YES), it removes the obstructed screen area (step S43), detects the monitor screen for the unobstructed area (step S44), and stores the detected monitor image data in the image DB 541 (step S45).

[0053] The control unit 51 extracts information about the monitor screen from the monitor image data stored in the image DB 541 (step S46), and stores the extracted information together with the image capture time in the measurement information DB 542 (step S47).

[0054] The control unit 51 determines whether the obstruction is the surgeon (step S48). If the control unit 51 determines that the obstruction is the surgeon (step S48: YES), it identifies the surgeon's position as between the monitor 201 and the image capture device 10, between the monitor 211 and the image capture device 10, or between the monitor 221 and the image capture device 10 (step S49), and stores the identified position together with the image capture date and time in the measurement information DB 542 (step S50). The control unit 51 determines whether there are any unprocessed monitor images (step S51). If it determines that there are any unprocessed monitor images (step S51: YES), it returns the process to step S46. If it determines that there are no unprocessed monitor images (step S51: NO), the control unit 51 ends the process.

[0055] When the control unit 51 determines that the obstructing object is not the surgeon (step S48: NO), the control unit 51 shifts the processing to step S51.

[0056] In the third embodiment, it is possible to extract measurement information from a monitor screen that is not blocked by an obstruction, and also to identify and record the position of the operator.

[0057] (Embodiment 4) The information processing system according to embodiment 4 differs from embodiment 1 in that the information processing device 50 further acquires actual measurements from each measuring device, and therefore the following mainly describes this difference. The other configurations and effects are the same as those of embodiment 1, so corresponding parts are given the same reference numerals and detailed descriptions are omitted.

[0058] 10 is a block diagram showing an example of the configuration of an information processing system according to embodiment 4. A plurality of measuring devices 20, 21, 22, ... transmit actual measurement values ​​they have measured to an information processing device 50 via a network N. The measuring devices 20, 21, 22, ... and the information processing device 50 are connected via the network N, but they may also be directly connected by a wired cable, or information may be transmitted and received wirelessly via Bluetooth (registered trademark), for example.

[0059] In the information processing device 50, the control unit 51 compares actual measurement values ​​measured by the measuring devices 20, 21, 22, ... with measurement values ​​extracted from the surgical space image (hereinafter referred to as extracted measurement values), and if the difference between the two is equal to or greater than a predetermined threshold, stores the extracted measurement values ​​in association with the monitor screen from which the measurement was extracted and the imaging date and time in the measurement information DB 542 of the mass storage unit 54. Here, the predetermined threshold may be set appropriately as needed and stored in advance in the storage unit 53. Note that if the information about the monitor screen extracted from the surgical space image is graph data, the control unit 51 may convert the graph data into a plurality of extracted measurement values ​​and compare the converted extracted measurement values ​​with the actual measurement values.

[0060] FIG. 11 is a flowchart showing the procedure of information processing executed by the control unit 51 of the information processing device 50 according to the fourth embodiment.

[0061] The control unit 51 acquires the actual measurement value (step S61) and compares the acquired actual measurement value with the extracted measurement value corresponding to the same imaging date and time as the acquisition date and time stored in the measurement information DB 542 (step S62).

[0062] The control unit 51 determines whether the difference between the two values ​​is equal to or greater than a predetermined threshold (step S63). If it determines that the difference is equal to or greater than the predetermined threshold (step S63: YES), the control unit 51 stores the monitor screen displaying the extracted measurement value in association with the imaging date and time in the measurement information DB 542 (step S64).

[0063] The control unit 51 outputs the warning information to the display unit 56 (step S65), and ends the process.

[0064] If it is determined that the value is not equal to or greater than the predetermined threshold value (step S63: NO), the control unit 51 ends the process.

[0065] 12 and 13 are schematic diagrams of examples of warning information displayed on the display unit 56. The control unit 51 creates trend graphs based on the extracted measurement values ​​and the actual measurement values, and displays them on the display unit 56. In FIGS. 12 and 13, the horizontal axis represents time, the vertical axis represents measurement values, ▲ marks represent actual measurement values, and black circles represent extracted measurement values. At time 6, the difference between the actual measurement values ​​and the extracted measurement values ​​exceeds a predetermined threshold, so an arrow indicating the difference between the actual measurement values ​​and the extracted measurement values ​​is displayed on the trend graph, as shown in FIG. 12. In trend graphs created thereafter, the actual measurement values ​​and the extracted measurement values ​​at time 6 are highlighted with circles, as shown in FIG. 13.

[0066] In the fourth embodiment, by comparing the actual measurement value with the extracted measurement value, it is possible to determine whether or not there is an abnormality in the measurement device. If there is an abnormality in the measurement device, warning information is displayed, making the abnormality situation clear at a glance. Furthermore, it is also possible to preempt extracted measurement values ​​that are determined to be abnormal, which is something that has not been considered in the prior art.

[0067] (Embodiment 5) The information processing system according to embodiment 5 differs from embodiment 1 in that the information processing device 50 detects abnormalities in a patient (subject), and therefore the following mainly describes this difference. The other configurations and effects are the same as those of embodiment 1, so corresponding parts are denoted by the same reference numerals and detailed descriptions are omitted.

[0068] In the information processing device 50, the control unit 51 detects abnormalities in a patient based on various measurement information stored in the measurement information DB 542. For example, a table in which values ​​indicating normal ranges are set for each measurement item may be stored in advance in the storage unit 53, and by referring to the table, an abnormality may be determined if the patient's measurement information is outside the normal range. Alternatively, a table in which information indicating normal trends is set for each measurement item may be stored in advance in the storage unit 53, and by referring to the table, an abnormality may be determined if the trend in the patient's measurement information differs from the normal trend.

[0069] When the control unit 51 determines that an abnormality has occurred, it outputs information about the abnormality to the display unit 56. The information about the abnormality may be information presenting abnormal measurement values ​​or trend information, or information presenting the measurement item in which the abnormality has occurred. Furthermore, when the control unit 51 determines that an abnormality has occurred, it outputs advice information for resolving the abnormality to the display unit 56.

[0070] In this embodiment, the measurement device is an extracorporeal circulation device. FIG. 14 shows a lookup table for determining abnormalities in an extracorporeal circulation device. It stores trend information for various measurement values ​​from the extracorporeal circulation device, the type of abnormality, and advice information for the abnormality in association with each other. The lookup table is pre-stored in, for example, the storage unit 53. As shown in FIG. 14 , the lookup table includes an abnormality type column, a measurement item column, and an advice column. The abnormality type column records abnormality types A, B, C, and D. The measurement item column records changes in flow rate or pressure per unit time for each measurement item K1, K2, K3, and K4. The advice column records advice for each type of abnormality. In the lookup table shown in FIG. 14 , the intensity of change for each measurement item is represented by x, 2x, and 3x in the order of "low," "medium," and "high." The direction of change is represented by "+" indicating an upward trend and "-" indicating a downward trend. The intensity of change can be preset as needed.

[0071] Specifically, the control unit 51 calculates the change in flow rate or pressure per unit time for each measurement item, stores the change in flow rate or pressure per unit time in, for example, the mass storage unit 54, and determines the presence or absence of an abnormality based on the lookup table. If an abnormality is determined to exist, the control unit 51 identifies the type of abnormality and outputs advice information for resolving the abnormality to the display unit 56. For example, if the measurement items K1, K2, K3, and K4 each show a downward trend, the control unit 51 determines that an abnormality A exists and displays (outputs) a message on the display unit 56 stating, "There is a possibility of a thrombus in the blood removal cannula," as advice information for resolving the abnormality. The control unit 51 may also display the type of abnormality along with the advice information on the display unit 56. Furthermore, the control unit 51 may combine trend information of various measurements taken by the extracorporeal circulation device with advice information and display it on the display unit 56. Alternatively, the control unit 51 may display an image related to the lookup table on the display unit 56.

[0072] When the control unit 51 detects that abnormality information is being displayed on the monitor screen, the control unit 51 may determine that there is a possibility that the patient has an abnormality. When the control unit 51 determines that there is an abnormality, the control unit 51 may store the monitor screen on which the abnormality information is being displayed in association with the imaging date and time in the measurement information DB 542.

[0073] FIG. 15 is a flowchart showing the procedure of information processing executed by the control unit 51 of the information processing device 50 according to the fifth embodiment.

[0074] The control unit 51 acquires measurement information for each measurement item (step S71), calculates the change in the measurement information for each measurement item (step S72), and stores the calculated change in the measurement information in the mass storage unit 54 (step S73). The control unit 51 determines whether there is an abnormality in the measurement information based on the lookup table (step S74), and if it determines that there is no abnormality (step S74: NO), it determines that the information is normal (step S75), and ends the process.

[0075] If the control unit 51 determines that an abnormality has occurred (step S74: YES), it reads advice information for resolving the abnormality from the reference table (step S76), outputs it to the display unit 56 (step S77), and terminates the processing.

[0076] In the sixth embodiment, if an abnormality is found in the information of a medical device, the details of the abnormality can be recorded and a countermeasure can be presented according to the abnormality, thereby reducing the burden on medical staff.

[0077] (Embodiment 6) The information processing system according to embodiment 6 differs from embodiment 1 in that the information processing device 50 further performs behavior recognition processing based on an image of the surgeon captured in the surgical space image, and therefore, the following mainly describes this difference. Since the other configurations and effects are the same as those of embodiment 1, the same reference numerals are used for corresponding parts, and detailed descriptions thereof will be omitted.

[0078] 16 is a block diagram showing an example of the configuration of the storage unit 53 of the information processing device 50 according to embodiment 6. The storage unit 53 of the information processing device 50 further stores a behavior recognition model M, which is a trained model trained by machine learning. The behavior recognition model M is expected to be used as a program module that functions as part of artificial intelligence software.

[0079] The imaging device 10 captures images of the surgical space and sequentially acquires one frame of surgical space image. The imaging device 10 is configured to acquire, for example, 60, 30, or 15 frames of surgical space images per second, and the surgical space images acquired by the imaging device 10 are sequentially transmitted to the information processing device 50.

[0080] In addition to extracting information about the monitor screen, the control unit 51 performs behavior recognition processing to recognize the behavior of the surgeon present in the surgical space captured by the imaging device 10, and stores the information about the recognized behavior of the surgeon in association with the information about the monitor screen in chronological order. Regarding the behavior recognition processing, the control unit uses a behavior recognition model M stored in the storage unit 53 to recognize the behavior of the surgeon from the surgical space image. Examples of the surgeon's behavior include, but are not limited to, incision, suturing, injection, endoscopic operation, catheter insertion, balloon dilation, use of an electric scalpel, hemostasis, and waiting.

[0081] FIG. 17 is a schematic diagram showing an example configuration of the behavior recognition model M. The behavior recognition model M of this embodiment is configured, for example, as shown in FIG. 17 , by combining an object detection algorithm (neural network) and LSTM (Long Short-Term Memory). Examples of object detection algorithms that can be used include R-CNN (Regions with Convolution Neural Network), Fast R-CNN, Faster R-CNN, Mask R-CNN, SSD (Single Shot Multibook Detector), and YOLO (You Only Look Once). The behavior recognition model M is a trained model that receives time-series images of the surgical space captured by the surgeon as input, calculates whether the captured surgeon is performing a predetermined behavior based on the input images, and outputs the calculation results.

[0082] The action recognition model M shown in FIG. 17 includes a convolutional layer, a pooling layer, an LSTM layer, and a fully connected layer. In the action recognition model M of this embodiment, multiple frames of surgical space images over a predetermined unit time are input via an input layer. The surgical space images of each frame input via the input layer are subjected to filtering or other processing in the convolutional layer to extract image features and generate a feature map. The pooling layer reduces the position sensitivity of the image features, enabling recognition even when the position of the feature deviates from the position of the feature map. Multiple convolutional layers and pooling layers are repeatedly provided, and the feature maps generated by multiple convolutional layers and pooling layers are input to a fully connected layer, then to an LSTM layer, and then to a fully connected layer. Each of the fully connected layers and LSTM layers calculates output values ​​of nodes in each layer using various functions, thresholds, etc. based on the input feature map. The calculated output values ​​of the nodes in each layer are sequentially input to nodes in the subsequent layers, and the final fully connected layer provides each output value to each output node in the output layer. In the action recognition model M, the number of input nodes in the input layer, and the number of convolutional layers, pooling layers, and fully connected layers are not limited to the example shown in Fig. 17. Note that an image whose skeleton has been extracted using VisonPose or the like may be input as the input image.

[0083] The behavior recognition model M is trained using training data including correct labels indicating each behavior and captured images of people performing each behavior. When a captured image included in the training data is input, the behavior recognition model M is trained to output a discriminant label corresponding to the behavior indicated by the correct label included in the training data. In the training process, the behavior recognition model M is trained to optimize the weighting coefficients and function coefficients that connect the nodes in each layer. As a result, when a captured image is input, a trained behavior recognition model M is obtained that is trained to identify the behavior performed by the person in the captured image and output information about the behavior as a result of the identification. Note that, as for the captured images used in the training data, images of people performing various behaviors can be used as images of each behavior.

[0084] In the behavior recognition model M of this embodiment, for example, a plurality of frames of captured images sequentially captured by the imaging device 10 may be input, or a plurality of frames of captured images extracted at predetermined time intervals (for example, one second) from the captured images captured by the imaging device 10 may be input. Learning of the behavior recognition model M is performed by another learning device, but may also be performed by the information processing device 50. The behavior recognition model M is not limited to a neural network such as that shown in Fig. 17 , and learning models constructed by various machine learning algorithms may be used.

[0085] The control unit 51 stores information about the behavior of the surgeon output from the behavior recognition model M in the measurement information DB 542. Furthermore, as in the fourth and fifth embodiments described above, the control unit 51 determines whether or not there is an abnormality and identifies the type of abnormality, and if it determines that there is an abnormality, outputs abnormality information including, for example, the behavior of the surgeon and the type of abnormality to the display unit 56.

[0086] FIG. 18 is an explanatory diagram showing another example of the record layout of the measurement information DB 542. The measurement information DB 542 includes an imaging date / time column, a measurement information column, an abnormality flag column, a surgeon action column, and an abnormality type column. The imaging date / time column stores image imaging date / time information. The measurement information column stores extracted measurement information. The abnormality flag column stores a flag indicating the presence or absence of an abnormality. For example, if an abnormality is determined to exist, such as when it is determined that the actual measurement value does not match the extracted measurement value as in the fourth embodiment, or when it is determined that the measurement information is abnormal as in the fifth embodiment, a "1" is stored. If an abnormality is determined to exist, a "0" is stored. The surgeon action column stores information regarding the surgeon's actions output by the aforementioned action recognition model M. The abnormality type column stores information regarding the type of abnormality. For example, if it is determined that there is no abnormality, "none" is stored. If the type of abnormality can be identified as in the fifth embodiment, the type of abnormality is stored. Furthermore, although it was determined in the fifth embodiment that there was no abnormality, if the abnormality flag is 1 and the surgeon used a surgical tool that is prone to generating noise, such as an electric scalpel, the type of abnormality is determined to be "noise." The measurement information DB 542 may further store surgical space images and monitor images.

[0087] FIG. 19 is a flowchart showing the procedure of information processing executed by the control unit 51 of the information processing device 50 according to the sixth embodiment.

[0088] The control unit 51 acquires a surgical space image (step S81) and inputs the image to the behavior recognition model M (step S82). The control unit 51 acquires information related to the behavior of the surgeon output from the behavior recognition model M (step S83), and stores the acquired information in the measurement information DB 542 in association with the measurement information (step S84).

[0089] The control unit 51 determines whether or not there is an abnormality flag (step S85), and if it determines that there is no abnormality flag (step S85: NO), it stores the type of abnormality as "none" (step S86) and terminates the processing.

[0090] If the control unit 51 determines that an abnormality flag has been set (step S85: YES), it determines whether the type of abnormality has been identified (step S87), and if it determines that the type of abnormality has been identified (step S87: YES), it stores the identified type of abnormality in the abnormality type column of the measurement information DB 542 (step S88), outputs the abnormality information to the display unit 56 (step S89), and terminates the processing.

[0091] If the control unit 51 determines that the type of abnormality cannot be identified (step S87: NO), it determines whether the surgeon's behavior is an behavior that is likely to produce noise (step S90), and if it determines that the surgeon's behavior is an behavior that is likely to produce noise (step S90: YES), it stores ``noise'' as the type of abnormality in the abnormality type column of the measurement information DB 542 (step S91), and returns the processing to step S89.

[0092] If the control unit 51 determines that the surgeon's behavior is not one that is likely to produce noise (step S90: NO), it stores ``Unknown'' as the type of abnormality in the abnormality type column of the measurement information DB 542 (step S92), outputs a message to the display unit 56 stating ``Please check the equipment'' (step S93), and terminates the processing.

[0093] In the sixth embodiment, the type of action of the surgeon is identified and recorded together with the measurement information, thereby making it possible to assist in identifying the cause of an abnormality in the measurement information.

[0094] (Embodiment 7) The information processing system according to embodiment 7 differs from embodiment 1 in that the information processing device 50 does not store the detection results of the monitor screen, and therefore, the following mainly describes this difference. Since the other configurations and effects are the same as those of embodiment 1, the same reference numerals are used for corresponding parts, and detailed descriptions thereof will be omitted.

[0095] FIG. 20 is a flowchart showing the procedure of information processing executed by the control unit 51 of the information processing device 50 according to the seventh embodiment.

[0096] The control unit 51 acquires an operation space image captured by the imaging device 10 (step S11), and detects the monitor screen from the acquired image (step S12).

[0097] The control unit 51 extracts information about the monitor screen from the detected monitor screen (step S13), stores the extracted information together with the image capture date and time in the measurement information DB 542 (step S14), and ends the process.

[0098] In the seventh embodiment, the results of the detection on the monitor screen are not stored but are directly used in the extraction process of the measurement information, thereby improving the real-time performance of the process. Furthermore, since the capacity for storing the results of the detection on the monitor screen is not required, there is no need to worry about the capacity of the mass storage unit 54.

[0099] The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed. The embodiments disclosed herein are illustrative in all respects and should be considered as not limiting. The scope of the present invention is defined by the claims, not by the meanings described above, and is intended to include all modifications within the meaning and scope of the claims, including equivalents.

[0100] 10, 11, 12 Imaging device 100 Operating room 1P Program 20, 21, 22 Measuring device 201, 211, 221 Monitor 50 Information processing device 51 Control unit 52 Communication unit 53 Storage unit 54 Large-capacity storage unit 541 Image DB 542 Measurement information DB 55 Reading unit 56 Display unit 57 Operation unit M Behavior recognition model

Claims

1. Acquire a surgical space image including monitor screens of a plurality of display devices that are distributed in the surgical space, the image being captured by an imaging device that captures an image of the surgical space; Based on the acquired surgical space image, information about the monitor screen of each display device is extracted, The extracted information about the monitor screen of each display device is stored in chronological order. A program that causes a computer to carry out processing.

2. The measured values, text data, graph data, or medical images displayed on the monitor screen are extracted for each monitor screen as information related to the monitor screen. The program according to claim 1.

3. Creating and / or displaying a trend graph using the extracted monitor screen information The program according to claim 1.

4. acquiring actual measurements from a measurement device connected to the plurality of display devices; Comparing the extracted measurements with the actual measurements taken; If the difference between the extracted measurement value and the actual measurement value is equal to or greater than a predetermined threshold, the monitor screen from which the measurement value was extracted and the image capture date and time are stored in association with each other. The program according to claim 1.

5. If the difference between the extracted measurement value and the actual measurement value is equal to or greater than a predetermined threshold, a warning message is output. The program according to claim 2.

6. Accepting area coordinates in the surgical space image corresponding to the monitor screen; Extracting information about the monitor screen from the image in the received area coordinates The program according to claim 1.

7. Recognizing an obstacle between the monitor screen and the imaging device; Extract information about the monitor screen by removing the recognized occluded areas The program according to claim 1.

8. Identifies the position of the caster if the caster is recognized as an obstruction The program according to claim 7.

9. The imaging device includes a first imaging device and a second imaging device installed at different locations, The surgical space image includes a first image captured by the first imaging device and a second image captured by the second imaging device, When extracting information about the monitor screen, the first image and the second image are selectively used depending on installation locations of the first imaging device and the second imaging device. The program according to claim 1.

10. When an obstruction is recognized on the monitor screen of the first image, the monitor screen in the second image is used instead of the monitor screen of the first image, and information related to the monitor screen is extracted. The program according to claim 9.

11. extracting the plurality of types of measurement values ​​displayed on the monitor screen as information related to the monitor screen; Based on the extracted multiple measurement values, the presence or absence of abnormalities in the subject is determined, If an abnormality is detected, information about the abnormality is output. The program according to claim 1.

12. If an abnormality is detected, advice on how to resolve the abnormality is output. The program according to claim 11.

13. The surgical space image further includes an image of a surgeon who was present in the surgical space captured by the imaging device, The acquired surgical space image is input to a model that has been trained to output behavioral information regarding the surgeon's behavior when a surgical space image is input, and the model is made to output the behavioral information of the surgeon; The outputted action information of the surgeon and the information on the monitor screen are stored in chronological order in association with each other.

13. A program according to any one of claims 1 to 12.

14. If an abnormality is found in the extracted monitor screen information, the type of abnormality is identified based on the operator's behavior information. The program according to claim 13.

15. An acquisition unit that acquires a surgical space image including monitor screens of a plurality of display devices that are distributed in the surgical space and that is captured by an imaging device that captures an image of the surgical space; An extraction unit that extracts information about the monitor screen of each display device based on the surgical space image acquired by the acquisition unit; a storage unit that stores information about the monitor screen of each display device extracted by the extraction unit in chronological order; An information processing device comprising:

16. An imaging device that captures an image of the surgical space and obtains an image of the surgical space including monitor screens of a plurality of display devices that are distributed in the surgical space; an information processing device that communicates with the imaging device; Equipped with The information processing device includes: An acquisition unit that acquires an operation space image obtained by the imaging device; An extraction unit that extracts information about the monitor screen of each display device based on the surgical space image acquired by the acquisition unit; a storage unit that stores information about the monitor screen of each display device extracted by the extraction unit in chronological order; An information processing system comprising:

17. A step of acquiring a surgical space image including monitor screens of a plurality of display devices that are distributed in the surgical space and captured by an imaging device that captures an image of the surgical space; Extracting information about the monitor screen of each display device based on the acquired surgical space image; storing the extracted information about the monitor screen of each display device in chronological order; A computer-implemented information processing method, comprising: