Medical information processing device, system, and X-ray diagnostic device

Abstract

To detect an abnormality that may occur in an inspection room before it occurs.SOLUTION: A medical information processing apparatus according to an embodiment comprises an acquisition unit and a detection unit. The acquisition unit acquires images obtained by picking up images of a space where procedures for an analyte are performed, and non-image information on at least one of the procedures, analyte, and space. The detection unit detects a state where an abnormality in the space is highly likely to occur, based on the image acquired by the acquisition unit, the non-image information acquired by the acquisition unit, and first correspondence information. The first correspondence information indicates the correspondence between the plurality of images obtained by picking up images of the space, the non-image information on at least one of the procedures, analyte, and space, and the abnormality that is highly likely to occur in the space.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] An embodiment of the present invention relates to a medical image processing device, a system, and an X-ray diagnostic apparatus. [Background technology]

[0002] Conventionally, in an examination room where diagnosis or treatment is performed on a subject, a plurality of medical staff members perform work using various medical equipment. In such an examination room, a technique is known for preventing contact between an X-ray diagnostic device that images a subject and the subject or the surgeon. However, in an examination room, not only contact between the X-ray diagnostic device and the subject or the surgeon, but also various other abnormalities may occur, and it may be difficult to predict these abnormalities in advance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. US2017 / 0181720 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to detect abnormalities that may occur in an examination room before they occur. [Means for solving the problem]

[0005] A medical image processing apparatus according to an embodiment includes an acquisition unit and a detection unit. The acquisition unit acquires images of a space in which a procedure is being performed on a subject, and non-image information related to at least one of the procedure, the subject, or the space. The detection unit detects a state in which an abnormality is likely to occur in the space based on the images acquired by the acquisition unit, the non-image information acquired by the acquisition unit, and first correspondence information. The first correspondence information is based on a plurality of images of the space and , nonIt shows the correspondence between image information and anomalies that are likely to occur in space. The non-image information includes voice data including instructions issued from outside the space to the surgeon in the space. The detection unit detects a state in which an abnormality is likely to occur when an inconsistency occurs between the image and the instructions included in the voice data. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram showing an example of a configuration of a system according to the first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of an examination room according to the first embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of detection of a risk of an abnormality occurring and notification of the risk according to the first embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of a method for generating a first trained model according to the first embodiment. [Diagram 5] FIG. 5 is a diagram illustrating an example of a method for generating the second trained model according to the first embodiment. [Figure 6] FIG. 6 is a flowchart showing an example of the flow of the abnormality detection process according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, embodiments of a medical image processing apparatus, a system, and an X-ray diagnostic apparatus will be described in detail with reference to the drawings.

[0008] (Embodiment) 1 is a block diagram showing an example of the configuration of a system S according to this embodiment. As shown in FIG.

[0009] The X-ray diagnostic apparatus 10 includes an X-ray high voltage device 11, an X-ray tube 12, an X-ray aperture 13, a tabletop 14, a C-arm 15, an X-ray detector 16, a C-arm rotation / movement mechanism 17, a tabletop movement mechanism 18, a C-arm / tabletop mechanism control circuit 19, an aperture control circuit 20, a processing circuit 21, an input interface 22, displays 23a and 23b, and a memory circuit 24. The X-ray diagnostic apparatus 10 is, for example, an X-ray angiography apparatus. In this embodiment, the X-ray diagnostic apparatus 10 is an example of a medical information processing apparatus.

[0010] Of the components of the X-ray diagnostic apparatus 10, the X-ray high voltage device 11, the X-ray tube 12, the X-ray aperture 13, the tabletop 14, the C-arm 15, the X-ray detector 16, the C-arm rotation / movement mechanism 17, the tabletop movement mechanism 18, the C-arm / tabletop mechanism control circuit 19, the aperture control circuit 20, and the display 23b are installed in the examination room R1.

[0011] Among the components of the X-ray diagnostic apparatus 10, the processing circuitry 21, the input interface 22, the display 23a, and the memory circuitry 24 are installed in, for example, an operation room R2.

[0012] The examination room R1 is a room where a procedure is performed on the subject P. For example, in this embodiment, as an example of a procedure, an operator performs catheter treatment on the subject P. For this reason, the examination room R1 is also called a catheter room. In the examination room R1, an operator such as a doctor and multiple other medical staff members perform work. The examination room R1 is an example of a space in which a procedure is performed on the subject P in this embodiment. The medical staff members include not only doctors but also nurses, engineers, etc. In this embodiment, the operator and other medical staff members are collectively referred to as medical professionals.

[0013] The operation room R2 is, for example, a separate room adjacent to the examination room R1. In the operation room R2, an operator such as a technician or a supervisor operates the input interface 22 or refers to the display 23a. Note that a configuration in which all components of the X-ray diagnostic apparatus 10 are installed in the examination room R1 may be adopted.

[0014] The imaging device 30 captures an image of the interior of the examination room R1 and transmits the captured image to the X-ray diagnostic apparatus 10. In this embodiment, the imaging device 30 captures a moving image. The moving image captured by the imaging device 30 inside the examination room R1 is called an examination room moving image (or an examination room image).

[0015] Fig. 2 is a diagram showing an example of an examination room R1 according to the present embodiment. As shown in Fig. 2, the imaging device 30 is installed, for example, on the ceiling of the examination room R1. The imaging device 30 images, for example, a tabletop 14 on which a subject P is placed in the examination room R1, a C-arm 15, operators D1 and D2 (hereinafter simply referred to as operators D) who perform procedures on the subject P, a plurality of medical staff W1 to W3 (hereinafter simply referred to as medical staff W) working in the examination room R1, a floor 50 of the examination room R1, various devices installed in the examination room R1, a treatment table 40, and medical instruments 401a and 401b on the treatment table 40 (hereinafter simply referred to as medical instruments 401 when no particular distinction is made), etc.

[0016] Although one imaging device 30 is illustrated in FIGS. 1 and 2, the number of imaging devices 30 is not particularly limited, and a plurality of imaging devices 30 may be installed.

[0017] Returning to FIG. 1, the X-ray high voltage device 11 is a high voltage power supply that generates a high voltage under the control of the processing circuit 21 and supplies the generated high voltage to the X-ray tube 12.

[0018] The X-ray tube 12 generates X-rays using a high voltage supplied from the X-ray high voltage device 11. The X-ray aperture 13, under the control of the aperture control circuit 20, narrows down the X-rays generated by the X-ray tube 12 so that they are selectively irradiated onto a region of interest (ROI) of the subject P.

[0019] The top board 14 is a bed on which the subject P rests, and is placed on a bed device (not shown). The bed device (not shown) may be included in the X-ray diagnostic apparatus 10, or may be external to the X-ray diagnostic apparatus 10. Even if the bed device is not included in the X-ray diagnostic apparatus 10, it may be included in the system S. The subject P is not included in the X-ray diagnostic apparatus 10.

[0020] The X-ray detector 16 detects the X-rays that have passed through the subject P, and transmits the detection result to the processing circuitry 21.

[0021] The C-arm 15 holds the X-ray tube 12, the X-ray aperture 13, and the X-ray detector 16. The C-arm rotation / movement mechanism 17 is a mechanism for rotating and moving the C-arm 15 by driving a motor or the like provided on the support. The top board movement mechanism 18 is a mechanism for moving the top board 14. For example, the top board movement mechanism 18 moves the top board 14 using power generated by an actuator.

[0022] Under the control of the processing circuitry 21, the C-arm / top mechanism control circuit 19 controls the C-arm rotation / movement mechanism 17 and the top movement mechanism 18 to adjust the rotation and movement of the C-arm 15 and the movement of the top 14. Under the control of the processing circuitry 21, the aperture control circuit 20 adjusts the opening of the aperture blades of the X-ray aperture 13 to control the irradiation range of the X-rays irradiated to the subject P.

[0023] The input interface 22 is realized by a trackball, a switch button, a mouse, a keyboard, a touch pad, a foot switch for irradiating X-rays, etc. The input interface 22 is connected to the processing circuitry 21, and converts an input operation received from an operator into an electrical signal and outputs the electrical signal to the processing circuitry 21.

[0024] The displays 23a and 23b display a GUI (Graphical User Interface) for receiving instructions from an operator, setting values ​​of the X-ray diagnostic apparatus 10, or various images generated by the processing circuitry 21. For example, in this embodiment, the displays 23a and 23b display a risk notification image indicating the risk of an abnormality occurring detected by a detection function 212, which will be described later.

[0025] More specifically, the display 23a is hung from the ceiling of the examination room R1, for example. The display 23a is not limited to having one screen, and may have multiple screens. The display 23a is also called an in-examination room display.

[0026] Moreover, the display 23b is installed in the operation room R2. The display 23b is also referred to as an operation room display. The number of displays 23b is not particularly limited, and may be one or more. Hereinafter, when there is no particular distinction between the displays 23a and 23b, they will simply be referred to as the display 23.

[0027] In this embodiment, abnormalities that may occur in the examination room R1 include, for example, the dropping of a medical instrument 401, contact between a medical instrument and a subject P or a medical professional, insufficient preparation of medical instruments 401 to be used in a procedure, a medical professional falling, contamination of a disinfected area, etc.

[0028] Contact between medical equipment and the subject P or medical personnel includes, for example, contact between the C-arm 15 or the table top 14 and other medical equipment, the subject, or medical personnel, contact between medical personnel and other medical equipment other than the X-ray diagnostic apparatus 10, and contact between medical personnel.

[0029] Insufficient preparation of the medical instrument 401 used in the procedure occurs when the medical instrument 401 that should be in a predetermined position is not in place before the start of the procedure.

[0030] A healthcare professional falling may occur, for example, when a healthcare professional trips over cables scattered on the floor of examination room R1 and falls, or when a healthcare professional is about to fall and loses their balance.

[0031] Contamination of a disinfected area occurs, for example, when a disinfected area comes into contact with a non-disinfected object or with a healthcare worker or the like.

[0032] The above-mentioned examples are merely examples, and abnormalities occurring in the examination room R1 are not limited to these.

[0033] In addition, in the X-ray diagnostic apparatus 10, each processing function is stored in the memory circuitry 24 in the form of a program executable by a computer. The C-arm / top mechanism control circuitry 19, the aperture control circuitry 20, and the processing circuitry 21 are processors that realize the function corresponding to each program by reading and executing the program from the memory circuitry 24. In other words, each circuit in the state where each program has been read has the function corresponding to the read program.

[0034] The memory circuitry 24 stores programs corresponding to various functions read and executed by each circuit shown in Fig. 1. The memory circuitry 24 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, a hard disk, an optical disk, etc. The memory circuitry 24 is an example of a memory unit.

[0035] The processing circuit 21 has an acquisition function 211, a detection function 212, an explanatory text generation function 213, an alarm function 214, a reception function 215, a control function 216, and a learning function 217. The acquisition function 211 is an example of an acquisition unit. The detection function 212 is an example of a detection unit. The explanatory text generation function 213 is an example of an explanatory text generation unit. The alarm function 214 is an example of an alarm unit. The reception function 215 is an example of a reception unit. The control function 216 is an example of a control unit. The learning function 217 is an example of a learning unit.

[0036] In FIG. 1, it has been described that the processing functions of the acquisition function 211, the detection function 212, the explanatory text generation function 213, the notification function 214, the reception function 215, the control function 216, and the learning function 217 are realized by a single processing circuit 21. However, the processing circuit 21 may be configured by combining a plurality of independent processors, and each processor may execute a respective program to realize each processing function.

[0037] The term "processor" used in the above description refers to circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an Application Specific Integrated Circuit (ASIC), a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). Instead of storing a program in the memory circuit 24, the processor may be configured to directly incorporate the program into its circuit. In this case, the processor realizes its function by reading and executing the program incorporated in the circuit.

[0038] The acquisition function 211 acquires one or more images (image data) of the examination room R1 where a procedure on a subject is being performed. More specifically, the acquisition function 211 of this embodiment acquires, from the imaging device 30, an examination room video of the examination room R1.

[0039] The acquisition function 211 also acquires non-image data relating to at least one of the procedure, the subject P, or the examination room R1.

[0040] The non-image data includes, for example, at least one of procedure information related to the procedure, subject information related to the subject P, surgeon information related to the surgeon D performing the procedure, equipment information related to equipment used in the procedure, and medical institution information related to the medical institution where the examination room R1 is located. The non-image data is an example of non-image information in this embodiment.

[0041] The procedure information is, for example, the type of procedure. Specifically, the type of procedure is the type of protocol or program executed by the X-ray diagnostic apparatus 10 when the procedure is performed. For example, a protocol is a processing procedure determined for each treatment target part. Also, a program is a process executed by each protocol, and an example of the program is a program that executes a scan step such as DSA (Digital Subtraction Angiography).

[0042] Furthermore, the procedure information may be information about the time period when the procedure is performed. The time period when the procedure is performed may be, for example, the start time of the procedure, or a time period category such as "morning", "afternoon", or "late night" that includes the start time of the procedure. For example, late night procedures are often emergency procedures, and there may be a pattern in which the incidence of abnormalities is higher than in the morning or afternoon.

[0043] The procedure information may also be information indicating the day of the week on which the procedure is performed. For example, depending on the medical institution, there may be a pattern in the schedule according to the day of the week, such as treatment for a subject P with severe symptoms being performed in the first half of the week, and treatment for a subject P with mild symptoms or examinations with low difficulty being performed in the second half of the week.

[0044] Furthermore, the technique information may be information that indicates the difficulty of the technique.

[0045] The subject information is, for example, information regarding the severity of the subject P's condition, the subject P's age, the subject P's sex, and the like.

[0046] The surgeon information is, for example, information that indicates the proficiency of surgeon D. For example, the surgeon information may be a category according to the proficiency, such as "young surgeon," "veteran," or "supervising surgeon." The surgeon information may also be a numerical representation of the proficiency of surgeon D. The surgeon information is not limited to this, and may include various types of information other than proficiency.

[0047] The equipment used in the procedure is, for example, the X-ray diagnostic apparatus 10. That is, the equipment information on the equipment used in the procedure is, for example, equipment information on the X-ray diagnostic apparatus 10. The equipment information on the X-ray diagnostic apparatus 10 is, for example, position information of a movable mechanism included in the X-ray diagnostic apparatus 10. The movable mechanism includes at least either the C-arm 15 or the tabletop 14. The movable mechanism may also include both the C-arm 15 and the tabletop 14. For example, the acquisition function 211 acquires the position information of the C-arm 15 or the tabletop 14 from the C-arm / tabletop mechanism control circuit 19. The equipment information is also referred to as machine information.

[0048] The equipment information may also be information about a device used in a procedure. The information about a device used in a procedure may be, for example, the number of catheters used in the procedure, or the type of catheter. The device is not limited to a catheter, and may be a balloon, a stent, or the like. If a device used in a procedure is different, the device to be prepared will also be different.

[0049] The medical institution information is, for example, information indicating the region where the medical institution equipped with the examination room R1 is located. The region where the medical institution equipped with the examination room R1 is located, that is, the region where the medical institution where the X-ray diagnostic apparatus 10 is installed is, for example, the prefecture where the medical institution is located. If the insurance system differs depending on the region, the type of procedure or the type and number of devices used for the procedure will differ.

[0050] The source of the non-image data may be another component in the X-ray diagnostic apparatus 10 or an external device. For example, the acquisition function 211 may acquire non-image data input by an operator via an input interface. Alternatively, the acquisition function 211 may acquire subject information on the subject P and the area where the medical institution equipped with the examination room R1 is located from an electronic medical record system.

[0051] The acquisition function 211 sends the acquired laboratory video and non-image data to the detection function 212 .

[0052] The detection function 212 detects a state in which an abnormality is likely to occur in the examination room R1 based on one or more acquired images. More specifically, the detection function 212 detects a risk area in the examination room video based on the examination room video acquired by the acquisition function 211, non-image data, and the first trained model.

[0053] The first trained model indicates a correspondence relationship between a plurality of examination room videos in which the examination room R1 is captured and an abnormality that is likely to occur in the examination room R1. More specifically, the first trained model of this embodiment indicates a correspondence relationship between a plurality of examination room videos in which the examination room R1 is captured, non-image data, and an abnormality that is likely to occur in the examination room R1. The first trained model is an example of first correspondence information in this embodiment.

[0054] In this embodiment, the first trained model 91 is generated by a learning function 217 described later and stored in the memory circuitry 24. The first trained model 91 is composed of a neural network and trained parameter data.

[0055] The risk region is an image region in the examination room video where there is a high possibility that an abnormality will occur, or more specifically, where a state immediately before the occurrence of an abnormality is depicted.

[0056] The "state where an abnormality is highly likely to occur" refers to a state where the probability of an abnormality occurring is higher than or equal to a threshold. For example, one example of an abnormality is "a medical instrument 401 falling." Even if the medical instrument falling is not depicted in the examination room video itself, if a state where the medical instrument 401 is more likely to fall than or equal to a threshold is depicted in any frame included in the examination room video, the first trained model 91 outputs an image area in the frame where the medical instrument that is highly likely to fall is depicted as a risk area.

[0057] Furthermore, one example of an abnormality is "a medical worker falling down." Even if the medical worker has not yet fallen down, if the examination room video shows cables scattered on the floor 50, the first trained model 91 outputs the image region in which the scattered cables are depicted in the frame showing such a state as a risk region. Note that multiple risk regions may be detected in one frame.

[0058] FIG. 3 is a diagram showing an example of detection of a risk of an abnormality occurring and notification of the risk according to this embodiment.

[0059] The first trained model 91 is, for example, a trained model generated by deep learning such as a neural network. As a deep learning method, a recurrent neural network (RNN), a long short-term memory (LSTM), or the like can be applied. The first trained model 91 may also be generated by other deep learning or machine learning.

[0060] The detection function 212 reads out the first trained model 91 from the memory circuitry 24, and inputs the examination room video 71 and the non-image data 72 into the first trained model 91, as shown in FIG. 3.

[0061] The first trained model 91 detects a risk area depicting a state in which an abnormality is likely to occur from a plurality of frames included in the examination room video 71 based on the input examination room video 71 and non-image data 72. When the first trained model 91 detects that a state in which an abnormality is likely to occur is depicted in any of a plurality of frames included in the examination room video 71, the first trained model 91 generates a risk image 801 representing the risk area on the frame depicting the state in which an abnormality is likely to occur.

[0062] In this embodiment, the detection function 212 inputs the examination room video 71 into the first trained model 91, and detects a state in which an abnormality is likely to occur based on at least one of the movement speed or direction of movement of an object or person depicted in the examination room video 71, based on the chronological relationship between multiple frames contained in the examination room video 71.

[0063] For example, the first trained model 91 detects an increased possibility of an abnormality occurring based on the direction and speed of movement of the medical equipment or medical staff depicted in the examination room video 71. As a specific example, when the surgeon D turns his back to the treatment table 40 and moves toward the treatment table 40, and the distance between the surgeon D and the treatment table 40 becomes equal to or less than a predetermined distance, the first trained model 91 detects that an abnormality of "contact between the surgeon D and the treatment table 40" occurs. In addition, even if the distance between the surgeon D and the treatment table 40 becomes equal to or less than the predetermined distance, the first trained model 91 does not detect that an abnormality occurs if the surgeon D faces the treatment table 40. Note that this example of abnormality detection is merely an example, and does not limit the function of abnormality detection by the detection function 212.

[0064] Furthermore, the first trained model 91 detects the possibility of an abnormality occurring based on procedure information related to the procedure, subject information related to the subject P, surgeon information related to the surgeon D performing the procedure, equipment information related to the equipment used in the procedure, or medical institution information related to the medical institution where the examination room R1 is located, which are input as non-image data 72 in addition to the examination room video 71. For example, the possibility of an abnormality occurring may be higher or lower depending on the proficiency of the surgeon D, the time of day when the procedure is performed, etc. The first trained model 91 detects an increase in the possibility of an abnormality occurring by taking into account the fluctuation in the possibility of an abnormality occurring based on the non-image data 72.

[0065] In addition, since the equipment information contained in the non-image data 72 includes position information of the movable mechanisms contained in the X-ray diagnostic apparatus 10, the first trained model 91 detects an increased possibility of an abnormality occurring by taking into account fluctuations in the possibility of an abnormality occurring based on the positions of the C-arm 15 and the table top 14 in the blind spot of the imaging device 30 that is not depicted in the examination room video 71.

[0066] Hereinafter, an image including the risk image 801 generated by the detection function 212 is referred to as a risk examination room image 701. In this embodiment, the risk examination room image 701 is an image in which the risk image 801 is superimposed on a frame depicting a state in which there is a high possibility of an abnormality occurring, which is included in the examination room video 71.

[0067] The risk image 801 is, for example, a heat map showing the likelihood of an abnormality occurring, as shown in Fig. 3. A known technique such as Grad-CAM (Gradient-weighted Class Activation Mapping) can be applied as a method for generating a heat map showing the likelihood of an abnormality occurring. Note that the display form of the risk image 801 is not limited to a heat map, and may be a frame surrounding a risk area.

[0068] The detection function 212 sends the risk laboratory image 701 including the generated risk image 801 to the description generation function 213.

[0069] Returning to FIG. 1, the explanatory text generation function 213 generates a sentence explaining anomalies that are likely to occur in the risk area detected by the detection function 212 based on the second trained model.

[0070] The second trained model indicates a correspondence relationship between multiple risk areas in multiple images captured of the examination room R1 and words and phrases that describe abnormalities that are likely to occur in the multiple risk areas. For example, the second trained model 92 is a model that has learned natural language that describes "states in which abnormalities are likely to occur" depicted in risk areas included in the examination room video, in addition to the correspondence relationship between objects depicted in the examination room video and the names of the objects. The second trained model is an example of second correspondence information in this embodiment.

[0071] More specifically, the second trained model is a trained model generated by deep learning such as an RNN or a convolutional neural network (CNN), and outputs an explanatory text that describes an input image in text. The second trained model may also be generated by other deep learning or machine learning. In this embodiment, the second trained model is generated by a learning function 217 described later.

[0072] 3 again, the generation of an explanatory text by the second trained model 92 will be described. The explanatory text generation function 213 reads out the second trained model 92 from the memory circuitry 24, and inputs the risk examination room image 701 generated by the detection function 212 to the second trained model 92.

[0073] The second trained model 92 performs image recognition of objects depicted in the risk examination room image 701 and identifies the name of each object. The objects include people. The objects depicted in the risk examination room image 701 include, but are not limited to, medical instruments 401, a treatment table 40, a C-arm 15, a tabletop 14, an operator D, and a subject P. In addition, the second trained model 92 is capable of individually recognizing the names of various medical instruments 401 with respect to the medical instruments 401.

[0074] Furthermore, the second trained model 92 outputs an explanatory text that explains the "state in which an abnormality is likely to occur" depicted in the risk area represented by the risk image 801 in the risk examination room image 701. In this explanatory text, the second trained model 92 uses the name of an object recognized from the risk examination room image 701 as necessary.

[0075] In the example shown in Fig. 3, the second trained model 92 generates an explanatory text M saying "There is a risk that xxx may fall from the treatment table." "xxx" in the explanatory text M is the name of the medical instrument 401 that may fall. The explanatory text M shown in Fig. 3 is an example, and the text explaining the abnormality that is likely to occur in the risk area is not limited to this.

[0076] The explanatory text generation function 213 sends the notification image 702, in which explanatory text M has been added to the risk examination room image 701, to the notification function 214. The notification image 702 is an image including the risk examination room image 701, which includes a risk image 801 such as a heat map, and explanatory text M.

[0077] Returning to FIG. 1, when the detection function 212 detects that there is a high possibility of an abnormality occurring, the notification function 214 notifies that there is a high possibility of an abnormality occurring. Furthermore, the notification function 214 outputs the explanation M generated by the explanation generation function 213. More specifically, the notification function 214 displays the notification image 702 generated by the explanation generation function 213 on the displays 23a and 23b. The notification function 214 may display the notification image 702 on both the displays 23a and 23b, or may display the notification image 702 on only one of the displays.

[0078] For example, when the display 23a in the examination room R1 includes multiple screens and any of the screens is not in use, the notification function 214 displays the notification image 702 on the unused screen of the display 23a. In addition, when multiple displays 23b are installed in the operation room R2, the notification function 214 displays the notification image 702 on the display 23b on which no other images are displayed.

[0079] Furthermore, when there is no free screen on the displays 23a and 23b, the notification function 214 replaces any of the images being displayed with the notification image 702. In this embodiment, the priority of information displayed on the displays 23a and 23b is highest for medical images, followed by the notification image 702. When an image with a lower priority than the notification image 702 (e.g., setting values ​​of the X-ray diagnostic apparatus 10, etc.) is displayed on the displays 23a and 23b, the notification function 214 ends the display of the image with the lower priority and displays the notification image 702 instead. Note that the display location of the notification image 702 is not limited to the above example.

[0080] Furthermore, the notification function 214 may control a speaker (not shown) or the like to output the explanatory text M as sound. Furthermore, the notification image 702 may output a warning sound together with the display of the notification image 702.

[0081] Furthermore, when the reception function 215 described later receives an operation to stop the notification from the operator, the notification function 214 stops displaying the notification image 702, outputting the audio of the explanatory text M, or outputting the warning sound.

[0082] The reception function 215 receives an operation input by an operator via the input interface 22. For example, the reception function 215 receives an operation to stop notification input by the operator. The reception function 215 sends a notification that the operation to stop notification has been received to the notification function 214. The reception function 215 also receives an operation to start or end imaging processing of the X-ray diagnostic apparatus 10 input by the operator. When the reception function 215 receives an operation to start or end imaging processing, it sends a notification that the operation to start or end imaging processing has been received to the control function 216.

[0083] The control function 216 controls the entire X-ray diagnostic apparatus 10 based on an operation to start or end the imaging process received by the reception function 215, and executes imaging process to image the subject P. For example, the control function 216 controls the rotation and movement of the C-arm 15 and the movement of the top board 14 by controlling the C-arm / top board mechanism control circuit 19. The control function 216 also controls the X-ray high voltage device 11 and the aperture control circuit 20 to control the irradiation of X-rays to the subject P. The control function 216 also acquires an X-ray image based on an electrical signal converted from X-rays by the X-ray detector 16.

[0084] The learning function 217 generates a first trained model 91 by learning the relationship between a plurality of pre-abnormality occurrence images and a pre-abnormality occurrence image area in which the location where an abnormality occurs is depicted in the plurality of pre-abnormality occurrence images.

[0085] The image before the occurrence of an abnormality is an image captured in the examination room R1 at a time before the occurrence of an abnormality during the procedure.

[0086] The learning function 217 also learns the relationship between the non-image data 72 related to the procedure in which the pre-abnormality image was captured and the pre-abnormality image area. The learning function 217 may also learn the relationship between not only the pre-abnormality image but also an image in which an abnormality has actually occurred or an image in which no abnormality has occurred, and the presence or absence of an abnormality.

[0087] The generation of the first trained model 91 in this embodiment will be specifically described with reference to FIG. 4. FIG. 4 is a diagram showing an example of a method for generating the first trained model 91 according to this embodiment. In this embodiment, as shown in FIG. 4, the learning function 217 acquires a plurality of abnormality-containing examination room videos 1071a-1071n, a plurality of abnormality-free examination room videos 2071a-2071n, a plurality of abnormality-containing non-image data for learning 1072a-1072n, and a plurality of abnormality-free non-image data for learning 2072a-2072n as learning data. In addition, the learning function 217 acquires a plurality of risk area information 802a-802n as teacher data.

[0088] Each of the multiple abnormality-containing examination room videos 1071a to 1071n is a video captured in a state in which an abnormality occurs during a procedure in the examination room R1. Hereinafter, when there is no need to distinguish between the individual abnormality-containing examination room videos 1071a to 1071n, they will simply be referred to as abnormality-containing examination room videos 1071.

[0089] Each of the multiple abnormality-free examination room videos 2071a to 2071n is a video captured in a state where no abnormality occurred during a procedure in the examination room R1. Hereinafter, when there is no need to distinguish between the individual abnormality-free examination room videos 2071a to 2071n, they will simply be referred to as abnormality-free examination room videos 2071.

[0090] The abnormality-present examination room video 1071 and the abnormality-free examination room video 2071 may be videos captured from the start to the end of the procedure, or may be videos captured during a part of the procedure. The abnormality-present examination room video 1071 and the abnormality-free examination room video 2071 may be videos captured in the examination room R1, or may be videos captured in other examination rooms. The source of the abnormality-present examination room video 1071 and the abnormality-free examination room video 2071 is not particularly limited. For example, the learning function 217 may acquire the abnormality-present examination room video 1071 and the abnormality-free examination room video 2071 from an external device outside the system S, or may acquire them from the imaging device 30.

[0091] Each of the multiple learning non-image data with abnormality 1072a to 1072n is non-image data corresponding to each of the multiple examination room videos with abnormality 1071a to 1071n. For example, the learning non-image data with abnormality 1072a is associated with the examination room video with abnormality 1071a. Hereinafter, when the individual learning non-image data with abnormality 1072a to 1072n are not distinguished from each other, they are simply referred to as learning non-image data with abnormality 1072.

[0092] Each of the multiple learning non-image data without abnormality 2072a to 2072n is non-image data corresponding to each of the multiple abnormality-free examination room videos 2071a to 2071n. Hereinafter, when there is no need to distinguish between the individual learning non-image data without abnormality 2072a to 2072n, they will simply be referred to as learning non-image data without abnormality 2072.

[0093] The learning non-image data with abnormalities 1072 and the learning non-image data without abnormalities 2072 each include at least one of procedure information related to the procedure, subject information related to the subject P, surgeon information related to the surgeon D performing the procedure, equipment information related to the equipment used in the procedure, or medical institution information related to the medical institution in which the examination room R1 is located, similar to the non-image data 72.

[0094] Each of the multiple risk area information 802a to 802n is teacher data for identifying risk areas in the multiple abnormality-containing examination room videos 1071a to 1071n. Hereinafter, when the individual risk area information 802a to 802n is not distinguished from one another, they will simply be referred to as risk area information 802.

[0095] Specifically, the risk area information 802 is information capable of identifying a frame immediately before a frame depicting an abnormality included in the abnormality-containing examination room video 1071, and a risk area in the frame immediately before the frame depicting the abnormality. The frame immediately before the frame depicting the abnormality is an example of an image before the occurrence of an abnormality in this embodiment.

[0096] Moreover, the risk area in the frame immediately before the frame in which the abnormality is depicted is an example of an image area before the occurrence of the abnormality in this embodiment. The risk area in the frame immediately before the frame in which the abnormality is depicted is, for example, an image area in which the cause of the abnormality is depicted. Specifically, if the abnormality that has occurred is "dropping of the medical instrument 401", the image area in which the medical instrument 401 is depicted in the frame in which the state before the medical instrument 401 was dropped is the risk area. Moreover, if the abnormality that has occurred is "contact between the medical instrument and the surgeon D", the image area in which the medical instrument and the surgeon D are depicted in the frame in which the state immediately before the contact between the medical instrument and the surgeon D is depicted is the risk area.

[0097] There is no particular limitation on the source of acquisition of the risk region information 802, the learning non-image data with abnormality 1072, and the learning non-image data without abnormality 2072. For example, the learning function 217 may acquire the risk region information 802, the learning non-image data with abnormality 1072, and the learning non-image data without abnormality 2072 input by an operator from the input interface 22, or may acquire the risk region information 802, the learning non-image data with abnormality 1072, and the learning non-image data without abnormality 2072 from an external device.

[0098] The learning function 217 performs deep learning such as CNN or RNN based on these learning data and teacher data, and generates a first trained model 91.

[0099] For example, in this embodiment, the learning function 217 learns a state in which an abnormality is likely to occur according to at least one of the moving speed or moving direction of an object or human depicted in the abnormality-present examination room video 1071 and the abnormality-free examination room video 2071 based on the time series relationship between a plurality of frames included in the abnormality-present examination room video 1071 and the abnormality-free examination room video 2071. For example, even if the distance between the surgeon D and the treatment table 40 is the same in the abnormality-present examination room video 1071 and the abnormality-free examination room video 2071, an abnormality such as "contact between the surgeon D and the treatment table 40" may or may not occur depending on the moving speed, moving direction, or body direction of the surgeon D. The learning function 217 learns that the possibility of an abnormality occurring changes based on the moving direction or speed of such medical equipment or medical personnel. Through such learning, the learning function 217 can generate the first trained model 91 that takes into account the occurrence of an abnormality caused by the movement of an object or human.

[0100] Furthermore, in this embodiment, the learning function 217 learns that the possibility of an abnormality occurring varies depending on procedure information related to the procedure, subject information related to the subject P, operator information related to the operator D performing the procedure, equipment information related to the equipment used in the procedure, or medical institution information related to the medical institution where the examination room R1 is located, based on the learning non-image data with abnormality 1072 and the learning non-image data without abnormality 2072. For example, the learning function 217 learns when the possibility of an abnormality occurring increases or decreases depending on the proficiency of the operator D or the time of day when the procedure is performed.

[0101] In addition, based on the learning non-image data with abnormality 1072 and the learning non-image data without abnormality 2072, the learning function 217 learns the relationship between the positional information of the movable mechanism included in the X-ray diagnostic device 10 in the blind spot of the imaging device 30 that is not depicted in the examination room video with abnormality 1071 or the examination room video without abnormality 2071, and the likelihood of an abnormality occurring.

[0102] The learning function 217 stores the generated first trained model 91 in the memory circuitry 24.

[0103] In addition, the learning function 217 generates a second trained model 92 by learning the correspondence between objects depicted in the examination room video and the names of the objects, and the correspondence between risk areas included in the examination room video and words that describe abnormalities that are likely to occur in the risk areas included in the examination room video.

[0104] Fig. 5 is a diagram showing an example of a method for generating the second trained model 92 according to this embodiment. As shown in Fig. 5, the learning function 217 generates the second trained model 92 by learning a plurality of training risk laboratory images 1701a to 1701n, a plurality of label information 901a to 901n, and a plurality of training explanation sentences Ma to Mn.

[0105] Each of the multiple learning risk laboratory images 1701a to 1701n is an image in which a risk image 801 is superimposed on a frame depicting a state in which an abnormality is highly likely to occur. Hereinafter, when there is no need to distinguish between the individual learning risk laboratory images 1701a to 1701n, they will simply be referred to as learning risk laboratory images 1701.

[0106] Each of the plurality of pieces of label information 901a to 901n is information that associates an image area in which an object is depicted in the risk examination room image for learning 1701 with the name of the object.

[0107] Each of the multiple learning explanation sentences Ma to Mn is a sentence that explains an abnormality that is likely to occur in the risk area represented by the risk image 801 of the learning risk examination room image 1701.

[0108] The sources of the multiple learning risk laboratory images 1701a-1701n, the multiple label information 901a-901n, and the multiple learning explanation sentences Ma-Mn are not particularly limited. The above-mentioned method for generating the second trained model 92 is an example, and the learning function 217 is not limited to the above-mentioned method, and can apply known natural language learning methods. The learning function 217 stores the generated second trained model 92 in the memory circuitry 24.

[0109] Next, a flow of an abnormality detection process executed in the X-ray diagnostic apparatus 10 of the system S of this embodiment configured as above will be described.

[0110] Fig. 6 is a flowchart showing an example of the flow of an anomaly detection process according to the present embodiment. The process shown in Fig. 6 starts, for example, when the imaging process of the subject P by the control function 216 starts. In addition, it is assumed that the first trained model 91 and the second trained model 92 are generated and stored in the memory circuitry 24 before the start of this flowchart.

[0111] First, the acquisition function 211 acquires non-image data 72 from an external device or the input interface 22 or the like (S1).

[0112] Moreover, the acquisition function 211 acquires the examination room video 71, which is an image of the inside of the examination room R1, from the imaging device 30 (S2).

[0113] Next, the detection function 212 inputs the examination room video 71 and the non-image data 72 acquired by the acquisition function 211 to the first trained model 91 (S3). For the examination room video 71, the detection function 212 inputs the frames included in the examination room video 71 to the first trained model 91 in chronological order.

[0114] Then, the detection function 212 determines whether or not the risk examination room image 701 has been output from the first trained model 91 (S4). If the risk examination room image 701 has not been output from the first trained model 91, the examination room video 71 does not include a risk area.

[0115] Therefore, if the detection function 212 determines that the risk examination room image 701 has not been output from the first trained model 91 (S4 “No”), it determines that there is no abnormality that is likely to occur in the examination room R1.

[0116] In this case, the detection function 212 notifies the notification function 214 of the determination result that there is no abnormality that is likely to occur in the examination room R1. If the notification image 702 is already being displayed at this time (S5 "Yes"), the notification function 214 ends the display of the notification image 702 (S6). Then, the process proceeds to S12.

[0117] Also, the notification function 214 proceeds to the process of S12 even if the notification image 702 is not being displayed (S5 "No").

[0118] In addition, if the detection function 212 determines that a risk examination room image 701 has been output from the first trained model 91 (S4 “Yes”), it sends the output risk examination room image 701 to the explanation generation function 213.

[0119] Next, the explanatory text generation function 213 inputs the risk examination room image 701 to the second trained model 92 (S7). From the second trained model 92, an explanatory text M is generated that explains the "state in which an abnormality is likely to occur" depicted in the risk area represented by the risk image 801 in the risk examination room image 701.

[0120] Then, the explanation generating function 213 generates a notification image 702 by adding the explanation M to the risk examination room image 701 (S8). The explanation generating function 213 sends the generated notification image 702 to the notification function 214.

[0121] Then, the notification function 214 displays the notification image 702 generated by the explanatory text generating function 213 on the displays 23a and 23b (S9). The notification function 214 may also control a speaker (not shown) or the like to output the explanatory text M as sound or to output a warning sound.

[0122] Next, the reception function 215 judges whether or not an operation to stop the notification input by the operator has been received (S10). When the reception function 215 judges that an operation to stop the notification has been received (S10 "Yes"), it notifies the notification function 214 that the operation to stop the notification has been received. In this case, the notification function 214 ends the display of the notification image 702 (S11). Furthermore, when the notification function 214 has been outputting the voice of the explanatory text M or the warning sound, it stops these outputs.

[0123] Next, the reception function 215 determines whether or not an operation to end the procedure input by the operator has been received (S12). The operation to end the procedure is, for example, an operation to end the imaging process. If the reception function 215 determines that the operation to end the procedure has not been received (S12 "No"), the process returns to S2, and the acquisition function 211 acquires a new examination room video 71.

[0124] Furthermore, in the process of S10, when it is determined that an operation to stop the notification has not been received (S10 “No”), the reception function 215 determines whether or not the waiting time has elapsed (S13). The length of the waiting time is not particularly limited, but may be, for example, a time that is sufficient for a medical worker in the examination room R1 to eliminate the cause of the abnormality based on the notification image 702. The waiting time may be determined according to the content of the abnormality that is determined to be highly likely to occur by the detection function 212, or may be a fixed length of time.

[0125] While the reception function 215 determines that the standby time has not elapsed (S13 "No"), it repeats the processes of S10 and S13 until it determines that an operation to stop notification has been received.

[0126] Furthermore, if the reception function 215 determines that the waiting time has elapsed (S13 "Yes"), the process returns to S2, and the acquisition function 211 acquires a new examination room video 71. Thereafter, S2 to S4 are executed, and if the detection function 212 determines that the risk examination room image 701 has not been output from the first trained model 91, the display of the notification image 702 displayed in S9 is terminated by the processes of S5 and S6. In other words, even if the operator does not perform an operation to stop the notification, if the examination room video 71 sequentially acquired by the X-ray diagnostic apparatus 10 no longer includes a risk area, the display of the notification image 702 is terminated. Note that the determination of whether the waiting time has elapsed in S13 may be performed by the acquisition function 211.

[0127] Furthermore, in the process of S12, if it is determined that the reception function 215 has received an operation to end the procedure (S12 "Yes"), the process of this flowchart ends.

[0128] In this flowchart, the X-ray diagnostic apparatus 10 acquires non-image data 72 at the beginning of the process and then performs an abnormality detection process using the same non-image data 72. However, new non-image data 72 may also be acquired during the procedure.

[0129] In this way, the X-ray diagnostic apparatus 10 of this embodiment detects a state in which an abnormality is likely to occur in the examination room R1, based on the examination room video 71 capturing an image of the examination room R1 where a procedure is being performed on the subject P, and the first trained model 91. Therefore, according to the X-ray diagnostic apparatus 10 of this embodiment, by detecting an abnormality that is likely to occur in the future from the examination room video 71, it is possible to detect an abnormality that may occur in the examination room R1 in advance.

[0130] In addition, the X-ray diagnostic apparatus 10 of this embodiment can detect a state in which an abnormality is likely to occur depending on at least one of the moving speed and the moving direction of an object or person depicted in the examination room video 71, thereby detecting an abnormality caused by the movement of an object or person with high accuracy.

[0131] Moreover, the first trained model 91 of this embodiment indicates a correspondence relationship between a plurality of abnormality-containing examination room videos 1071a-1071n captured in the examination room R1 and risk area information 802a-802n in which an abnormality is likely to occur in the plurality of abnormality-containing examination room videos 1071a-1071n. The X-ray diagnostic apparatus 10 of this embodiment inputs the examination room video 71 to the first trained model 91, thereby being able to detect a risk area, which is an image area in which an abnormality is likely to occur in the examination room video 71.

[0132] Moreover, the X-ray diagnostic device 10 of this embodiment acquires non-image data related to at least one of the procedure, the subject P, or the examination room R1. Moreover, the first trained model 91 indicates a correspondence relationship between a plurality of images captured in the examination room R1 (a plurality of examination room videos 1071a-1071n with abnormalities, a plurality of examination room videos 2071a-2071n without abnormalities), non-image data (a plurality of learning non-image data with abnormalities 1072a-1072n, a plurality of learning non-image data without abnormalities 2072a-2072n), and abnormalities that are likely to occur in the examination room R1. Therefore, according to the X-ray diagnostic device 10 of this embodiment, it is possible to detect a state in which an abnormality is likely to occur by taking into account information that is not depicted in the examination room video 71.

[0133] Furthermore, when the X-ray diagnostic apparatus 10 of this embodiment detects that there is a high possibility of an abnormality occurring, it notifies the medical staff and others in the examination room R1 or the operation room R2 of the high possibility of an abnormality occurring, thereby enabling the medical staff and others in the examination room R1 or the operation room R2 to be aware of the high possibility of an abnormality occurring before the abnormality occurs.

[0134] Furthermore, the non-image data 72 of this embodiment includes at least one of procedure information related to the procedure, subject information related to the subject P, operator information related to the operator D performing the procedure, equipment information related to the equipment used in the procedure, and medical institution information related to the medical institution where the examination room R1 is located. The probability of abnormality occurring varies depending on the type of procedure, the time period when the procedure is performed, the equipment such as devices used in the procedure, the severity of the subject P, or the proficiency of the operator D. For this reason, according to the X-ray diagnostic apparatus 10 of this embodiment, by using these non-image data 72 for detection, it is possible to detect a state in the examination room R1 where there is a high possibility of abnormality occurring with high accuracy.

[0135] In addition, the X-ray diagnostic apparatus 10 of this embodiment generates an explanatory text M that explains abnormalities that are likely to occur in the detected risk area and notifies the user of the explanatory text M, so that medical staff and others in the examination room R1 or the control room R2 can easily understand the content of the abnormalities that are likely to occur.

[0136] Furthermore, the X-ray diagnostic apparatus 10 of this embodiment generates the first trained model 91 by learning the relationship between a pre-abnormality image in which a space is captured at a time before the abnormality occurs during a procedure and a pre-abnormality image region in which a location where the abnormality occurs is depicted in the pre-abnormality image. Therefore, the X-ray diagnostic apparatus 10 of this embodiment can generate the first trained model 91 that can detect a state before the abnormality occurs from the examination room video 71.

[0137] In addition, the first trained model 91 and the second trained model 92 in this embodiment include a “self-learning model” that further updates the internal algorithms of the first trained model 91 and the second trained model 92 by obtaining user feedback on the risk examination room image 701 or the alarm image 702 output by the X-ray diagnostic apparatus 10.

[0138] The first trained model 91 and the second trained model 92 may be constructed by integrated circuits such as ASIC and FPGA. The first correspondence information and the second correspondence information are not limited to trained models. For example, the first correspondence information and the second correspondence information may be a mathematical model, a lookup table, a database, or the like.

[0139] Second Embodiment In the X-ray diagnostic apparatus 10 of this embodiment, different first trained models 91 are used depending on the examination protocol.

[0140] The system S of this embodiment includes an X-ray diagnostic apparatus 10 and an imaging apparatus 30, similar to the first embodiment. The imaging apparatus 30 of this embodiment includes functions similar to those of the first embodiment. The X-ray diagnostic apparatus 10 of this embodiment includes functions similar to those of the first embodiment. The X-ray diagnostic apparatus 10 is an example of a medical information processing apparatus in this embodiment.

[0141] The processing circuitry 21 of the X-ray diagnostic apparatus 10 of this embodiment, like the first embodiment, includes an acquisition function 211, a detection function 212, an explanatory text generation function 213, a notification function 214, a reception function 215, a control function 216, and a learning function 217. The acquisition function 211, the explanatory text generation function 213, the notification function 214, the reception function 215, and the control function 216 have the same functions as those of the first embodiment.

[0142] Moreover, the X-ray diagnostic apparatus 10 of this embodiment includes a plurality of first trained models 91 trained for each of the non-image data 1072, 2072.

[0143] More specifically, the non-image data 72, 1072, 2072 of the present embodiment includes a protocol type among procedure information related to a procedure. Hereinafter, the protocol type is simply referred to as a "protocol." The protocol type is an example of a type of non-image information.

[0144] As described in the first embodiment, a protocol is a processing procedure determined for each treatment target region. In this embodiment, for example, a processing procedure determined for a treatment target region included in an examination performed by the X-ray diagnostic apparatus 10 is called an examination protocol. That is, the non-image data 72, 1072, 2072 of this embodiment includes a protocol of the examination performed by the X-ray diagnostic apparatus 10.

[0145] Moreover, the X-ray diagnostic apparatus 10 of this embodiment includes a plurality of first trained models 91 trained for each examination protocol included in the non-image data 1072, 2072. The plurality of first trained models 91 are stored in the memory circuitry 24, for example, in association with each examination protocol.

[0146] In addition to the functions of the first embodiment, the detection function 212 of this embodiment detects a risk area, which is an image area in which an abnormality is likely to occur in the acquired image, by inputting the examination room video 71 and non-image data 72 acquired by the acquisition function 211 into a first trained model 91 corresponding to the non-image data 72 acquired by the acquisition function 211. More specifically, the detection function 212 inputs the examination room video 71 and non-image data 72 acquired by the acquisition function 211 into a first trained model 91 corresponding to the examination protocol included in the non-image data 72 acquired by the acquisition function 211.

[0147] The examination room video 71 is an example of an image acquired by the acquisition function 211. Moreover, the non-image data 72 is an example of non-image information.

[0148] In other words, the detection function 212 selects, from among the multiple first trained models 91 stored in the memory circuitry 24, a first trained model 91 associated with the same protocol as the protocol of the examination included in the non-image data 72 acquired by the acquisition function 211. Then, the detection function 212 inputs the examination room video 71 and the non-image data 72 acquired by the acquisition function 211 to the selected first trained model 91.

[0149] The positions of the operator D and medical staff W in the examination room R1 differ depending on the examination protocol. Also, the position and movement timing of the C-arm 15 or the tabletop 14 differ depending on the examination protocol.

[0150] Therefore, according to the X-ray diagnostic apparatus 10 of this embodiment, by using the first trained model 91 according to the examination protocol, it is possible to detect the occurrence of an abnormality in the examination room R1 with high accuracy.

[0151] Further, the learning function 217 of this embodiment generates a first trained model 91 for each protocol of the examination in addition to the function of the first embodiment. For example, the learning function 217 acquires a plurality of abnormality-containing examination room videos 1071a to 1071n, a plurality of abnormality-free examination room videos 2071a to 2071n, a plurality of abnormality-containing non-image data for learning 1072a to 1072n, and a plurality of abnormality-free non-image data for learning 2072a to 2072n as training data for each protocol of the examination. Furthermore, the learning function 217 acquires a plurality of risk area information 802a to 802n as teacher data. The learning function 217 executes deep learning such as CNN or RNN based on the training data and the teacher data to generate the first trained model 91.

[0152] In this embodiment, the first trained model 91 is generated for each examination protocol among the information contained in the non-image data 1072, 2072, but the first trained model 91 may also be generated for each information other than the examination protocol.

[0153] (Variation 1) In the above-mentioned first and second embodiments, the X-ray diagnostic apparatus 10 is an example of a medical information processing apparatus, but the medical information processing apparatus may be an information processing apparatus other than the X-ray diagnostic apparatus 10. For example, the medical information processing apparatus may be a PC (Personal Computer) or a server apparatus or the like provided outside the examination room R1. The PC (Personal Computer) or the server apparatus or the like is connected to the X-ray diagnostic apparatus 10 via, for example, an in-hospital network or the like.

[0154] Furthermore, the medical information processing device may be various modalities such as an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, an ultrasound diagnostic device, a PET (Positron Emission Tomography) device, or a SPECT (Single Photon Emission Computed Tomography) device.

[0155] (Variation 2) In the first and second embodiments described above, the examination room R1 where the catheter treatment is performed is taken as an example of the space, but the space to be detected as an abnormality is not limited to this. For example, an operating room or the inside of an ambulance may be taken as an example of the space.

[0156] (Variation 3) In the first and second embodiments described above, the first trained model 91 and the second trained model 92 are generated by the learning function 217, but the first trained model 91 and the second trained model 92 may be generated by another information processing device. The X-ray diagnostic device 10 acquires the first trained model 91 and the second trained model 92 from the other information processing device and stores them in the memory circuitry 24.

[0157] (Variation 4) Furthermore, the system S may include various sensors installed in the examination room R1 in addition to the imaging device 30. For example, the system S may include a 3D sensor capable of measuring the distance to an object. When this configuration is adopted, the detection function 212 of the X-ray diagnostic apparatus 10 further detects a state in which an abnormality is likely to occur in the examination room R1 based on the distance to the object measured by the 3D sensor. When this configuration is adopted, the learning function 217 further generates a first trained model 91 using the distance to the object measured by the 3D sensor as learning data.

[0158] (Variation 5) The system S may also include a position sensor attached to a medical professional in the examination room R1. The position sensor is, for example, a wearable device that transmits the position of the medical professional.

[0159] In this modification, the acquisition function 211 acquires position information of each of a plurality of medical staff during the procedure. In this case, the operator information included in the non-image data 72 includes the position information of the operator D in the examination room R1. The non-image data 72 may include the position information of not only the operator D but also all the medical staff in the examination room R1.

[0160] In this modification, the detection function 212 further detects a state in which an abnormality is likely to occur in the examination room R1 based on position information of the operator D or other medical personnel during the procedure. According to this modification, the X-ray diagnostic apparatus 10 can detect a state in which an abnormality is likely to occur in the examination room R1 by taking into account, for example, position information of the operator D or other medical personnel in a blind spot of the imaging device 30 that is not depicted in the examination room video 71. For example, the X-ray diagnostic apparatus 10 can detect contact between medical personnel in a position not depicted in the examination room video 71, contact between a medical personnel and a medical device, etc.

[0161] In addition, when this configuration is adopted, the learning function 217 further generates a first trained model 91 using the acquired positions of medical personnel as training data.

[0162] (Variation 6) In the first and second embodiments described above, the imaging device 30 captures the examination room video 71, but the imaging device 30 may capture a still image. When this configuration is adopted, the detection function 212 detects a risk area in the still image based on the first trained model 91 and at least one still image captured from the examination room video 71. In this case, the generation process of the first trained model 91 by the learning function 217 also uses a still image, not a video. When a still image is used as training data, the first trained model 91 may be generated by CNN.

[0163] (Variation 7) In the above-described first and second embodiments, the first trained model 91 is stored in the memory circuitry 24, but the first trained model 91 may be incorporated in the detection function 212. The second trained model 92 may also be incorporated in the explanation generation function 213.

[0164] (Variation 8) In the above-described first and second embodiments, the learning function 217 executes supervised learning using the risk area information 802 when generating the first trained model 91, but the learning method is not limited to this, and for example, semi-supervised learning may be adopted. Moreover, the learning function 217 may also employ semi-supervised learning when generating the second trained model 92.

[0165] (Variation 9) In addition, the learning function 217 in the first and second embodiments described above uses the risk area information 802 as teacher data, but the teacher data is not limited to this. For example, the learning function 217 may use abnormal area information capable of identifying an abnormal area depicting a state in which an abnormality has actually occurred, rather than immediately before the occurrence of an abnormality, as teacher data. For example, the abnormal area information is information capable of identifying a frame in which an abnormality is depicted, which is included in the abnormality-containing examination room video 1071, and an abnormal area that is an image area in which the abnormality is depicted in the frame. When this configuration is adopted, the learning function 217 learns an image area in which the same object as the abnormal area is depicted in a frame immediately before a frame in which an abnormality specified by the abnormal area information is depicted, as a risk area in which an abnormality is likely to occur.

[0166] (Variation 10) The non-image data 72, 1072, 2072 may include operation information of the C-arm 15 in addition to the information exemplified in the first and second embodiments described above.

[0167] The operation information of the C-arm 15 includes, for example, the direction and speed of operation in the case of manual operation. Also, in the case of sequential operation, the operation information of the C-arm 15 includes the range of operation of the C-arm 15. Also, the movable mechanism of this modified example includes at least the C-arm 15. Manual operation means that the operator D operates the operation unit of the X-ray diagnostic apparatus 10 to manually operate the C-arm 15. Also, sequential operation means that the C-arm 15 automatically operates according to an input sequence.

[0168] The X-ray diagnostic apparatus 10 of this modification detects a state in which an abnormality is likely to occur, based on the operation information of the C-arm 15. Therefore, the X-ray diagnostic apparatus 10 of this modification can estimate the occurrence of interference between the C-arm 15 and the operator D or medical staff W with high accuracy.

[0169] (Variation 11) The non-image data 72, 1072, 2072 may include voice data related to the exchange between the examination room R1 and the operation room R2 in addition to the information exemplified in the first and second embodiments described above.

[0170] Between the examination room R1 and the operation room R2, audio data is transmitted by a microphone and a speaker provided in each of the examination room R1 and the operation room R2. The non-image data 72, 1072, and 2072 of this modification include the audio data.

[0171] For example, when a supervisor in the operation room R2 verbally instructs an operator D in the examination room R1, the voice data contains the contents of the instruction. If the operator D performs an action that is inconsistent with the instruction due to mishearing or the like, a mismatch occurs between the state of the examination room R1 contained in the examination room video 71 and the contents of the instruction contained in the voice data. For this reason, the X-ray diagnostic apparatus 10 of this modified example can detect information indicating a high possibility of an abnormality occurring in the examination room R1 with high accuracy by using non-image data 72 that contains voice data.

[0172] The voice data included in the non-image data 72, 1072, 2072 is not limited to the voice data actually transmitted and received between the examination room R1 and the operation room R2. For example, the voice data may be collected by a microphone installed in the examination room R1 or the operation room R2.

[0173] (Variation 12) In the first and second embodiments described above, the imaging device 30 is installed, for example, on the ceiling of the examination room R1, but the installation position of the imaging device 30 is not limited to this. For example, the imaging device 30 may capture an image of the examination room R1 from outside the examination room R1. When the operation room R2 and the examination room R1 are separated by a transparent material such as glass, the imaging device 30 may be installed in the operation room R2. In this case, the imaging device 30 captures an image of the inside of the examination room R1 from the operation room R2. The imaging device 30 may also capture an image of the inside of the operation room R2.

[0174] Moreover, the imaging device 30 is disposed so that at least a part or all of the movable mechanism is captured within the field of view of the imaging device 30, regardless of the position of the movable mechanism. More preferably, the imaging device 30 is disposed so that a part or all of the C-arm 15 of the movable mechanism is captured within the field of view of the imaging device 30, regardless of the position of the movable mechanism.

[0175] (Variation 13) The X-ray diagnostic apparatus 10 may have a biplane configuration including a C-arm 15 and an Ω-arm. The moving mechanism of this modification includes at least the C-arm 15 and the Ω-arm. The moving mechanism may also include a tabletop 14.

[0176] Moreover, the imaging device 30 is preferably disposed so that the C-arm 15 and the Ω-arm, which are part of the movable mechanism, are partly or entirely within the field of view of the imaging device 30, regardless of the position of the movable mechanism.

[0177] According to at least one of the embodiments described above, it is possible to detect abnormalities that may occur in the examination room R1 before they occur.

[0178] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0179] 10. X-ray diagnostic equipment 21 Processing circuit 23,23a,23b Display 24 Memory circuit 30 Imaging device 71 Examination Room Video 72 Non-image data 91 First trained model 92 Second trained model 211 Acquisition Function 212 Detection function 213 Description generation function 214 Notification function 215 Reception Function 216 Control Functions 217 Learning Function 701 Risk Inspection Room Images 702 Notification Image 801 Risk Image 802, 802a~802n Risk Area Information 901a~901n Label information 1071, 1071a~1071n Abnormality inspection room video 1072, 1072a~1072n Image data with or without abnormalities for learning 1701, 1701a~1701n Educational Risk Examination Room Images 2071, 2071a~2071n No abnormalities in the examination room video 2072, 2072a~2072n Learning non-image data without abnormalities D,D1,D2 Caster M Description Ma~Mn Learning explanation P Subject R1 Examination Room R2 control room S System

Claims

1. an acquisition unit that acquires an image of a space in which a procedure on a subject is being performed and non-image information related to at least one of the procedure, the subject, or the space; a detection unit that detects a state in which an abnormality is likely to occur in the space based on the image acquired by the acquisition unit, the non-image information acquired by the acquisition unit, and first correspondence information; Equipped with The first correspondence information indicates a correspondence relationship between a plurality of images of the space, the non-image information, and an abnormality that is likely to occur in the space. The non-image information includes voice data including instructions issued from outside the space to a surgeon in the space, the detection unit detects a state in which the abnormality is likely to occur when an inconsistency occurs between the image and the instruction included in the audio data. Medical information processing equipment.

2. the image acquired by the acquisition unit is a video, The detection unit detects a state in which the abnormality is likely to occur according to at least one of a moving speed and a moving direction of an object or a human depicted in the video acquired by the acquisition unit. The medical information processing device according to claim 1 .

3. The first correspondence information is a plurality of trained models that are trained for each type of the non-image information and indicate a correspondence relationship between a plurality of images of the space and an image region in which an abnormality is likely to occur in the plurality of images; The detection unit detects a risk area, which is an image area in which the abnormality is likely to occur in the image acquired by the acquisition unit, by inputting the image acquired by the acquisition unit into a trained model corresponding to non-image information acquired by the acquisition unit. The medical information processing device according to claim 1 .

4. The first correspondence information is a trained model indicating a correspondence relationship between a plurality of images of the space, the non-image information, and an image region in which an abnormality is likely to occur in the plurality of images, The detection unit detects a risk area, which is an image area in which the abnormality is likely to occur in the image acquired by the acquisition unit, by inputting the image acquired by the acquisition unit and the non-image information into the trained model. The medical information processing device according to claim 1 .

5. The non-image information includes at least one of procedure information related to the procedure, subject information related to the subject, surgeon information related to a surgeon performing the procedure, equipment information related to equipment used in the procedure, and medical institution information related to a medical institution in which the space is provided. The medical information processing device according to claim 1 .

6. and a notification unit that notifies the user that the abnormality is likely to occur when the detection unit detects that the abnormality is likely to occur. The medical information processing device according to claim 1 .

7. and an explanation generation unit that generates a sentence explaining the abnormality that is likely to occur in the risk area detected by the detection unit based on second correspondence information indicating a correspondence relationship between a plurality of risk areas in a plurality of images obtained by capturing the space and a phrase explaining the abnormality that is likely to occur in the plurality of risk areas, The notification unit outputs the sentence generated by the explanation generation unit. The medical information processing device according to claim 6 .

8. A learning unit that generates the trained model by learning a relationship between a pre-abnormality occurrence image in which the space is captured at a time before the occurrence of the abnormality during the procedure and a pre-abnormality occurrence image region in which a location where the abnormality occurs is depicted in the pre-abnormality occurrence image, The medical information processing device according to claim 3 .

9. An imaging device and a medical information processing device, The imaging device images a space in which a procedure on a subject is being performed, The medical information processing device includes: an acquisition unit that acquires an image captured by the imaging device and non-image information related to at least one of the procedure, the subject, or the space; a detection unit that detects a state in which an abnormality is likely to occur in the space based on the image acquired by the acquisition unit, information on the non-image acquired by the acquisition unit, and first correspondence information, the first correspondence information indicates a correspondence relationship between a plurality of images of the space, the non-image information, and an abnormality that is likely to occur in the space, The non-image information includes voice data including instructions issued from outside the space to a surgeon in the space, the detection unit detects a state in which the abnormality is likely to occur when an inconsistency occurs between the image and the instruction included in the audio data. system.

10. an acquisition unit that acquires an image of an examination room in which a procedure on a subject is being performed and non-image information related to at least one of the procedure, the subject, or the examination room; a detection unit that detects a state in which an abnormality is likely to occur in the examination room based on the image acquired by the acquisition unit, the non-image information acquired by the acquisition unit, and first correspondence information; Equipped with the first correspondence information indicates a correspondence relationship between a plurality of images of the examination room, the non-image information, and an abnormality that is likely to occur in the examination room; the non-image information includes voice data including instructions issued from outside the examination room to an operator in the examination room; the detection unit detects a state in which the abnormality is likely to occur when an inconsistency occurs between the image and the instruction included in the audio data. X-ray diagnostic equipment.

11. an acquisition unit that acquires an image of a space in which a procedure on a subject is being performed and voice data including instructions issued from outside the space to an operator in the space; A detection unit that detects a state in which an abnormality is likely to occur in the space by inputting the image and the voice data acquired by the acquisition unit to a trained model that outputs information on an abnormality that is likely to occur in the space based on an image of the space and the voice data; Equipped with the detection unit detects a state in which the abnormality is likely to occur when an inconsistency occurs between the image and the instruction included in the audio data. A medical information processing device.

Images