Medical information processing device, medical system, medical information processing method, and program

The medical information processing device addresses delays in remote diagnosis by generating synchronized medical images based on identified delays and organ motion, ensuring accurate image provision for practitioners.

JP7582821B2Active Publication Date: 2024-11-13CANON MEDICAL SYST CORP
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2020150584
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-09-08
Publication Date
2024-11-13
Estimated Expiration
2040-09-08

AI Technical Summary

Technical Problem

In remote diagnosis and treatment, there is a delay between the actual condition of the treatment target and the condition depicted in medical images due to communication issues, leading to inappropriate image provision for practitioners.

Method used

A medical information processing device that acquires image information of organs moving periodically, identifies delay amounts, and generates medical images based on these delays and the organ's periodic motion to provide synchronized and appropriate images.

Benefits of technology

The solution enables practitioners to receive timely and accurate medical images, reducing the impact of communication delays and enhancing remote diagnosis and treatment efficacy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007582821000001
    Figure 0007582821000001
  • Figure 0007582821000002
    Figure 0007582821000002
  • Figure 0007582821000003
    Figure 0007582821000003
Patent Text Reader

Abstract

To provide a user with a more appropriate medical image in diagnosis and medical practice on a subject.SOLUTION: A medical information processing apparatus of an embodiment has an acquisition unit, a specification unit, and a creation unit. The acquisition unit acquires image information in a plurality of time phases on an organ that makes a periodic movement. The specification unit specifies at least one of a first delay amount indicating a first delay related to display based on the image information in the plurality of time phases, and a second delay amount indicating a second delay related to processing on a predetermined object included in the image information. The creation unit creates a medical image based on at least one of the first delay amount and the second delay amount and the periodic movement of the organ.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The embodiments disclosed in this specification and the drawings relate to a medical information processing device, a medical system, a medical information processing method, and a program. [Background technology]

[0002] In the medical industry, the demand for remote diagnosis and remote treatment is increasing due to the shortage of doctors in rural areas and the shortage of doctors familiar with new infectious diseases. When performing remote diagnosis and remote treatment, the practitioner performs the diagnosis and treatment while checking images including the object captured on-site. However, due to the influence of the communication environment, data volume, etc., there are cases where a delay occurs between the actual condition of the treatment object and the condition of the treatment object contained in the image. As a result, there are cases where appropriate medical images cannot be provided to users such as practitioners. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-522975 [Patent Document 2] Patent No. 4491420 [Patent Document 3] Special Publication No. 5367215 Summary of the Invention [Problem to be solved by the invention]

[0004] One of the problems to be solved by the embodiments disclosed in this specification and the drawings is to provide a user with more appropriate medical images for diagnosing and treating a subject. However, the problems to be solved by the embodiments disclosed in this specification and the drawings are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems. [Means for solving the problem]

[0005] A medical image processing device according to an embodiment includes an acquisition unit, an identification unit, and a generation unit. The acquisition unit acquires image information of a plurality of time phases relating to an organ that performs periodic motion. The identification unit identifies at least one of a first delay amount indicating a first delay relating to a display based on the image information of the plurality of time phases and a second delay amount indicating a second delay relating to a process for a predetermined object included in the image information. The generation unit generates a medical image based on at least one of the first delay amount and the second delay amount and the periodic motion of the organ. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram showing an example of a medical system 1 including a medical information processing apparatus according to a first embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of a medical information processing apparatus 200 according to a first embodiment. [Diagram 3] FIG. 2 is a diagram for explaining a specific function 243. [Figure 4] FIG. 13 is a diagram showing an example of the contents of delay information 256. [Diagram 5] 5A to 5C are diagrams for explaining medical images generated by a generation function 244 in the first embodiment. [Figure 6] FIG. 1 is a diagram for explaining generation of a medical image. [Figure 7] FIG. 2 is a diagram for explaining an image IM1 displayed on a display 230. [Figure 8] 4 is a flowchart showing an example of a flow of processing executed by the medical information processing apparatus 200 in the first embodiment. [Figure 9] FIG. 13 is a diagram showing an example of a medical information processing apparatus 200A according to a second embodiment. [Figure 10] FIG. 11 is a diagram for explaining the function of a generation function 244A in the second embodiment. [Figure 11] 10 is a flowchart showing an example of a flow of processing executed by a medical information processing apparatus 200A in the second embodiment. [Figure 12] FIG. 13 is a diagram showing an example of a medical device 100B according to a third embodiment. [Figure 13] FIG. 13 is a diagram showing an example of a medical information processing apparatus 200B according to the third embodiment. [Figure 14] 13 is a flowchart showing an example of the flow of processing executed by a medical information processing apparatus 200B in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, a medical information processing device, a medical system, a medical information processing method, and a program will be described with reference to the drawings.

[0008] (First embodiment) 1 is a diagram showing an example of a medical system 1 including a medical information processing device of the first embodiment. The medical system 1 includes a medical device 100 and a medical information processing device 200. The medical device 100 and the medical information processing device 200 are connected via a network NW. The network NW includes, for example, a wide area network (WAN), a local area network (LAN), the Internet, a dedicated line, a wireless base station, a provider, and the like.

[0009] The medical device 100 performs a predetermined process on, for example, a periodic organ of the subject P. The medical device 100 is, for example, an X-ray diagnostic device that generates images for diagnosing or treating the subject P using X-rays or the like. Hereinafter, images generated by the medical device 100 are referred to as "intraoperative images." The intraoperative images include images at the time of diagnosis or treatment. The intraoperative images may also include images before diagnosis or treatment. The medical device 100 may also be, for example, an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a nuclear medicine diagnosis device, or the like. The medical device 100 may also perform a process according to an instruction from the medical information processing device 200.

[0010] The medical information processing device 200 transmits and receives information to and from the medical device 100 via a network NW. The medical information processing device 200 may be installed at a remote location that is a predetermined distance or more away from the medical device 100, or may be set at a location less than the predetermined distance. The medical information processing device 200 is realized by one or more processors. For example, the medical information processing device 200 may be a computer included in a cloud computing system, or may be a computer (standalone computer) that operates independently without relying on other devices. The medical information processing device 200 provides image information to a user of the medical information processing device 200 based on information from the medical device 100, and transmits instruction information from the user to the medical device 100 to operate the medical device 100. The user is, for example, a practitioner who diagnoses or performs treatment on the subject P.

[0011] 1, the medical system 1 includes one medical device 100 and one medical information processing device 200, but is not limited to this and may include multiple devices of either one or both. In addition, the medical device 100 or the medical information processing device 200 may be integrated with the other device.

[0012] Next, the configuration of the medical device 100 will be specifically described. The medical device 100 shown in Fig. 1 includes, for example, an imaging device 110 and a console device 140. The imaging device 110 includes, for example, a high-voltage generator 111, an arm unit 112, an X-ray generator 113, an X-ray detector 114, an arm movement mechanism 115, a bed 116, a bed movement mechanism 117, a device drive mechanism 118, and an electrocardiograph 119. Hereinafter, the combination of the arm unit 112, the X-ray generator 113, and the X-ray detector 114 may be referred to as an "imaging system." The device drive mechanism 118 is an example of a "drive unit."

[0013] The high voltage generator 111 generates a high voltage under the control of a processing circuit 150 (described later), and supplies the generated high voltage to the X-ray generator 113 .

[0014] The arm unit 112 is a holding device having a curved or bent shape such as a C-shape in part or in whole. The arm unit 112 can rotate and move when an arm moving mechanism 115 is driven under the control of a mechanism control circuit 146 described later. The arm unit 112 holds an X-ray generator 113 at one end and an X-ray detector 114 at the other end so as to face the X-ray generator 113. The X-ray generator 113 includes, for example, an X-ray tube (not shown) and a collimator (not shown). The X-ray tube generates X-rays from an X-ray focus when a high voltage (tube voltage) is applied from a high voltage generator 111 and a tube current is supplied. The collimator is attached to an emission window of the X-ray tube and adjusts an X-ray irradiation field on a detection surface of the X-ray detector 114. The collimator adjusts the X-ray irradiation field, thereby reducing unnecessary exposure to the subject P. The X-ray detector 114 includes, for example, a plurality of X-ray detection elements. The multiple X-ray detection elements are arranged in a two-dimensional array. A two-dimensional array detector is called an FPD (Flat Panel Display). Each element of the FPD detects X-rays emitted from an X-ray generator 113 and transmitted through a subject. Each element of the FPD outputs an electrical signal corresponding to the intensity of the detected X-rays. Note that the X-ray detector 114 may be composed of a combination of an image intensifier and a TV camera instead of the above-mentioned FPD.

[0015] The arm moving mechanism 115 rotates and moves the arm unit 112 to a predetermined position on the three-dimensional coordinate system (XYZ coordinate system) under the control of the mechanism control circuit 146 described later. For example, the arm moving mechanism 115 moves the arm unit 112 so that the diagnostic target of the subject P is included in an imaging range corresponding to the X-ray irradiation field of the imaging system.

[0016] The bed 116 movably supports a top board (not shown) on which the subject P is placed. The top board is moved by driving a bed moving mechanism 117 under the control of the mechanism control circuit 146. In the example of FIG. 1, the subject P is placed on the top board facing up. The bed moving mechanism 117 moves the top board to a predetermined position on the three-dimensional coordinate system (XYZ coordinate system) under the control of the mechanism control circuit 146 described later.

[0017] The device driving mechanism 118 performs a predetermined drive on the medical device 120 under the control of the mechanism control circuit 146 described later. For example, the device driving mechanism 118 moves the medical device 120 to a predetermined position on a three-dimensional coordinate system (XYZ coordinate system), changes the shape of the medical device 120, or injects a substance into the subject P via the medical device 120. The medical device 120 includes, for example, Mitra Clip (registered trademark), a catheter, an injector, a puncture needle, a stent, and the like. The medical device 120 may also include a robot arm, and the like.

[0018] The electrocardiograph 119 acquires an electrocardiogram (ECG) of the subject P to which a terminal (not shown) is attached, and outputs to the processing circuitry 150 electrocardiogram data in which the acquired electrocardiogram waveform is associated with time information.

[0019] The console device 140 includes, for example, a memory 141, a communication interface 142, an input interface 143, a display 144, an aperture control circuit 145, a mechanism control circuit 146, and a processing circuit 150. The memory 141 is realized by, for example, a semiconductor memory element such as a random access memory (RAM), a flash memory, a hard disk, an optical disk, or the like. The memory 141 stores, for example, an image captured by an imaging system (an example of an intraoperative image), a program, and various other information. These data may be stored in an external memory with which the medical device 100 can communicate, instead of the memory 141 (or in addition to the memory 141). The external memory is controlled by a cloud server that manages the external memory, for example, by the cloud server accepting a read / write request.

[0020] The communication interface 142 includes a communication interface such as a NIC (Network Interface Controller). The communication interface 142 communicates with the imaging device 110 and the medical information processing device 200, and outputs acquired information to the processing circuitry 150, etc. The communication interface 142 may also be controlled by the processing circuitry 150 to transmit information to other devices connected via the network NW.

[0021] The input interface 143 accepts various input operations by a user and outputs an electric signal indicating the content of the accepted input operation to the processing circuit 150. For example, the input interface 143 accepts input operations such as collection conditions for collecting imaging data and image processing conditions for performing predetermined processing on an image. For example, the input interface 143 is realized by a mouse, a keyboard, a touch panel, a drag ball, a switch, a button, a joystick, a foot pedal, a camera, an infrared sensor, a microphone, etc. The input interface 143 may be provided in the imaging device 110. In addition, the input interface 143 may be realized by a display device (for example, a tablet terminal) capable of wireless communication with the main body of the console device 140.

[0022] The display 144 displays various information. For example, the display 144 displays intraoperative images generated by the processing circuit 150, GUI (Graphical User Interface) images that accept various operations by a user, and the like. The display 144 is, for example, a liquid crystal display, a CRT (Cathode Ray Tube), an organic EL (Electro Luminescence) display, or the like. The display 144 may be provided in the imaging device 110. The display 144 may be a desktop type, or may be a display device (for example, a tablet terminal) that can wirelessly communicate with the main body of the console device 140.

[0023] The aperture control circuit 145 controls the irradiation range (X-ray irradiation field) of the X-rays irradiated to the subject P, for example, based on the control by the processing circuitry 150. Information on the irradiation range is also called FOV (Field Of View) information. The aperture control circuit 145 controls the irradiation range of the X-rays by, for example, adjusting the opening of the aperture blades of the collimator of the X-ray generator 113.

[0024] The mechanism control circuit 146 controls the arm moving mechanism 115 and the bed moving mechanism 117 based on the control by the processing circuitry 150, thereby changing the imaging range in the imaging system, the relative position of the imaging system with respect to the subject P, the imaging angle, etc. Furthermore, the mechanism control circuit 146 controls the device driving mechanism 118 based on the control by the processing circuitry 150, thereby controlling the operation, etc. of the medical device 120. Note that the mechanism control circuit 146 may control the target mechanism based on an operation instruction (e.g., a command operation) from the medical information processing device 200. Furthermore, the mechanism control circuit 146 outputs information on the mechanism control content to the processing circuitry 150.

[0025] The processing circuit 150 controls the overall operation of the medical device 100. The processing circuit 150 includes, for example, a control function 151, an image data generating function 152, an image processing function 153, and a display control function 154. The processing circuit 150 realizes these functions by, for example, a hardware processor executing a program stored in the memory 141. The hardware processor means, for example, a circuit such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), a programmable logic device (for example, a Simple Programmable Logic Device (SPLD) or a Complex Programmable Logic Device (CPLD), or a Field Programmable Gate Array (FPGA)). Instead of storing the program in the memory 141, the program may be directly incorporated into the circuit of the hardware processor. In this case, the hardware processor realizes the functions by reading and executing the program embedded in the circuit. The hardware processor is not limited to being configured as a single circuit, but may be configured as one hardware processor by combining multiple independent circuits to realize each function. Also, multiple components may be integrated into one hardware processor to realize each function. The image processing function 153 is an example of an "image processing unit."

[0026] Each component of the console device 140 or the processing circuit 150 may be distributed and realized by multiple pieces of hardware. The processing circuit 150 may be realized by a processing device capable of communicating with the console device 140, rather than being a component of the console device 140. The processing device is, for example, a workstation connected to one medical device, or a device (for example, a cloud server) connected to multiple medical devices and collectively executing processing equivalent to that of the processing circuit 150 described below.

[0027] The control function 151 controls various functions of the processing circuit 150 based on, for example, an input operation received by the input interface 143 or an operation content acquired from the medical information processing device 200. Specifically, the control function 151 causes, for example, an imaging system to capture intraoperative images including a target part (for example, a heart, lungs, a valve body, a blood vessel, etc.) of the subject P at a plurality of time points. The target part is, for example, a part to be diagnosed or operated on. The target part includes an organ that performs a periodic operation. The periodic operation is, for example, pulsation when the part is the heart, and operation due to a respiratory cycle when the part is the lung. The target part may also include a part having a specific form or function. The intraoperative image may also include the medical device 120 during the operation.

[0028] When performing the above-mentioned imaging, the control function 151 performs predetermined control on the high voltage generator 111, the device driving mechanism 118, the electrocardiograph 119, the aperture control circuit 145, and the mechanism control circuit 146 based on the instruction contents from the input interface 143, etc., the imaging contents, etc. The control function 151 also causes the image data generating function 152, the image processing function 153, and the display control function 154 to perform predetermined processing on the captured time-series images. The control function 151 also performs control to transmit the captured intraoperative images (e.g., image information of multiple time phases), parameter information at the time of imaging (e.g., aperture control contents and mechanism control contents), measurement results of the electrocardiograph 119, etc. to the medical information processing device 200 via the network NW.

[0029] The image data generating function 152 generates image data using an electrical signal converted from X-rays by the X-ray detector 114, and stores the generated image data in the memory 141. For example, the image data generating function 152 performs current-voltage conversion, A (Analog) / D (Digital) conversion, and parallel-serial conversion on the electrical signal received from the X-ray detector 114 to generate image data. Then, the image data generating function 152 stores the generated image data in the memory 141.

[0030] The image processing function 153 generates intraoperative images by performing various image processing on the image data stored in the memory 141. For example, the image processing function 153 may perform noise reduction processing on the image data stored in the memory 141 using an image processing filter such as a moving average (smoothing) filter, a Gaussian filter, or a median filter, or may perform preprocessing including positional deviation correction and noise removal on each of a group of multiple X-ray images captured over time.

[0031] The display control function 154 controls the display content and display mode displayed on the display 144. Specifically, the display control function 154 causes the display 144 to display, for example, a GUI image for accepting instructions from a user and image data generated by the image data generation function 152. In addition, the display control function 154 may cause the display 144 to display analysis results received from the medical information processing device 200.

[0032] Next, the configuration of the medical information processing device 200 will be described. Fig. 2 is a diagram showing an example of the medical information processing device 200 according to the first embodiment. The medical information processing device 200 includes, for example, a communication interface 210, an input interface 220, a display 230, a processing circuit 240, and a memory 250. The display 230 is an example of a "display unit."

[0033] The communication interface 210 includes, for example, a communication interface such as a NIC. The communication interface 210 communicates with the medical device 100 via the network NW and receives information from the medical device 100. The communication interface 210 outputs the received information to the processing circuit 240. The communication interface 210 may also transmit information to another device connected via the network NW under the control of the processing circuit 240. The other device may be, for example, a terminal device that can be used by an image reader such as a doctor or a nurse.

[0034] The input interface 220 accepts various input operations from a user, converts the accepted input operations into electrical signals, and outputs the electrical signals to the processing circuit 240. For example, the input interface 220 is realized by a mouse, a keyboard, a trackball, a switch, a button, a joystick, a touch panel, or the like. The input interface 220 may also be realized by a user interface that accepts audio input from a microphone, for example. When the input interface 220 is a touch panel, the display 230 may be formed integrally with the input interface 220.

[0035] The display 230 displays various information. For example, the display 230 displays an image generated by the processing circuit 240 or an image transmitted by the medical device 100, or displays a GUI for receiving various input operations from a user. For example, the display 230 is an LCD, a CRT display, an organic EL display, or the like.

[0036] The processing circuit 240 includes, for example, a control function 241, an acquisition function 242, a specification function 243, a generation function 244, and a display control function 245. The processing circuit 240 realizes these functions by, for example, a hardware processor as described above executing a program stored in the memory 250. Instead of storing the program in the memory 250, the program may be directly incorporated into the circuit of the hardware processor. In addition, some or all of the multiple functions included in the processing circuit 240 may be realized by hardware (circuitry) such as an LSI, ASIC, or FPGA, or may be realized by cooperation between software and hardware. In addition, the above-mentioned program may be stored in the memory 250 in advance, or may be stored in a removable storage medium such as a DVD or CD-ROM, and may be installed in the memory 250 from the storage medium by mounting the storage medium in a drive device of the medical information processing device 200. The acquisition function 242 is an example of an "acquisition unit." The specification function 243 is an example of an "specification unit." The generating function 244 is an example of a “generating unit.” The display control function 245 is an example of a “display control unit.”

[0037] The memory 250 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, a hard disk, an optical disk, or the like. These non-transient storage media may be realized by other storage devices connected via a network NW, such as a NAS or an external storage server device. The memory 250 may also include a transient storage medium such as a ROM or a register. The memory 250 stores, for example, time phase image data 252, electrocardiogram data 254, delay information 256, a program, and various other information. The time phase image data 252 is generated by the medical device 100.

[0038] The control function 241 controls various functions of the medical information processing device 200 based on, for example, an input operation accepted by the input interface 220. Specifically, the control function 241 controls the acquisition of various information by the acquisition function 242 via the communication interface 210, the identification of the amount of delay by the identification function 243, the generation of medical images by the generation function 244, and the display of images such as medical images by the display control function 245. In addition, the control function 241 controls the transmission of operation information (operation instructions, etc.) for the medical device 100 inputted by the input interface 220 to the medical device 100 via the communication interface 210.

[0039] The acquisition function 242 causes the communication interface 210 to communicate with the medical device 100, and acquires information such as intraoperative images, electrocardiogram data, and mechanical control details from the communicating medical device 100. The acquisition function 242 stores the acquired intraoperative images as time-phase image data 252 in the memory 250, and stores the electrocardiogram data as electrocardiogram data 254 in the memory 250. Therefore, since the time-phase image data 252 and the electrocardiogram data 254 contain time information, it is possible to acquire each piece of synchronized information based on the time information.

[0040] The identification function 243 identifies at least one of a first delay amount indicating a first delay related to a display based on image information of multiple time phases included in the time phase image data 252 and a second delay amount indicating a second delay related to a process for a predetermined object included in the image information of multiple time phases. The delay amount is, for example, information related to a period. The predetermined object is, for example, an object that does not perform a periodic motion, or an object whose periodic motion is much smaller than the non-periodic motion. For example, the predetermined object includes an object such as the medical device 120 that is moved by the practitioner. In the following description, the predetermined object is assumed to be the medical device 120.

[0041] The generation function 244 generates a medical image based on at least one of the first delay amount and the second delay amount identified by the identification function 243 and the periodic movement of a target area (e.g., an organ) included in the time phase image data 252.

[0042] The display control function 245 causes the display 230 to display images including the medical images (generated images) generated by the generation function 244. The display control function 245 may also cause the display 230 to display information necessary for processing in the medical information processing device 200.

[0043] Next, the specification function 243 will be specifically described. For example, the specification function 243 specifies how much delay will occur when transmitting a certain amount of data in communication with the medical device 100. For example, the specification function 243 transmits and receives predetermined data to and from the medical device 100 before or at the time when diagnosis or treatment is started, and specifies the first delay amount and the second delay amount based on the actual measurement result.

[0044] 3 is a diagram for explaining the identification function 243. For example, the identification function 243 acquires a first response time from when an image request is transmitted to the medical device 100 until when image data corresponding to the image request is received, and identifies a first delay amount based on the acquired first response time. In this case, the identification function 243 subtracts the time from when the image request is transmitted to when the medical device 100 receives the image request from the first response time, and calculates the subtracted value as the first delay amount. The identification function 243 may also subtract the transmission time added to the image data by the medical device 100 when transmitting the image data from the reception time of the image data, and identify the subtracted time as the first delay amount.

[0045] The image data transmitted by the medical device 100 may be, for example, specific data whose data format or data amount is predetermined, or may be a data format used for an intraoperative image or an actual intraoperative image of the subject P. The specifying function 243 may specify the first delay amount based on the result of receiving a plurality of different types of image data.

[0046] The identifying function 243 also acquires a second response time from the transmission of an operation request (e.g., a command request) for operating the device driving mechanism 118 of the medical apparatus 100 to the reception of information (operation completion information) indicating that the operation corresponding to the operation request has been completed, and identifies a second delay amount based on the acquired second response time. In this case, the identifying function 243 acquires a time from the transmission of the operation completion information from the medical apparatus 100 to the reception of the operation completion information by the medical apparatus 100, and calculates a value obtained by subtracting the acquired time from the second response time as the second delay amount. The identifying function 243 may also perform a similar process on components (e.g., the bed 116 and the arm unit 112) other than the device driving mechanism 118 provided in the medical apparatus 100, and identify the second delay amount for each component (mechanism). The identifying function 243 may also identify a final second delay amount by averaging the second delay amounts obtained from each component or performing a weighting process.

[0047] Furthermore, the identification function 243 may store information on the identified first delay amount and second delay amount as delay information 256 in the memory 250. Fig. 4 is a diagram showing an example of the contents of the delay information 256. The delay information 256 is information in which the first delay amount [msec] and the second delay amount [msec] are associated with a device ID as identification information for identifying the medical device 100 and communication conditions with the medical device 100. The communication conditions may include information on, for example, a communication method, a communication time, a communication carrier (communication operator), and a data amount.

[0048] For example, by storing the delay information 256, the identifying function 243 can acquire the first delay amount and the second delay amount by referring to the delay information 256, without calculating the delay amount described above. In this case, the identifying function 243 refers to the delay information 256 based on the device ID and communication conditions of the medical device 100, and acquires the first delay amount and the second delay amount associated with the matching device ID and communication conditions. Furthermore, if there is no match, the identifying function 243 identifies the first delay amount and the second delay amount as described above, and updates the delay information 256. This makes it possible to acquire the first delay amount and the second delay amount more efficiently.

[0049] Next, the generating function 244 will be specifically described. The generating function 244 predicts the state of an organ without delay based on one or both of the first delay amount and the second delay amount, and acquires first image information including the organ from the time phase image data 252 based on the predicted state of the organ. The generating function 244 also acquires second image information related to the medical device 120 from the time phase image data 252. The generating function 244 then generates a medical image based on the first image information and the second image information.

[0050] FIG. 5 is a diagram for explaining medical images generated by the generating function 244 in the first embodiment. In the example of FIG. 5, for convenience of explanation, an image 310 showing the state of a heart valve (specifically, a mitral valve) and an image 320 showing the state of a MitraClip are shown as examples of organs and medical devices 120 obtained from an intraoperative image 300 transmitted by a medical apparatus 100. The "state" includes, for example, a position and a shape. In addition, at the time T shown in FIG. 5, the time t11 is the earliest, followed by the times t12, t13, and t14 in the order of latest. In addition, in the example of FIG. 5, the intraoperative image at time t*, the image 310 showing the state of an organ, the image 320 showing the state of a MitraClip, and the medical image 330 are represented by an intraoperative image 300(t*), an image 310(t*), an image 320(t*), and a medical image 330(t*), respectively.

[0051] The generating function 244 acquires the period of one heart beat based on, for example, the shape and size of the electrocardiogram waveform included in the electrocardiogram data 254. The generating function 244 may also acquire the interval of the waveform (for example, the sinus period). Based on the acquired period, the generating function 244 acquires the opening and closing period of the cardiac valve and the opening and closing state of the cardiac valve over time. The opening and closing state of the cardiac valve includes, for example, the degree of opening of the valve (for example, 20%, 80%) and information on whether the cardiac valve is changing to open or close.

[0052] The generating function 244 also predicts the state of the heart valve after the period corresponding to the first and second delays has elapsed from the generated cardiac cycle, the first delay amount, the second delay amount, and the intraoperative image. For example, the intraoperative image 300 (t12) at time t12 shown in FIG. 5 is the latest image among the intraoperative images (image information of multiple time phases) acquired from the medical device 100. The intraoperative image 300 (t12) is an image delayed by the first delay amount P1 from the actual state of the heart valve and MitraClip on the site (medical device 100) side. In other words, the state of the heart valve and MitraClip on site at time t12 is displayed at time t13 after the first delay amount P1 has elapsed. In addition, when a user of the medical information processing device 200 operates the MytraClip in response to an operation request (operation instruction) from the medical information processing device 200, a second delay amount P2 occurs from the time the operation instruction is sent from the medical information processing device 200 until it reaches the medical device 100 and processing in response to the request is executed.

[0053] Therefore, the generating function 244 predicts the state of the valve after a period corresponding to the sum of the first delay amount P1 and the second delay amount P2 has elapsed, and obtains the state of the valve after a period corresponding to the sum has elapsed (for example, the state of the valve at time t14) from the open / closed state of the cardiac valve at each time acquired from the electrocardiogram data 254. Then, the generating function 244 obtains an image of the same valve state as the obtained state from the intraoperative images 300 of multiple time phases before (past) time t12. The same valve may include valves within a predetermined error range. For example, the generating function 244 obtains images of the same degree of opening of the valve (including a predetermined error range) and the same state change (for example, a state changing to open) based on the open / closed state of the cardiac valve. In the example of FIG. 5, it is assumed that the intraoperative image 300 (t11) has been obtained. The intraoperative image 300 (t11) is an example of the first image information.

[0054] In addition, in the relationship between the first delay amount P1 and the second delay amount P2, when one delay amount is greater than the other delay amount by a threshold value or more, the generating function 244 may acquire the first image information using only the larger delay amount. In addition, when one delay amount cannot be identified by the identifying function 243, the generating function 244 may acquire the first image information using the other identified delay amount. In addition, when one delay amount is selected by the user via the input interface 220, the generating function 244 may acquire the first image information using only that delay amount.

[0055] The generating function 244 also obtains the MytraClip image 320 from the intraoperative image 300 (t12), which is the latest image among the intraoperative images obtained by the medical information processing device 200. The intraoperative image 300 (t12) is an example of second image information. The generating function 244 then generates a medical image 330 (t12) to be displayed at time t12 based on the intraoperative image 300 (t11) and the intraoperative image 300 (t12).

[0056] FIG. 6 is a diagram for explaining the generation of medical images. The generating function 244 also extracts an image 310 (t11) showing the shape of the heart valve from the intraoperative image 300 (t12). The generating function 244 may delete an image 320 (t11) showing the state of the MitraClip from the intraoperative image 300 (t12) based on the brightness difference, shape information, difference in feature information from the surrounding images, etc., and may interpolate the deleted image area based on information from the surrounding images and past intraoperative images. The generating function 244 may also obtain an image showing the state of the valve corresponding to the image 310 (t11) from an intraoperative image that does not include an image showing the state of the MitraClip.

[0057] The generating function 244 also obtains an image 320 (t12) showing the state of the MitraClip by segmentation processing based on, for example, brightness difference, shape information, and differences in feature information from the surrounding images from the intraoperative image (t12). The generating function 244 then generates a medical image 330 (t12) by superimposing the image 320 (t12) on the image 310 (t11). The medical image 330 (t12) is the actual state of the valve and the MitraClip at the time when the operation instruction (operation request) from the medical information processing device 200 reaches the medical device 100, predicted based on the first delay amount P1 and the second delay amount.

[0058] Next, the display control function 245 will be specifically described. The display control function 245 generates an image including information acquired by the acquisition function 242 and a medical image generated by the generation function 244, and displays the generated image on the display 230. FIG. 7 is a diagram for explaining an image IM1 displayed on the display 230. Note that the layout, size, type, and display contents of the image IM1 are not limited to the example of FIG. 7. The image IM1 includes, for example, display areas A1 to A4. The display area A1 is, for example, an acquired image display area in which an image (acquired image) 400 acquired from the medical device 100 is displayed. The acquired image 400 includes, for example, an image 410 of a target part of the subject P (for example, a heart including a valve) and an image 420 of a medical device (for example, a MitraClip) 120.

[0059] The display area A2 is a delay information display area in which, for example, information on a first delay amount (imaging delay) and a second delay amount (command delay) identified by the identification function 243 is displayed. Note that, for example, an image showing information on a response time with the medical device 100, information on a communication environment in the network NW, and information on a distance between the medical device 100 and the medical information processing device 200 may be displayed in the display area A2.

[0060] The display area A3 is an electrocardiogram data display area that displays information related to the electrocardiogram data 254 acquired from the medical device 100. Note that the display area A3 may also display information related to the pulsation period acquired by the generating function 244 (for example, information related to the start and end times of the pulsation).

[0061] The display area A4 is a medical image display area where a medical image (generated image) 500 generated by the generating function 244 is displayed. The medical image 500 is, for example, image information of multiple time phases generated by the generating function 244. The medical image 500 includes an image 510 of a target site predicted to be in a state at the time when an operation instruction or the like reaches the medical apparatus 100 and an image 520 of the medical device 120. By displaying the image IM10, a more appropriate medical image can be provided to the user (practitioner) in the diagnosis and treatment of the subject P. In addition, for example, when remotely operating the medical apparatus 100 from the medical information processing device 200, the user can operate the medical apparatus 100 at a more appropriate timing for the valve 510V in operation by transmitting an operation instruction with reference to the medical image 500 that takes the delay amount into consideration.

[0062] Here, depending on the state of examination or treatment of the organ using the medical device 100, the type of organ, and the amount of deformation, the image or the content of the image to be focused on may differ. Therefore, the display control function 245 may change the display mode of a part or all of the images displayed on the display 230 based on the state of examination or treatment of the organ, the type of organ, and the amount of deformation. Examples of changing the display mode include changing the size, color, or brightness of one or both of the acquired image 400 and the medical image 500, highlighting the image by surrounding it with a predetermined color (e.g., red, etc.), hiding, masking, or blurring an image with a low priority to be recognized by the practitioner, etc.

[0063] For example, there are cases where it is desired to check the blood flow distribution in an intraoperative image. A generated image obtained by synthesizing images of the valve shape and the medical device at different times is an image that ignores the geometric relationship between the structure and the fluid. Therefore, when the generated image is displayed, the surgeon feels uncomfortable with the valve shape, the position of the medical device, and the fluid distribution. Therefore, in the above-mentioned case, the display control function 245 changes the display mode so that the acquired image 400 is emphasized. In addition, when the shape of the heart is significantly deformed, the display control function 245 may change the display mode so that the acquired image 400 is emphasized. It is to be noted that, whether or not the shape of the heart is significantly deformed is determined to be significantly deformed, for example, when the magnitude of deformation of the anatomical feature amount of the organ included in the intraoperative image is equal to or greater than a predetermined magnitude. This includes making it possible for the user to more reliably view high-priority images that need to be viewed by the surgeon.

[0064] Next, a flow of processing executed by the medical information processing device 200 in the first embodiment will be described. In the following, an example will be described in which a practitioner uses the medical information processing device 200 to remotely operate the medical device 100 to perform a treatment on a target part (e.g., a valve) of the heart of a subject P. FIG. 8 is a flowchart showing an example of a flow of processing executed by the medical information processing device 200 in the first embodiment. In the example of FIG. 8, the acquisition function 242 acquires an intraoperative image from the medical device 100 (step S100). Next, the acquisition function 242 acquires the cardiac cycle of the subject (step S110). Next, the identification function 243 identifies a first delay amount and a second delay amount (step S120).

[0065] Next, the generating function 244 predicts the state of the target site (heart) after the delay based on the identified first delay amount and second delay amount (step S130). Next, the generating function 244 acquires an image of the target site showing the predicted state from images of multiple time phases (step S140). Next, the generating function 244 acquires an image of the medical device operated by the practitioner from the intraoperative image (step S150). Next, the generating function 244 generates a medical image by superimposing the image of the target site and the image of the medical device (step S160). Next, the display control function 245 causes the display 230 to display the generated medical image (step S170).

[0066] Next, the control function 241 judges whether or not an operation instruction has been received from the input interface 220 (step S180). If it is judged that an operation instruction has been received, the control function 241 transmits the operation instruction to the medical device 100 via the communication interface 210 (step S190). After the process of step S190, or if it is judged that an operation instruction has not been received in the process of step S180, the control function 241 judges whether or not to end the treatment based on information inputted by the input interface 220 (step S200). If it is judged that the treatment will not be ended, the process returns to S100, and if it is judged that the treatment will be ended, the process of this flowchart is ended.

[0067] According to the first embodiment described above, it is possible to provide the practitioner with more appropriate medical images in the diagnosis and treatment of the subject. Specifically, for intraoperative images that move periodically, the shape of the organ after a delay is predicted using the periodicity, and image data of a shape similar to the predicted shape of the organ is obtained from past intraoperative images. Also, for intraoperative images that do not move periodically (e.g., a medical device), an image is obtained from new images by segmentation. Then, an image without delay can be displayed by displaying an image in which the two images are superimposed. Therefore, for example, the effect of delay on a practitioner performing remote operation can be suppressed.

[0068] Second embodiment The second embodiment differs from the first embodiment in that, when an organ that performs periodic motion performs non-periodic motion in a certain period, a medical image is generated using intraoperative images other than the period in which the non-periodic motion occurred. Therefore, the following mainly focuses on the above-mentioned differences. Also, the second embodiment differs from the medical system 1 of the first embodiment in that the medical information processing device 200A is provided instead of the medical information processing device 200. Therefore, the following mainly focuses on the functional configuration of the medical information processing device 200A. Also, the same reference numerals are used for the functional configurations similar to those in the first embodiment, and the description here is omitted. The same applies to the following embodiments.

[0069] Fig. 9 is a diagram showing an example of a medical information processing device 200A in the second embodiment. The medical information processing device 200A shown in Fig. 9 includes, for example, a communication interface 210, an input interface 220, a display 230, a processing circuit 240A, and a memory 250. The processing circuit 240A includes, for example, a control function 241A, an acquisition function 242, a specification function 243, a generation function 244A, a display control function 245, and a determination function 246. The following description will mainly focus on the control function 241A, the generation function 244A, and the determination function 246. The determination function 246 is an example of a "determination unit."

[0070] The control function 241A controls various functions of the medical information processing device 200A based on, for example, an input operation accepted by the input interface 220. Specifically, the control function 241A controls the acquisition of various information by the acquisition function 242 via the communication interface 210, the determination of the amount of delay by the determination function 243, the generation of medical images by the generation function 244A, the display of images such as medical images by the display control function 245, the determination of non-periodic periods by the determination function 246, and the like.

[0071] The determination function 246 determines whether the movement of an organ is periodic or not based on image information of multiple time phases related to the organ. A case where the movement of an organ is not periodic is, for example, a case where the movement of the organ is arrhythmia. In this case, the determination function 246, for example, compares the waveform of the electrocardiogram data 254 with a predetermined rhythm waveform, and determines that the movement of the organ is not periodic if the error is equal to or greater than a threshold, and determines that the movement of the organ is periodic if the error is less than the threshold. An error equal to or greater than the threshold includes, for example, a case where the shape of the waveform differs by a predetermined amount or more, or a case where the interval between the waveforms (for example, sinus cycle) is irregular. The determination function 246 also determines a period in which there is an error equal to or greater than a threshold as an arrhythmia period.

[0072] When the determination function 246 determines that the movement of the organ is not periodic, the generation function 244A generates a medical image using images from the time phase image data 252 during a period other than the period during which the movement is not periodic (arrhythmia period).

[0073] FIG. 10 is a diagram for explaining the function of the generating function 244A in the second embodiment. In the example of FIG. 10, an intraoperative image 300 transmitted by the medical device 100 is shown. In the example of FIG. 10, for example, an image 310 showing a change in the shape of an organ (e.g., a heart valve) that performs a periodic movement and an image 320 showing the position and state of the MitraClip relative to the valve are shown typically. In addition, in the example of FIG. 10, it is assumed that the time t21 is the earliest, followed by the times t22, t23, and t24 in the order of latest. In addition, it is assumed that the determining function 246 has determined that the time t31 to t32 shown in FIG. 10 is an arrhythmia period based on the electrocardiogram data 254 and the like.

[0074] At time t23, the generating function 244A obtains a cycle of one heart beat based on the shape, size, etc. of the electrocardiogram waveform included in the electrocardiogram data 254, and obtains the opening and closing cycle of the heart valve and the opening and closing state of the heart valve for each time based on the obtained cycle. The generating function 244A also predicts the state of the heart valve after the lapse of a period corresponding to the sum of the first delay amount P1 and the second delay amount P2 from the generated heart cycle, the first delay amount, the second delay amount, and the intraoperative image. In the example of FIG. 10, the state of the valve at time t24 is predicted. Then, the generating function 244A obtains an image of the same valve state as the obtained state from the intraoperative images 300 of multiple time phases before time t23. In this case, the generating function 244A obtains an image of the same valve state from intraoperative images other than the time t31 to t32, which is the arrhythmia period. In the example of FIG. 10, it is assumed that the intraoperative image 300 (t21) is obtained. The intraoperative image 300 (t21) is an example of the first image information.

[0075] The generation function 244A also acquires an image 320 (t23) of the MytraClip from an intraoperative image 300 (t23) (second image information), which is the latest image among the intraoperative images acquired by the medical information processing device 200A. Then, the image 310 (t21) and the image 320 (t23) are superimposed to generate a medical image 330 (t23) to be displayed at time t23. In this way, a more appropriate medical image can be generated and provided to the practitioner using images from a period in which the movement of the organ is cyclical.

[0076] Fig. 11 is a flowchart showing an example of the flow of processing executed by the medical image processing device 200A in the second embodiment. The processing in Fig. 11 differs from the processing of steps S100 to S200 shown in Fig. 8 described above in that step S112 is added and step S142 is added instead of step S140. Therefore, the following description will mainly focus on the processing of steps S112 and S142. In addition, in the processing in Fig. 11, the processing of the control function 241 and the generation function 244 in the first embodiment is to be read as the processing of the control function 241A and the generation function 244A.

[0077] After acquiring the cardiac cycle in the process of step S110, the determination function 246 determines whether or not there is an arrhythmia period (step S112), and then executes the process of step S120. In the process of step S112, it may be determined whether or not there is an arrhythmia period, and if it is determined that there is an arrhythmia period, the period may be determined.

[0078] After the process of step S130, the generating function 244 acquires an image of the target region showing the predicted state of the heart from the images of multiple time phases excluding the arrhythmia period (step S142), and performs the processes of step S150 and thereafter.

[0079] In the second embodiment, the judgment function 246 may judge whether the shape of the heart has been significantly deformed or not, instead of diagnosing arrhythmia, and if so, may generate a medical image using intraoperative images from a period other than that.

[0080] According to the second embodiment described above, in addition to achieving the same effects as the first embodiment, it is possible to generate medical images with higher prediction accuracy by not using image data from periods in which the movement of organs is determined to be non-periodic.

[0081] (Third embodiment) The third embodiment differs from the first embodiment in that it includes a tactile presentation device that provides a tactile sensation to the practitioner. Therefore, the following mainly focuses on the above-mentioned differences. Also, the third embodiment differs from the medical system 1 of the first embodiment in that it includes a medical device 100B and a medical information processing device 200B instead of the medical device 100 and the medical information processing device 200. Therefore, the following mainly describes the medical device 100B and the medical information processing device 200B.

[0082] Fig. 12 is a diagram showing an example of a medical device 100B in the third embodiment. The medical device 100B shown in Fig. 12 differs from the medical device 100 shown in Fig. 1 in that it includes a tactile device 131 and a tactile device drive mechanism 132, and further includes a mechanism control circuit 146B instead of the mechanism control circuit 146. Therefore, the following description will mainly focus on the tactile device 131, the tactile device drive mechanism 132, and the mechanism control circuit 146B.

[0083] The tactile device 131 is a member that can be moved in three dimensions (XYZ) by the tactile device drive mechanism 132. The tactile device 131 has, for example, an arm shape and has the function of gripping an object with the arm and moving the gripped object. The tactile device 131 is also provided with a sensor device such as a pressure sensor or an acceleration sensor, which detects the pressure, three-dimensional position, movement direction, etc. during movement or when an object is gripped. The detection result of the sensor device is output to the tactile device drive mechanism 132.

[0084] The tactile device drive mechanism 132 operates the tactile device 131 under the control of the mechanism control circuit 146B, and controls the tactile device 131 so as to operate appropriately based on the detection results of the sensor device of the tactile device 131.

[0085] The mechanism control circuit 146B has the same functions as the mechanism control circuit 146, and in addition, it controls the tactile device drive mechanism 132 to operate the tactile device 131. The mechanism control circuit 146B may also control the tactile device drive mechanism 132 to drive the tactile device 131 based on the operation content instructed from the medical information processing device 200B. The mechanism control circuit 146B also transmits information obtained from the tactile device 131 and the tactile device drive mechanism 132 to the medical information processing device 200B.

[0086] Fig. 13 is a diagram showing an example of a medical information processing device 200B in the third embodiment. The medical information processing device 200B shown in Fig. 13 includes, for example, a communication interface 210, an input interface 220, a display 230, a tactile sense providing device 235, a processing circuit 240B, and a memory 250. The processing circuit 240B includes, for example, a control function 241B, an acquisition function 242B, an identification function 243B, a generation function 244B, a display control function 245, and a tactile sense providing control function 247. The tactile sense providing device 235 and the tactile sense providing control function 247 are examples of a "tactile sense providing unit".

[0087] The tactile sense providing device 235 is a device that provides a tactile sense to the practitioner under the control of the tactile sense providing control function 247. The tactile sense providing device 235 provides a pseudo tactile sensation to the practitioner by transmitting vibrations, reaction forces, electric signals, etc. to the practitioner's body (fingers, hands, etc.). The tactile sense providing device 235 is, for example, a force sense presentation device using an actuator such as a motor, a joystick controller, etc., but may have other configurations.

[0088] The control function 241B controls various functions of the medical information processing device 200B based on, for example, an input operation accepted by the input interface 220. Specifically, the control function 241B controls the acquisition of various information via the communication interface 210 by the acquisition function 242B, the identification of the amount of delay by the identification function 243B, the generation of medical images by the generation function 244B, the display of images such as medical images by the display control function 245, and the control of the tactile sense provision device 235 by the tactile sense provision control function 247.

[0089] The acquisition function 242B has the same functions as the acquisition function 242 in the first embodiment, and in addition acquires information (tactile information) obtained by the tactile device 131 and the tactile device drive mechanism 132 and transmitted from the medical apparatus 100B.

[0090] The identifying function 243B identifies the first delay amount and the second delay amount, and also identifies the delay amount of information obtained by the tactile device 131 and the tactile device drive mechanism 132 (hereinafter referred to as the third delay amount). In this case, the identifying function 243B transmits and receives data (e.g., test data) according to a predetermined type of the tactile device 131, and identifies the third delay amount based on the result (response time, etc.). The identifying function 243B may transmit the test data multiple times and identify the third delay amount based on the average or standard deviation of the test data. The identified third delay amount may be stored in the delay information 256 in association with the device ID and the communication conditions.

[0091] The generating function 244B generates a medical image based on the first delay amount, the second delay amount, and the third delay amount identified by the identifying function 243B. For example, the generating function 244B predicts the state of an organ after a period of time equal to the sum of the first delay amount, the second delay amount, and the third delay amount has elapsed, and acquires an image corresponding to the predicted state from past image information of multiple time phases. The generating function 244B also acquires an image related to a medical device from the most recent image information of the image information of multiple time phases, and generates a medical image by superimposing the image on an image corresponding to the predicted state. Note that, if the third delay amount is smaller than the other delay amounts by a threshold or more, the state of the organ may be predicted using one or both of the first delay amount and the second delay amount, as in the first embodiment.

[0092] The tactile sense provision control function 247 provides tactile information to the tactile sense provision device 235 based on information obtained by the tactile device 131 and the tactile device drive mechanism 132 transmitted from the medical apparatus 100B. In this case, the tactile sense provision control function 247 may generate future tactile information without delay based on the first to third delay amounts.

[0093] In addition, in the third embodiment, the display control function 245, in addition to displaying the above-mentioned image IM1 on the display 230, may also display an image including information regarding the third delay amount and information regarding the position and state of the tactile device.

[0094] Fig. 14 is a flowchart showing an example of the flow of processing executed by the medical image processing device 200B in the third embodiment. The processing in Fig. 14 is different from the processing in steps S100 to S200 shown in Fig. 8 described above in that steps S114 and S172 are added, and steps S122 and S132 are added instead of steps S120 and S130. Therefore, the following mainly describes the processing in steps S114, S122, S132, and S172. In addition, in the processing in Fig. 14, the processing in the control function 241, the acquisition function 242, the identification function 243, and the generation function 244 in the first embodiment is replaced with the processing in the control function 241B, the acquisition function 242B, the identification function 243B, and the generation function 244B.

[0095] In the process of step S110, after acquiring the cardiac cycle, the acquisition function 242B acquires tactile information (step S114). Next, the identification function 243B identifies a first delay amount, a second delay amount, and a third delay amount (step S122). Next, the generation function 244B predicts the state of the target area after the delay based on the first to third delay amounts (step S132), and acquires an image of the target area showing the predicted state from images of multiple time phases (step S140).

[0096] Furthermore, after displaying the medical image in the process of step S170, the tactile provision control function 247 generates tactile provision information corresponding to the tactile information, causes the tactile provision device 235 to output the generated information (step S172), and executes the processes from step S180 onwards. Note that the operation instructions in steps S180 and S190 in the third embodiment may include operation instructions for the tactile device.

[0097] According to the third embodiment described above, in addition to the same effects as the first embodiment, the surgeon can feel a tactile sensation in synchronization with the medical image. Therefore, for example, when performing a procedure such as septal puncture on a subject, it is possible to realize more precise remote control, such as reducing the puncture force at the moment the puncture is completed.

[0098] In the third embodiment, the haptic device 131 may be provided integrally with an arm or the like that operates the medical device. In this case, the device driving mechanism 118 includes the function of the haptic device driving mechanism 132.

[0099] The first to third embodiments may be combined with a part or all of the other embodiments. In the above-mentioned embodiments, electrocardiogram data is used as data for acquiring the cycle and state of an organ, but instead of this, heart sound data or the like may be used. In this case, the medical device 100 may be provided with a phonocardiograph instead of the electrocardiograph 119, and the electrocardiograph 119 may acquire heart sounds. In the above-mentioned embodiments, the heart is used as an example of an organ, but instead of this, lungs or blood vessels may be used, or a valve body other than a heart valve may be used. In this case, the movement cycle of the organ includes at least one of the movement cycle of the valve body, the movement cycle of the lungs, the blood flow cycle in the blood vessel, and the size of the blood vessel, instead of the movement cycle of the heart.

[0100] Any of the above-described embodiments can be expressed as follows. A storage device for storing programs; A processor, The processor executes the program, Acquire image information of multiple time phases of organs that are undergoing cyclical movement, specifying at least one of a first delay amount indicating a first delay related to a display based on the image information of the multiple time phases and a second delay amount indicating a second delay related to a processing for a predetermined object included in the image information; generating a medical image based on at least one of the first delay amount and the second delay amount and the periodic movement of the organ; The medical information processing device is configured as follows.

[0101] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope of the invention and its equivalents described in the claims, as well as in the scope and spirit of the invention. [Explanation of symbols]

[0102] 1... medical system, 100... medical device, 110... imaging device, 118... device driving mechanism, 119... electrocardiograph, 140... console device, 141, 250... memory, 142, 210... communication interface, 143, 220... input interface, 144, 230... display, 145... aperture control circuit, 146... mechanism control circuit, 150, 240... processing circuit, 200... medical information processing device, 241... control function, 242... acquisition function, 243... identification function, 244... generation function, 245... display control function, 246... judgment function, 247... tactile sensation provision control function

Claims

1. An acquisition unit that acquires image information of multiple time phases relating to an organ that performs cyclical motion from a medical device; a determination unit that determines at least one of a first delay amount indicating a first delay related to a display based on the image information of the multiple time phases and a second delay amount indicating a second delay related to a processing for a predetermined object included in the image information; a generating unit that generates a medical image based on at least one of the first delay amount and the second delay amount and a periodic movement of the organ; the first delay amount is a delay amount based on a first response time from transmitting an image request to the medical device to receiving the image information of the multiple time phases; the second delay amount is a delay amount based on a second response time from transmitting an operation request for operating a device provided in the medical apparatus to the medical apparatus until receiving information indicating that the operation corresponding to the operation request has been completed; The generation unit is predicting a state of the organ, including a position and a shape of the organ, in a case where the first delay and the second delay are not occurring, based on the first delay amount and the second delay amount; and acquiring first image information including the organ in the same state as the predicted state of the organ from the image information of the multiple time phases; acquiring second image information relating to a predetermined object from the image information of the multiple time phases; generating the medical image by superimposing the second image information on the first image information; Medical information processing equipment.

2. A display control unit that displays the medical image generated by the generation unit on a display unit, the display control unit causes the display unit to display images included in the image information of the multiple time phases and the medical image. The medical information processing device according to claim 1 .

3. the display control unit changes the color or brightness of the image to be displayed on the display unit, highlights the image, or changes the display mode so that an image with a low priority is displayed in a manner that makes it difficult for a user to recognize the image, depending on the status of the examination or treatment for the organ, the type of the organ, and the amount of deformation. The medical information processing device according to claim 2 .

4. A determination unit for determining a period during which the movement of the organ is not periodic, the generating unit generates the medical image by using image information of the plurality of time phases other than the period determined by the determining unit. The medical information processing device according to claim 1 .

5. A tactile sensation providing unit that provides tactile information from the tactile device to a user, the specifying unit specifies a third delay amount indicating a third delay related to a process of outputting information associated with the haptic information from the haptic sense providing unit; the generating unit predicts a state of the organ when a period obtained by adding the first delay amount, the second delay amount, and the third delay amount has elapsed, acquires the first image information including the organ in the same state as the predicted state of the organ from the image information of the multiple time phases, and generates the medical image by superimposing the second image information on the acquired first image information. The medical image processing device according to claim 1 .

6. The predetermined object is an object that does not perform a periodic motion. The medical image processing device according to claim 1 .

7. The predetermined object includes an object operated by a user. The medical information processing device according to claim 6 .

8. The period of the organ movement includes at least one of a period of the heart movement, a period of the valve movement, a period of the lung movement, a period of the blood flow in a blood vessel, and a period of the size of the blood vessel. The medical image processing device according to claim 1 .

9. A medical system including a medical device for performing a predetermined process on an organ that periodically moves, and a medical information processing device for transmitting and receiving information to and from the medical device, The medical device comprises: A drive unit that drives a predetermined object; an image processing unit that generates image information of multiple time phases including the organ and the object; The medical information processing device includes: an acquisition unit that acquires the image information of the multiple time phases from the medical device; a determination unit that determines at least one of a first delay amount indicating a first delay related to a display based on the image information of the multiple time phases and a second delay amount indicating a second delay related to a processing for the predetermined object included in the image information; a generating unit that generates a medical image based on at least one of the first delay amount and the second delay amount and a periodic movement of the organ; the first delay amount is a delay amount based on a first response time from transmitting an image request to the medical device to receiving the image information of the multiple time phases; the second delay amount is a delay amount based on a second response time from transmitting an operation request for operating a device provided in the medical apparatus to the medical apparatus until receiving information indicating that the operation corresponding to the operation request has been completed; The generation unit is predicting a state of the organ, including a position and a shape of the organ, in a case where the first delay and the second delay are not occurring, based on the first delay amount and the second delay amount; and acquiring first image information including the organ in the same state as the predicted state of the organ from the image information of the multiple time phases; acquiring second image information relating to a predetermined object from the image information of the multiple time phases; generating the medical image by superimposing the second image information on the first image information; Medical systems.

10. A computer of a medical information processing device, Acquiring image information of multiple time phases of an organ that is periodically moving from a medical device; identifying at least one of a first delay amount indicating a first delay related to a display based on the image information of the multiple time phases and a second delay amount indicating a second delay related to a processing of a predetermined object included in the image information; generating a medical image based on at least one of the first delay amount and the second delay amount and a periodic movement of the organ; the first delay amount is a delay amount based on a first response time from transmitting an image request to the medical device to receiving the image information of the multiple time phases; the second delay amount is a delay amount based on a second response time from transmitting an operation request for operating a device provided in the medical apparatus to the medical apparatus until receiving information indicating that the operation corresponding to the operation request has been completed; predicting a state of the organ, including a position and a shape of the organ, in a case where the first delay and the second delay are not occurring, based on the first delay amount and the second delay amount; and acquiring first image information including the organ in the same state as the predicted state of the organ from the image information of the multiple time phases; acquiring second image information relating to a predetermined object from the image information of the multiple time phases; generating the medical image by superimposing the second image information on the first image information; Medical information processing method.

11. The computer of the medical information processing device Acquiring image information of multiple time phases of an organ that is periodically moving from a medical device; specifying at least one of a first delay amount indicating a first delay related to a display based on the image information of the multiple time phases and a second delay amount indicating a second delay related to a processing of a predetermined object included in the image information; generating a medical image based on at least one of the first delay amount and the second delay amount and a periodic movement of the organ; the first delay amount is a delay amount based on a first response time from transmitting an image request to the medical device to receiving the image information of the multiple time phases; the second delay amount is a delay amount based on a second response time from transmitting an operation request for operating a device provided in the medical apparatus to the medical apparatus until receiving information indicating that the operation corresponding to the operation request has been completed; predicting a state of the organ, including a position and a shape of the organ, in a case where the first delay and the second delay are not occurring, based on the first delay amount and the second delay amount; acquiring first image information including an organ in the same state as the predicted state of the organ from the image information of the multiple time phases; acquiring second image information relating to a predetermined object from the image information of the plurality of time phases; generating the medical image by superimposing the second image information on the first image information; program.

Citation Information

Patent Citations

  • Method of constructing fire proof wall surface of fire proof ceiling surface

    JP1978067215A

  • Surgery support device

    JP2007185278A

  • System and method for planning multiple interventional procedures

    JP2017522975A

  • Remote control communication system and relay method for the same, and program

    JP2018126851A

  • Surgical operation assistant device and surgical navigation system

    JP2020036708A