Information processing apparatus, information processing system, information processing method, and program

Abstract

To synchronize the temporal phases of the state of a subject between dynamic information, such as dynamic image data obtained by dynamic imaging using radiation, and ultrasonic information, such as ultrasonic image data obtained by ultrasonic imaging.SOLUTION: An information processing apparatus comprises a control unit. The control unit synchronizes and displays dynamic information and ultrasonic information such that the temporal phases of the state of the subject coincide. The dynamic information is a dynamic image 641, which is dynamic image data indicating a change in the subject obtained by dynamic imaging using radiation. The ultrasonic information is an ultrasonic image 642, which is ultrasonic image data of a moving image indicating a morphological change of the subject obtained by ultrasonic imaging of the moving image.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing system, an information processing method, and a program. [Background technology]

[0002] Dynamic radiography, which generates dynamic image data, which is radiographic image data of a moving image of a subject, is known as a conventional radiography method. In dynamic radiography, information on morphological changes of the subject can be obtained. Morphology refers to information on the position, shape, properties, and existence of the subject's organs, tissues, etc., which can be recognized from still or moving images. Morphological changes refer to changes in morphology over time. Functional information can also be obtained from information on morphological changes. Radiographic devices that perform dynamic radiography are known.

[0003] Also known is an X-ray diagnosis support device that displays an integrated X-ray moving image obtained by X-ray moving imaging and an X-ray still image obtained by CT (Computed Tomograph) imaging (see Patent Document 1). The X-ray still image is a CT image of a slice plane of a subject.

[0004] Also known is an ultrasound diagnostic device that applies an ultrasound probe to a subject, transmits and receives ultrasound waves, and generates ultrasound image data such as a tomographic image of the subject. The ultrasound image data may be a still image or a moving image. A user such as a doctor or technician can apply the ultrasound probe to any position on the subject's body surface. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 4797173 Summary of the Invention [Problem to be solved by the invention]

[0006] However, with the conventional radiography devices, information on morphological changes during dynamic imaging and functional information can only be obtained from the morphological changes seen from the X-ray irradiation direction, meaning that morphological information on the subject in directions other than the X-ray irradiation direction cannot be obtained.

[0007] Furthermore, while the conventional X-ray diagnosis support system described above can determine morphological changes in the subject from X-ray video images, it can only grasp the shape of a single fixed slice plane from CT images.

[0008] For this reason, a configuration in which dynamic image data and moving ultrasound image data are displayed side by side is conceivable. However, because the dynamic image data and moving ultrasound image data are obtained by separate imaging, the time phases of the subject's condition do not match. Therefore, there is a need to match the time phases of the subject's condition. For example, in the case of pulmonary embolism, dynamic analysis of dynamic image data can obtain functional information indicating that blood flow has stopped. There is a need to confirm that the right ventricle is expanding using moving ultrasound image data in synchronization with this functional information.

[0009] An object of the present invention is to align the time phases of the subject's condition between dynamic information such as dynamic image data obtained by dynamic radiography and ultrasound information such as ultrasound image data obtained by ultrasound photography. [Means for solving the problem]

[0010] In order to solve the above problem, the information processing device of the invention described in claim 1 comprises: The system is equipped with a control unit that synchronizes and displays dynamic information showing changes in the subject due to dynamic radiation imaging and ultrasound information showing morphological changes in the subject due to moving ultrasound imaging so that the time phases of the subject's condition are aligned.

[0011] The invention described in claim 2 is the information processing device described in claim 1, The dynamic information is at least one of dynamic image data and functional information analyzed from the dynamic image data, The ultrasound information is moving ultrasound image data.

[0012] The invention described in claim 3 is the information processing device described in claim 1, The control unit displays the dynamic information and the ultrasound information in synchronization based on synchronization information for synchronizing the dynamic information corresponding to the time of a predetermined wave of the first electrocardiogram data measured simultaneously with the dynamic imaging and the ultrasound information corresponding to the time of the predetermined wave of the second electrocardiogram data measured simultaneously with the ultrasound imaging.

[0013] The invention described in claim 4 is the information processing device described in claim 3, The control unit generates the synchronization information that matches the dynamic information corresponding to the time of a predetermined wave of the first electrocardiogram data with the ultrasound information corresponding to the time of a predetermined wave of the second electrocardiogram data.

[0014] The invention described in claim 5 is the information processing device described in claim 4, The control unit generates the synchronization information in response to an operation input that matches the dynamic information corresponding to the time of a predetermined wave of the first electrocardiogram data with the ultrasound information corresponding to the time of a predetermined wave of the second electrocardiogram data.

[0015] The invention described in claim 6 is the information processing device described in claim 1, The control unit uses a first trained model that extracts dynamic information corresponding to the time of a predetermined wave of an electrocardiogram from the dynamic information, extracts dynamic information corresponding to the time of the predetermined wave from the dynamic information, uses a second trained model that extracts ultrasound information corresponding to the time of a predetermined wave of an electrocardiogram from the ultrasound information, extracts ultrasound information corresponding to the time of the predetermined wave from the ultrasound information, generates synchronization information for synchronizing the extracted dynamic information and ultrasound information, and displays the dynamic information and the ultrasound information in synchronization based on the synchronization information.

[0016] The information processing system of the invention described in claim 7 comprises: an information processing device according to claim 3; a dynamic imaging control device that controls the dynamic imaging; an ultrasonic diagnostic device that performs the ultrasonic imaging; the synchronization information includes first synchronization information indicating the dynamic state information corresponding to a time of a predetermined wave of the first electrocardiogram data, and second synchronization information indicating the ultrasound information corresponding to a time of a predetermined wave of the second electrocardiogram data, the dynamic imaging control device generates the first synchronization information; The ultrasonic diagnostic apparatus generates the second synchronization information.

[0017] The invention described in claim 8 is the information processing system described in claim 7, the dynamic imaging control device generates the first synchronization information in response to an operation input that associates a time of a predetermined wave of the first electrocardiogram data with the dynamic information; The ultrasonic diagnostic apparatus generates the second synchronization information in response to an operation input that associates the time of a predetermined wave of the second electrocardiogram data with the ultrasound information.

[0018] The information processing system of the invention described in claim 9 comprises: The information processing device according to claim 1; a dynamic imaging control device that controls the dynamic imaging; an ultrasonic diagnostic device that performs the ultrasonic imaging; the dynamic imaging control device extracts dynamic information corresponding to the time of a predetermined wave from the dynamic information using a first trained model that extracts dynamic information corresponding to the time of the predetermined wave from the dynamic information, and generates first synchronization information indicating the extracted dynamic information; the ultrasound diagnostic device extracts ultrasound information corresponding to the time of a predetermined wave from the ultrasound information using a second trained model that extracts ultrasound information corresponding to the time of the predetermined wave from the ultrasound information, and generates second synchronization information indicating the extracted ultrasound information; The control unit displays the dynamic information and the ultrasound information in synchronization based on synchronization information including the first synchronization information and the second synchronization information for synchronizing the extracted dynamic information and ultrasound information.

[0019] The information processing method of the invention described in claim 10 comprises: The method includes a control process for synchronizing and displaying dynamic information showing changes in the subject obtained by dynamic radiation imaging and ultrasound information showing morphological changes in the subject obtained by moving ultrasound imaging so that the time phases of the subject's condition are matched.

[0020] The program of the invention described in claim 11 is Computer, a control unit that synchronizes and displays dynamic information showing changes in the subject obtained by dynamic radiography and ultrasound information showing morphological changes in the subject obtained by ultrasonic imaging of a moving image, so that the time phases of the state of the subject are matched; Function as. [Effects of the Invention]

[0021] According to the present invention, the time phase of the subject's condition between dynamic information obtained by dynamic radiation imaging and ultrasonic information obtained by ultrasonic imaging can be matched. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a block diagram showing an image capturing system according to a first embodiment of the present invention; [Figure 2] FIG. 2 is a block diagram showing the functional configuration of the ultrasound diagnostic apparatus. [Figure 3] FIG. 2 is a block diagram showing the functional configuration of the radiation imaging apparatus. [Figure 4] FIG. 2 is a block diagram showing the functional configuration of the console. [Figure 5] 1 is a block diagram showing a functional configuration of an electrocardiogram measurement device. [Figure 6] FIG. 2 is a block diagram showing the functional configuration of the image management device. [Figure 7] FIG. 1 is a diagram showing an example of an electrocardiogram. [Figure 8] 10 is a flowchart showing a dynamic image generation process. [Figure 9] 10 is a flowchart showing an ultrasound image generation process. [Figure 10] 10 is a flowchart showing a first image display process. [Figure 11] FIG. 2 is a diagram showing an image display screen. [Figure 12] 10 is a flowchart showing a second image display process. DETAILED DESCRIPTION OF THE INVENTION

[0023] The advantages and features provided by one or more embodiments of the present invention will be more fully understood from the following detailed description and the accompanying drawings. However, these drawings are for illustrative purposes only and are not intended to define the limits of the present invention. Below, first and second embodiments and modifications of the present invention will be described with reference to the drawings. However, the scope of the present invention is not limited to the disclosed embodiments.

[0024] (First embodiment) A first embodiment of the present invention will be described with reference to Figs. 1 to 11. First, the device configuration of this embodiment will be described with reference to Figs. 1 to 7. Fig. 1 is a block diagram showing an image capturing system 1000 of this embodiment. Fig. 2 is a block diagram showing the functional configuration of an ultrasound diagnostic device 100. Fig. 3 is a block diagram showing the functional configuration of a radiation imaging device 200. Fig. 4 is a block diagram showing the functional configuration of a console 300. Fig. 5 is a block diagram showing the functional configuration of an electrocardiogram measurement device 500. Fig. 6 is a block diagram showing the functional configuration of an image management device 600. Fig. 7 is a diagram showing an example of an electrocardiogram.

[0025] As shown in Fig. 1, an imaging system 1000 according to this embodiment is installed in a medical facility such as a hospital. The imaging system 1000 includes an ultrasound diagnostic device 100, a dynamic image generating device 2000, an electrocardiogram measuring device 500, and an image management device 600 as an information processing device. The dynamic image generating device 2000 includes a radiation imaging device 200, a console 300 as a dynamic imaging control device, and a dynamic analysis device 400. The devices in the imaging system 1000 are communicably connected via a communication network 3000. The image management device 600 functions as an

[0026] It is also possible to provide a configuration in which an HIS (Hospital Information System) / RIS (Radiology Information System) is provided on the communication network 3000. The HIS / RIS is a device that transmits order information to the console 300. The order information includes various information such as patient information, imaging type, imaging region, imaging direction, medical department, and imaging purpose. The patient information includes, for example, patient ID, patient name, sex, age, and whether or not the patient has a disease. The imaging type includes, for example, dynamic imaging and still images.

[0027] Each device constituting the imaging system 1000 conforms to a standard, and communication between the devices is performed in accordance with DICOM (Digital Image and Communications in Medicine). The communication network 3000 may be a LAN (Local Area Network), etc. However, the communication network 3000 may also be a WAN (Wide Area Network), the Internet, etc. The communication method of the communication network 3000 may be wired communication or wireless communication.

[0028] The ultrasound diagnostic device 100 is a diagnostic device that transmits and receives ultrasound waves to and from a subject, such as a patient's living body, and generates and displays ultrasound image data that visualizes the interior of the subject. As shown in FIG. 2, the ultrasound diagnostic device 100 includes an ultrasound diagnostic device main body 1 and an ultrasound probe 2. The ultrasound probe 2 is connected to the ultrasound diagnostic device main body 1. The ultrasound probe 2 transmits ultrasound waves (transmitted ultrasound waves) into the subject and receives reflected waves of the ultrasound waves (reflected ultrasound waves: echoes) reflected within the subject. The ultrasound probe 2 includes an ultrasound probe main body 21, a cable 22, and a connector 23. The ultrasound probe main body 21 is the head of the ultrasound probe 2 and transmits and receives ultrasound waves. The cable 22 is connected to the ultrasound probe main body 21 and the connector 23. The cable 22 is a cable through which a drive signal for the ultrasound probe main body 21 and a received ultrasound signal flow. The connector 23 is a plug connector for connecting to a receptacle connector (not shown) of the ultrasound diagnostic device main body 1.

[0029] The ultrasound diagnostic device main body 1 is connected to the ultrasound probe main body 21 via a connector 23 and a cable 22. The ultrasound diagnostic device main body 1 transmits an electrical drive signal to the ultrasound probe main body 21, causing the ultrasound probe main body 21 to transmit ultrasound waves to the subject. The ultrasound probe 2 generates a reception signal, which is an electrical signal, in response to the ultrasound reflected from inside the subject and received by the ultrasound probe main body 21. The ultrasound diagnostic device main body 1 creates an image of the internal state of the subject as ultrasound image data based on the reception signal generated by the ultrasound probe 2.

[0030] The ultrasound probe main body 21 has a transducer 211 on the tip side. In this embodiment, a convex scanning electronic scanning probe is adopted as the ultrasound probe 2. However, the ultrasound probe 2 may be either an electronic scanning type or a mechanical scanning type. The ultrasound probe 2 may also be any of a linear scanning type, a sector scanning type, or a convex scanning type. The ultrasound diagnostic device main body 1 and the ultrasound probe 2 may be configured to communicate wirelessly instead of by wire via a cable 22. This wireless communication may be UWB (Ultra Wide Band) or the like.

[0031] The ultrasound diagnostic device main body 1 includes an operation unit 11, a transmission unit 12, a reception unit 13, a signal processing unit 14, an image processing unit 15, a display control unit 16, a display unit 17, a control unit 18, a storage unit 19, and a communication unit 31. The image processing unit 15 includes an image memory unit 151.

[0032] The operation unit 11 is a control panel or the like that accepts various operation inputs from users such as doctors and technicians. The operation unit 11 has operation elements such as push buttons, encoders, lever switches, joysticks, trackballs, keyboards, touchpads, and multifunction switches. The operation unit 11 accepts various operation inputs from the user and outputs the operation signals to the control unit 18. The operation unit 11 may be formed integrally with the display screen of the display unit 17 and may include a touch panel that accepts touch inputs from the user.

[0033] The transmitter 12, under the control of the controller 18, supplies a drive signal, which is an electrical signal, to the ultrasonic probe 2, causing the ultrasonic probe 2 to generate transmitted ultrasonic waves. The transmitter 12 generates transmitted ultrasonic waves by, for example, driving a continuous portion (e.g., 64) of a plurality of transducers (e.g., 192) arranged in the ultrasonic probe 2. Then, the transmitter 12 performs scanning by shifting the driven transducers in the scanning direction each time a transmitted ultrasonic wave is generated.

[0034] The receiving unit 13 receives the received signal, which is an analog electrical signal received from the ultrasound probe 2, and amplifies and AD (Analog to Digital) converts it in accordance with the control of the control unit 18. The receiving unit 13 adjusts the time phase of the digital received signal after AD conversion by providing a delay time for each individual path corresponding to each transducer, and then adds the signals (phased addition) to generate sound ray data.

[0035] The signal processing unit 14 performs processes such as envelope detection and logarithmic compression on the sound ray data from the receiving unit 13 under the control of the control unit 18. The signal processing unit 14 adjusts the dynamic range and gain of the sound ray data after processing such as logarithmic compression to convert the brightness, and generates B-mode image data. In other words, B-mode image data represents the strength of the received signal by brightness. However, the signal processing unit 14 may also be configured to generate ultrasound image data in an image mode other than B-mode, such as color Doppler image data.

[0036] The image memory unit 151 is configured with a semiconductor memory such as a DRAM (Dynamic Random Access Memory). The image processing unit 15 stores the ultrasound image data (B-mode image data) input from the signal processing unit 14 in units of frames in the image memory unit 151 under the control of the control unit 18. The image processing unit 15 outputs the ultrasound image data stored in the image memory unit 151 to the display control unit 16 for one frame at a time at predetermined intervals.

[0037] The display control unit 16 is, for example, a DSC (Digital Scan Converter). Under the control of the control unit 18, the display control unit 16 performs processing such as coordinate conversion on the ultrasound image data input from the image processing unit 15 to convert the data into an image signal for display.

[0038] The display unit 17 has a display panel such as an LCD (Liquid Crystal Display) or an EL (Electro-Luminescence) display. The display unit 17 displays an ultrasound image on the display panel in accordance with an image signal output from the display control unit 16 under the control of the control unit 18. The display unit 17 also displays various display information input from the control unit 18 on the display panel.

[0039] The control unit 18 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The control unit 18 reads various processing programs stored in the ROM, loads them into the RAM, and controls each unit of the ultrasound diagnostic device 100 in cooperation with the CPU and the loaded programs. The ROM is composed of non-volatile memory such as a semiconductor. The ROM stores a system program corresponding to the ultrasound diagnostic device 100, various processing programs executable on the system program, and various data such as a gamma table. In particular, the ROM stores an ultrasound image generation program for executing the ultrasound image generation process described below. These programs are stored in the RAM in the form of computer-readable program codes. The CPU sequentially executes operations in accordance with the program codes in the RAM. The RAM forms a work area for temporarily storing various programs executed by the CPU and data related to these programs.

[0040] The storage unit 19 is a storage unit such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores information such as ultrasound image data in a writable and readable manner.

[0041] The communication unit 31 includes a network card for wired communication that is communicatively connected to the communication network 3000, a wireless communication module for wireless communication, etc. The control unit 18 transmits and receives information to and from external devices such as the electrocardiogram measurement device 500 and the image management device 600 on the communication network 3000 via the communication unit 31.

[0042] The dynamic image generating device 2000 performs dynamic imaging of a subject. Dynamic imaging is a method of repeatedly irradiating a subject with pulsed radiation such as X-rays at predetermined time intervals (pulse irradiation) to obtain multiple radiation image data showing the dynamic state of the subject. The series of multiple radiation image data is called dynamic image data. Each of the multiple images constituting the dynamic image data is called a frame image. Dynamic imaging can also be performed by continuously irradiating the subject with radiation at a low dose rate without interruption (continuous irradiation). In this embodiment, a configuration for performing dynamic imaging of a subject using pulsed radiation irradiation will be described as an example.

[0043] The radiation imaging apparatus 200 is installed, for example, in a general imaging room of a medical facility. As shown in FIG.

[0044] The radiation source 201 is disposed at a position facing the FPD 203 across the subject 0. The radiation source 201 irradiates the subject 0 with radiation (X-rays) under the control of the radiation irradiation controller 202. The radiation irradiation controller 202 is connected to the console 300 and controls the radiation source 201 to perform radiography based on the radiation irradiation conditions input from the console 300. The radiation irradiation conditions input from the console 300 include the pulse rate, pulse width, pulse interval, the number of frames per imaging, the X-ray tube current value, the X-ray tube voltage value, and the type of additional filter. The pulse rate is the number of radiation irradiations per second and corresponds to the frame rate of the dynamic image data. The pulse width is the radiation irradiation time per radiation irradiation. The pulse interval is the time from the start of one radiation irradiation to the start of the next radiation irradiation and corresponds to the frame interval of the dynamic image data.

[0045] The FPD 203 is disposed opposite the radiation source 201 across the subject 0. The FPD 203 includes a glass substrate and the like. The FPD 203 has detecting elements (pixels) arranged in a matrix at predetermined positions on the glass substrate. The detecting elements detect radiation emitted from the radiation source 201 and transmitted through the subject 0 according to the intensity of the radiation. The detecting elements convert the detected radiation into an electrical signal and accumulate the electrical signal. Each detecting element includes a switching unit such as a TFT (Thin Film Transistor). The FPD 203 controls the switching unit of each pixel based on image reading conditions input from the console 300, switching the reading of the electrical signal accumulated in each pixel. The FPD 203 generates one frame of radiation image data by reading the electrical signal accumulated in each pixel. This radiation image data is frame image data.

[0046] The FPD 203 outputs the generated frame images to the console 300. Image reading conditions include the frame rate, frame interval, pixel size, and image size (matrix size). The frame rate is the number of image frames acquired per second and corresponds to the pulse rate. The frame interval is the time from the start of acquisition of one frame image to the start of acquisition of the next frame image and corresponds to the pulse interval. The radiation irradiation control device 202 and FPD 203 are connected to each other via wireless communication and exchange synchronization signals with each other to synchronize the radiation irradiation operation and the image reading operation.

[0047] The console 300 is installed, for example, in an operation room near a general radiography room. The console 300 outputs radiation irradiation conditions and image reading conditions to the radiography apparatus 200, and controls the radiography and radiographic image reading operations of the radiography apparatus 200. Furthermore, the console 300 displays dynamic image data acquired by the radiography apparatus 200 for confirmation by users such as doctors and technicians.

[0048] 4, the console 300 includes a control unit 301, an operation unit 302, a storage unit 303, a display unit 304, and a communication unit 305. The units of the console 300 are connected to each other via a bus.

[0049] The control unit 301 includes a CPU, RAM, etc. In response to an operation on the operation unit 302, the CPU reads out a system program and various processing programs stored in the storage unit 303, loads them into the RAM, and executes various processes. The control unit 301 controls the operation of each unit of the console 300 and the radiation irradiation operation and reading operation of the radiation imaging apparatus 200 in accordance with the loaded programs.

[0050] The operation unit 302 includes a keyboard having cursor keys, numeric input keys, and various function keys, and a pointing device such as a mouse. The operation unit 302 outputs operation information of input key operations and mouse operations to the control unit 301. The operation unit 302 may also include a touch panel that is integrated with the display panel of the display unit 304. The operation unit 302 also includes an exposure switch that accepts input of dynamic imaging instructions to the radiation irradiation control device 202.

[0051] The storage unit 303 is configured with an HDD or SSD. The storage unit 303 stores various programs executed by the control unit 301, parameters required for executing processes by the programs, data such as processing results, etc. The various programs are stored in the form of readable program code. The control unit 301 sequentially executes operations in accordance with the program code. In particular, the storage unit 303 stores a dynamic image generation program for executing a dynamic image generation process described below. The storage unit 303 also stores radiation irradiation conditions and image reading conditions corresponding to the imaging region. Furthermore, the storage unit 303 may store order information transmitted from the HIS / RIS.

[0052] The display unit 304 has a display panel such as an LCD, an EL display, etc. The display unit 304 displays input instructions from the operation unit 302, radiation image data, etc. in accordance with instructions of a display signal input from the control unit 301.

[0053] The communication unit 305 includes a network card for wired communication and a wireless communication module for wireless communication, which are communicatively connected to the communication network 3000. The control unit 301 transmits and receives information to and from external devices on the communication network 3000, such as the radiation imaging device 200, the electrocardiogram measurement device 500, and the image management device 600, via the communication unit 305.

[0054] The dynamic analysis device 400 is an information processing device that performs dynamic analysis on dynamic image data received from the console 300, generates functional information as the analysis result, and transmits it to the console 300. The functional information is, for example, information for visualizing the blood flow of the subject (for example, a colored display of the pulsation of the blood vessels). However, the functional information is not limited to this, and may be other dynamic analysis information, such as information for visualizing the respiratory system of the subject.

[0055] The electrocardiogram measurement device 500 is a device that measures electrical changes in the myocardium of a subject and generates electrocardiogram data. As shown in Fig. 5, the electrocardiogram measurement device 500 includes a control unit 501, an operation unit 502, a storage unit 503, a display unit 504, a communication unit 505, and an electrocardiogram measurement unit 506. The various units of the electrocardiogram measurement device 500 are connected via a bus.

[0056] The control unit 501 includes a CPU, RAM, etc. In response to an operation on the operation unit 502, the CPU reads out a system program and various processing programs stored in the storage unit 503 and loads them into the RAM. The CPU executes various processes in accordance with the loaded programs. For example, the control unit 501 measures an electrocardiogram using the electrocardiogram measurement unit 506 to generate electrocardiogram data. The control unit 501 transmits the electrocardiogram data to the ultrasound diagnostic device 100, the console 300, etc. via the communication unit 505.

[0057] The operation unit 502 includes various function keys and a pointing device such as a mouse. The operation unit 502 outputs instruction signals input by a user operating the keys or the mouse to the control unit 501. The operation unit 502 may also include a touch panel formed integrally with the display panel of the display unit 504.

[0058] The storage unit 503 is configured by an HDD, an SSD, etc. The storage unit 503 stores system programs, various programs, data such as parameters required for executing processes by the programs, measured electrocardiogram data, and the like.

[0059] The display unit 504 has a display panel such as an LCD, an EL display, etc. The display unit 504 displays input instructions from the operation unit 502 and various display information in accordance with instructions input from the control unit 501.

[0060] The communication unit 505 includes a network card for wired communication that is communicatively connected to the communication network 3000, and a wireless communication module for wireless communication. The control unit 501 transmits and receives information to and from external devices such as the console 300 and the image management device 600 on the communication network 3000 via the communication unit 505.

[0061] The electrocardiogram measurement unit 506 measures electrical changes in the myocardium via electrodes placed on the body surface of the subject and generates electrocardiogram data. As the electrocardiogram measurement unit 506, for example, a commonly used 12-lead electrocardiograph can be used, but an XYZ-lead electrocardiograph may also be used.

[0062] As shown in FIG. 7, an example of an electrocardiogram waveform will be described, with the horizontal axis representing time and the vertical axis representing the potential difference between the electrodes of the electrocardiogram measurement unit 506. The electrocardiogram waveform consists of P (Primary) waves, Q waves, R waves, S waves, T waves, and U waves, which occur in sequence over time. The P wave is an upward wave generated by excitation of the right and left atria of the subject's heart. The Q wave, R wave, and S wave are generated by excitation of the ventricles. Of these, the Q wave is the first downward wave to appear. The R wave is an upward wave with a large amplitude. The S wave is a downward wave. The T wave is an upward wave with a gentle slope generated when the excitation of the ventricles cools down. The U wave is a small upward wave that follows the T wave.

[0063] In this embodiment, the timing for synchronizing frames of simultaneously displayed moving image ultrasound image data and dynamic image data is set to the timing when the subject's state is an R wave of an electrocardiogram.

[0064] The image management device 600 is a PACS (Picture Archiving and Communication System) that manages ultrasound image data, dynamic image data, electrocardiogram data, and the like generated by the ultrasound diagnostic device 100, dynamic image generating device 2000, and electrocardiogram measuring device 500.

[0065] 6, the image management device 600 includes a control unit 601, an operation unit 602, a storage unit 603, a display unit 604, and a communication unit 605. The units of the image management device 600 are connected via a bus.

[0066] The control unit 601 includes a CPU, RAM, ROM, etc. The ROM stores various programs executed by the CPU and parameters required for executing the programs. In particular, the ROM stores a first image display program for executing a first image display process described below. The CPU reads out the various programs stored in the ROM, expands them in the RAM, and executes various processes in accordance with the expanded programs.

[0067] The operation unit 602 includes a keyboard equipped with cursor keys, numeric input keys, various function keys, etc., and a pointing device such as a mouse. The operation unit 602 outputs operation information of input key operations and mouse operations to the control unit 601. The operation unit 602 may also include a touch panel that is integrally provided on the display panel of the display unit 604.

[0068] The storage unit 603 is configured with an HDD, SSD, etc. The storage unit 603 stores data such as ultrasound image data, dynamic image data, and electrocardiogram data generated by the ultrasound diagnostic device 100, the dynamic image generating device 2000, and the electrocardiogram measuring device 500.

[0069] The display unit 604 has a display panel such as an LCD, an EL display, etc. The display unit 604 displays input instructions from the operation unit 602 and various display information in accordance with instructions input from the control unit 601.

[0070] The communication unit 605 includes a network card for wired communication that is communicatively connected to the communication network 3000, and a wireless communication unit for wireless communication. The control unit 601 transmits and receives information to and from external devices such as the radiation imaging apparatus 200 and the electrocardiogram measurement apparatus 500 on the communication network 3000 via the communication unit 605.

[0071] Next, the operation of the imaging system 1000 of this embodiment will be described with reference to Figs. 8 to 11. Fig. 8 is a flowchart showing dynamic image generation processing. Fig. 9 is a flowchart showing ultrasound image generation processing. Fig. 10 is a flowchart showing first image display processing. Fig. 11 is a diagram showing an image display screen 640.

[0072] In the imaging system 1000, a dynamic image generation process is executed by the dynamic image generation device 2000. Then, an ultrasound image generation process is executed by the ultrasound diagnostic device 100. The order of execution of the dynamic image generation process and the ultrasound image generation process may be reversed. The dynamic image generation process is a process of radiographing a subject to generate and store dynamic image data together with synchronization information. The ultrasound image generation process is a process of ultrasoundgraphing a subject to generate and store moving ultrasound image data together with synchronization information.

[0073] The dynamic image generation process and the ultrasound image generation process are not executed simultaneously. This is to prevent the examiner (user) of the ultrasound imaging from being exposed to radiation during the dynamic image generation process. Furthermore, this is to prevent the ultrasound probe 2 from appearing in the dynamic image when the dynamic image generation process is being executed. For this reason, there are no frames captured at the exact same time in the dynamic image data and the moving ultrasound image data. In this embodiment, synchronization refers to the synchronization of the display times of frames in which the same predetermined state of the subject's dynamics occurs in the dynamic image data and the moving ultrasound image data. The predetermined state of the subject's dynamics is a state in which an R wave is generated in the heart in the electrocardiogram data. As an example, it is assumed that the subject's heart is being radiographed.

[0074] The dynamic image generation process executed by the dynamic image generation device 2000 will be described with reference to Fig. 8. A living patient, or subject, is placed on an imaging table in a general imaging room of the radiation imaging device 200 in advance. Electrodes of the electrocardiogram measurement unit 506 of the electrocardiogram measurement device 500 are attached to the subject. A user, such as a doctor or technician performing radiation imaging, is on standby to use the console 300. Furthermore, the control unit 301 receives input of radiation irradiation conditions and image reading conditions for dynamic images from the user via the operation unit 302.

[0075] In the console 300, a user inputs an instruction to execute dynamic image generation processing via the operation unit 302. This execution instruction triggers the control unit 301 to execute the dynamic image generation processing.

[0076] The control unit 301 causes the electrocardiogram measurement device 500 to generate electrocardiogram data (step S11). In step S11, the control unit 501 measures the subject using the electrocardiogram measurement unit 506 to generate electrocardiogram data. When executing step S11 for the first time, the control unit 301 transmits an instruction to generate electrocardiogram data to the electrocardiogram measurement device 500 via the communication unit 305. The control unit 501 of the electrocardiogram measurement device 500 receives a request for electrocardiogram data from the console 300 via the communication unit 505. The electrocardiogram data has, for example, the magnitude (potential difference) of the electrocardiogram waveform for each measurement time information.

[0077] In parallel with step S11, the control unit 301 transmits radiation irradiation conditions to the radiation irradiation control device 202 and transmits image reading conditions to the FPD 203 via the communication unit 305 (step S12). In step S12, the control unit 301 causes the radiation irradiation control device 202 and the FPD 203 to perform radiation imaging of the subject and causes the FPD 203 to generate dynamic image data. In the dynamic image data, for example, imaging time information is associated with each frame of the dynamic image.

[0078] The control unit 301 determines whether to end the radiography, depending on whether or not a command to end the radiography is input from the user via the operation unit 302 (step S13). If the radiography is not to be ended (step S13; NO), the process proceeds to step S11. If the radiography is to be ended (step S13; YES), the control unit 301 receives the generated electrocardiogram data from the electrocardiogram measurement device 500 via the communication unit 305 (step S14). In response to step S14, the control unit 501 transmits the measured electrocardiogram data to the console 300 via the communication unit 505. In step S14, the control unit 301 receives the generated dynamic image data from the FPD 203 via the communication unit 305. In response to step S14, the FPD 203 transmits the generated dynamic image data to the console 300.

[0079] The control unit 301 generates synchronization information indicating a frame corresponding to a predetermined wave (R wave) of the electrocardiogram from the electrocardiogram data and dynamic image data received in step S14 (step S15). In step S15, for example, the control unit 301 analyzes the electrocardiogram data to obtain time information of the R wave and identifies a frame of the dynamic image data corresponding to the obtained time information. The control unit 301 adds a mark to the frame identified as the synchronization information. Note that the synchronization information is not limited to a mark of a frame corresponding to the R wave. For example, the control unit 301 may be configured to add frame information corresponding to the R wave as synchronization information to DICOM information as tag information of the dynamic image data. Alternatively, the control unit 301 may be configured to draw an electrocardiogram waveform of the electrocardiogram data corresponding to the time information on a radiographic image of each frame of the dynamic image data.

[0080] The control unit 301 transmits the dynamic image data, electrocardiogram data, and synchronization information of step S14, and the synchronization information of step S15, to the image management device 600 via the communication unit 305, and stores them (step S16). In response to step S16, the control unit 601 of the image management device 600 receives the dynamic image data, electrocardiogram data, and synchronization information from the console 300 via the communication unit 605. The control unit 601 stores the received dynamic image data, electrocardiogram data, and synchronization information in the storage unit 603. The dynamic image generation process then ends.

[0081] Referring to FIG. 9, the ultrasound image generation process executed by the ultrasound diagnostic device 100 will be described. A subject, a living patient after radiography, is placed on an examination table in a diagnostic room where the ultrasound diagnostic device 100 is installed. The electrodes of the electrocardiogram measurement unit 506 of the electrocardiogram measurement device 500 are attached to the subject. A user, such as a doctor or technician, is ready to use the ultrasound diagnostic device 100. The control unit 18 also receives input of setting information, such as various parameters for ultrasound imaging, from the user via the operation unit 11. The ultrasound imaging involves scanning the subject with ultrasound to generate, for example, B-mode image data as ultrasound image data. The user then presses the ultrasound probe 2 against the subject.

[0082] In the ultrasonic diagnostic apparatus 100, an instruction to execute ultrasonic image generation processing is input from the user via the operation unit 11. This execution instruction serves as a trigger for the control unit 18 to execute the ultrasonic image generation processing.

[0083] The control unit 18 causes the electrocardiogram measurement device 500 to generate electrocardiogram data (step S21). In step S21, the control unit 501 measures the subject using the electrocardiogram measurement unit 506 to generate electrocardiogram data. When step S21 is executed for the first time, the control unit 18 transmits an instruction to generate electrocardiogram data to the electrocardiogram measurement device 500 via the communication unit 31.

[0084] In parallel with step S21, the control unit 18 scans the subject and generates moving ultrasound image data (step S22) under the control of the transmission unit 12 to the image processing unit 15. In the moving ultrasound image data, for example, each frame of the ultrasound image is associated with time information of the image capture.

[0085] The control unit 18 determines whether to end the ultrasonic imaging depending on whether or not an instruction to end the ultrasonic imaging has been input from the user via the operation unit 11 (step S23). If the ultrasonic imaging is not to be ended (step S23; NO), the process proceeds to step S21. If the ultrasonic imaging is to be ended (step S23; YES), the control unit 18 receives the generated electrocardiogram data from the electrocardiogram measurement device 500 via the communication unit 31 (step S24). In response to step S24, the control unit 501 transmits the measured electrocardiogram data to the ultrasonic diagnostic device 100 via the communication unit 505. In step S24, the control unit 18 acquires ultrasonic image data of the moving image generated in step S22.

[0086] The control unit 18 generates synchronization information indicating a frame corresponding to a predetermined R wave of the electrocardiogram from the electrocardiogram data and ultrasound image data received in step S24 (step S25). In step S25, for example, the control unit 18 analyzes the electrocardiogram data to obtain time information of the R wave and identifies a frame of ultrasound image data of the moving image corresponding to the obtained time information. The control unit 18 adds a mark to the frame identified as the synchronization information. Note that the synchronization information is not limited to a mark of a frame corresponding to an R wave. For example, the control unit 18 may be configured to add frame information corresponding to an R wave as synchronization information to the DICOM information of the ultrasound image data. Alternatively, the control unit 18 may be configured to draw an electrocardiogram waveform of the electrocardiogram data corresponding to the time information on an ultrasound image of each frame of ultrasound image data of the moving image.

[0087] The control unit 18 transmits the moving image ultrasound image data, electrocardiogram data, and synchronization information of step S24 and step S25 to the image management device 600 via the communication unit 31 and stores them (step S26). In response to step S26, the control unit 601 receives the moving image ultrasound image data, electrocardiogram data, and synchronization information from the ultrasound diagnostic device 100 via the communication unit 605. The control unit 601 stores the received moving image ultrasound image data, electrocardiogram data, and synchronization information in the storage unit 603.

[0088] The first image display process executed by the image management device 600 will be described with reference to Fig. 10. The first image display process is a process for displaying dynamic image data and moving ultrasound image data in synchronization with the R wave frames of an electrocardiogram using synchronization information. Dynamic image data, electrocardiogram data, and synchronization information generated by the dynamic image generation process are stored in the storage unit 603 in advance. Also, dynamic ultrasound image data, electrocardiogram data, and synchronization information generated by the ultrasound image generation process are stored in the storage unit 603.

[0089] In the image management device 600, a user such as a clinician inputs an instruction to execute the first image display process via the operation unit 602. This execution instruction triggers the control unit 601 to execute the first image display process.

[0090] First, the control unit 601 reads and acquires dynamic image data, electrocardiogram data, and synchronization information from the storage unit 603 (step S31). In step S31, the control unit 601 reads and acquires ultrasonic image data of a moving image from the ultrasound image generation process, electrocardiogram data, and synchronization information from the storage unit 603.

[0091] Based on the synchronization information, the control unit 601 synchronizes the dynamic image data and the moving image ultrasound image data and displays them side by side on the display unit 604 together with the electrocardiogram waveforms of the respective electrocardiogram data (step S32). In step S32, the control unit 601 simultaneously displays frames corresponding to the R waves of the electrocardiogram waveforms of the dynamic image data and the moving image ultrasound image data so that the time phases are aligned. In this way, the control unit 601 synchronizes and plays back the dynamic image data and the moving image ultrasound image data together with the electrocardiogram waveform. For example, the first R wave frame of the dynamic image data and the first R wave frame of the ultrasound image data are played back so as to be displayed simultaneously.

[0092] As shown in FIG. 11 , in step S32, for example, an image display screen 640 is displayed. The image display screen 640 has a dynamic image 641, an ultrasound image 642, electrocardiogram waveforms 643 and 644, and frame position bars 645 and 646. The dynamic image 641 is a radiographic image of each frame of the dynamic image data. For example, the electrocardiogram waveform 643 is superimposed on the dynamic image 641, but this need not be the case. The ultrasound image 642 is an ultrasound image of each frame of the ultrasound image data of a moving image. The electrocardiogram waveform 643 is an electrocardiogram waveform of electrocardiogram data measured simultaneously with the dynamic imaging of the dynamic image 641. For example, the electrocardiogram waveform 644 is superimposed on the ultrasound image 642, but this need not be the case. The electrocardiogram waveform 644 is an electrocardiogram waveform of electrocardiogram data measured simultaneously with the ultrasound imaging of the ultrasound image 642. The playback position bar 645 indicates the playback position of the frame of the dynamic image 641 currently displayed in the electrocardiogram waveform 643. The playback position bar 646 indicates the playback position of the frame of the ultrasound image 642 currently being displayed in the electrocardiogram waveform 644 .

[0093] The dynamic image 641 and the ultrasound image 642 are played back in synchronization so that the R wave frame of the electrocardiogram waveform 643 and the R wave frame of the electrocardiogram waveform 644 are displayed simultaneously. In other words, the playback position of a playback position bar 645 from the R wave position of the electrocardiogram waveform 643 and the playback position of a playback position bar 646 from the R wave position of the electrocardiogram waveform 644 are the same.

[0094] In step S32, the FPS (Frames Per Second) of the display of the dynamic image data and the moving image ultrasound image data may differ. For example, the dynamic image data is usually 15 FPS, while the moving image ultrasound image data is 20 to 30 FPS. Thus, the FPS of the dynamic image data is usually lower than the FPS of the moving image ultrasound image data. For this reason, when playing back a moving image, the control unit 301 acquires FPS information from the dynamic image data and the moving image ultrasound image data, and displays frames based on the image data with the larger FPS information. This is to display the frames of the image data with the larger FPS in a detailed and smooth manner when playing back a moving image. Note that when advancing the frame while the moving image is stopped, the control unit 301 acquires FPS information from the dynamic image data and the moving image ultrasound image data, and displays frames based on the image data with the smaller FPS information. This is to reliably change the frames of the image data with the smaller FPS when advancing the frame while the moving image is stopped. The first image display process ends.

[0095] In step S31, dynamic image data and synchronization information from the dynamic image generation process, and moving image ultrasound image data and synchronization information from the ultrasound image generation process may be read from the storage unit 603. In step S32, the dynamic image data and moving image ultrasound image data are synchronized and displayed side by side. In other words, the electrocardiogram data (electrocardiogram waveform) is not displayed.

[0096] As described above, according to this embodiment, the image management device 600 includes a control unit 601. The control unit 601 displays dynamic information and ultrasound information on the display unit 604 in synchronization with each other so as to match the time phases of the subject's condition. The dynamic information is dynamic image data showing morphological changes of the subject due to dynamic radiography. The ultrasound information is moving ultrasound image data showing morphological changes of the subject due to dynamic ultrasound photography.

[0097] Therefore, the time phase of the subject's condition can be matched between dynamic image data obtained by dynamic radiography and moving ultrasound image data obtained by ultrasound photography, allowing for accurate diagnosis of morphological changes in the subject's heart, etc. Furthermore, in addition to changes in the subject's part in the direction of radiation irradiation using dynamic image data, morphological changes in the subject in any direction that is the same as or different from the direction of radiation irradiation can be recognized using ultrasound image data. Therefore, changes in the subject can be accurately diagnosed from multiple angles.

[0098] The control unit 601 synchronizes the dynamic image data and the ultrasound image data and displays them on the display unit 604 based on the synchronization information. The synchronization information is information for synchronizing frames of dynamic image data with frames of ultrasound data. The frames of the dynamic image data correspond to the time of the R wave of the electrocardiogram data measured simultaneously with the dynamic imaging. The frames of the ultrasound image data correspond to the time of the R wave of the electrocardiogram data measured simultaneously with the ultrasound imaging. Therefore, even in dynamic imaging and ultrasound imaging, which can only be performed at different times, the time phase can be synchronized with the electrocardiogram R wave appropriate for observing changes in the subject. This allows for more accurate diagnosis of changes in the subject over time.

[0099] The imaging system 1000 includes an image management device 600, a console 300 that controls dynamic imaging, and an ultrasound diagnostic device 100 that performs ultrasound imaging. The synchronization information includes first synchronization information and second synchronization information. The first synchronization information indicates a frame corresponding to the time of the R wave of electrocardiogram data measured simultaneously with the dynamic imaging. The second synchronization information indicates a frame corresponding to the time of the R wave of electrocardiogram data measured simultaneously with the ultrasound imaging. Therefore, the image management device 600 does not need to generate synchronization information, thereby reducing the processing load on the image management device 600.

[0100] (Second embodiment) A second embodiment of the present invention will be described with reference to Fig. 12. Fig. 12 is a flowchart showing a second image information processing.

[0101] In the first embodiment, the ultrasound diagnostic device 100 and the console 300 generate synchronization information together with dynamic image data and moving ultrasound image data. In the present embodiment, the image management device 600 generates synchronization information.

[0102] The device configuration of this embodiment is the same as that of the first embodiment, and uses the image capturing system 1000. However, instead of the first image display program, a second image display program for executing a second image display process (described later) is stored in the ROM of the control unit 601 of the image management device 600.

[0103] The operation of the imaging system 1000 of this embodiment will be described with reference to FIG. 12. As in the first embodiment, radiography is performed by the console 300 executing dynamic image generation processing. However, step S15 is not executed. In step S16, the control unit 301 transmits dynamic image data and electrocardiogram data to the image management device 600, which stores the data in the storage unit 603. Furthermore, ultrasound imaging is performed by the ultrasound diagnostic device 100 executing ultrasound image generation processing. However, step S25 is not executed. In step S26, the control unit 18 transmits ultrasound image data and electrocardiogram data to the image management device 600, which stores the data in the storage unit 603.

[0104] The second image display process executed by the image management device 600 will be described with reference to Fig. 12. The second image display process is a process of generating synchronization information and displaying dynamic image data and moving image ultrasound image data in synchronization with the R wave frames of an electrocardiogram. Dynamic image data and electrocardiogram data generated by the dynamic image generation process are stored in advance in the storage unit 603. Also, moving image ultrasound image data and electrocardiogram data generated by the ultrasound image generation process are stored in the storage unit 603.

[0105] In the image management device 600, a user such as a clinician inputs an instruction to execute the second image display process via the operation unit 602. This execution instruction serves as a trigger for the control unit 601 to execute the second image display process.

[0106] First, the control unit 601 reads and acquires the dynamic image data and electrocardiogram data generated by the dynamic image generation process from the storage unit 603 (step S41). In step S41, the control unit 601 reads and acquires the ultrasound image data and electrocardiogram data of the moving image generated by the ultrasound image generation process from the storage unit 603.

[0107] The control unit 601 generates synchronization information from the dynamic image data and electrocardiogram data obtained by the dynamic image generation process (step S42). In step S42, the control unit 601 generates synchronization information from the ultrasound image data of the moving image obtained by the ultrasound image generation process and the electrocardiogram data.

[0108] The control unit 601 synchronizes the dynamic image data and the moving image ultrasound image data based on the synchronization information generated in step S42, and displays them side by side on the display unit 604 along with the electrocardiogram waveforms of each electrocardiogram data (step S43). In step S43, the control unit 601 plays back the dynamic image data and the moving image ultrasound image data along with the electrocardiogram waveform so that the (first) R wave frames of the dynamic image data and the moving image ultrasound image data are simultaneously displayed. The second image display process then ends.

[0109] As described above, according to this embodiment, the control unit 601 generates synchronization information that associates frames of dynamic image data with frames of ultrasound image data. The frames of the dynamic image data correspond to the time of the R wave of electrocardiogram data measured simultaneously with ultrasound imaging. The frames of the ultrasound image data correspond to the time of the R wave of electrocardiogram data measured simultaneously with ultrasound imaging. Therefore, the console 300 and the ultrasound diagnostic device 100 do not need to generate synchronization information, which reduces the processing load on the console 300 and the ultrasound diagnostic device 100. Furthermore, ordinary devices can be used for the console 300 and the ultrasound diagnostic device 100.

[0110] (Variation) A modified example of the first and second embodiments will be described. In the first and second embodiments, the synchronization information is generated using electrocardiogram data. In this modified example, the synchronization information is generated without using electrocardiogram data.

[0111] The device configuration of this modification uses an imaging system 1000. However, the electrocardiogram measurement device 500 is not used.

[0112] Machine learning such as deep learning is performed using dynamic image data, in which frames corresponding to R waves of an electrocardiogram are specified in advance by a user or the like, as correct training data. This machine learning generates a trained model that extracts (identifies) frames corresponding to R waves from the dynamic image data. The trained model for dynamic image data is stored in the storage unit 303 of the console 300.

[0113] Furthermore, machine learning is performed using ultrasound image data, in which frames corresponding to R waves of an electrocardiogram are specified by a user or the like, as correct training data. This machine learning generates a trained model that extracts frames corresponding to R waves from the ultrasound image data of the video. The trained model for ultrasound image data is stored in the storage unit 19 of the ultrasound diagnostic device 100.

[0114] The dynamic image generation process of FIG. 8 is executed in the console 300. At this time, step S11 is not executed. In step S14, dynamic image data is received from the FPD 203. In step S15, the control unit 301 reads out a trained model for dynamic image data from the storage unit 303. The control unit 301 extracts R-wave frames from the dynamic image data using the trained model for dynamic image data. The control unit 301 generates synchronization information indicating the extracted frames of the dynamic image data. In step S16, the dynamic image data and the synchronization information are stored in the storage unit 603 of the image management device 600.

[0115] In the ultrasound diagnostic device 100, the ultrasound image generation process of FIG. 9 is executed. At this time, step S21 is not executed. In step S24, moving image ultrasound image data is acquired. In step S25, the control unit 18 reads out a trained model for ultrasound image data from the storage unit 19. The control unit 18 extracts R-wave frames from the moving image ultrasound image data using the trained model for ultrasound image data. The control unit 18 generates synchronization information indicating the extracted frames of the ultrasound image data. In step S26, the moving image ultrasound image data and the synchronization information are stored in the storage unit 603.

[0116] 10 is executed in the image management device 600. In step S31, dynamic image data, moving image ultrasound image data, and synchronization information are read from the storage unit 603. In step S32, the dynamic image data and moving image ultrasound image data are synchronized and displayed on the display unit 604.

[0117] A case will be described in which the second image display process of FIG. 11 is executed in the image management device 600 in accordance with the second embodiment. A trained model for dynamic image data and a trained model for ultrasound image data are stored in advance in the storage unit 603. In step S41, dynamic image data, moving image ultrasound image data, and synchronization information are read from the storage unit 603. In step S42, the control unit 601 reads the trained model for dynamic image data from the storage unit 603. The control unit 601 uses the trained model for dynamic image data to extract R-wave frames from the dynamic image data. The control unit 601 generates synchronization information indicating the extracted frames of the dynamic image data. In step S42, the control unit 601 reads the trained model for ultrasound image data from the storage unit 603. The control unit 601 uses the trained model for ultrasound image data to extract R-wave frames from the moving image ultrasound image data. The control unit 601 generates synchronization information indicating the extracted frames of the ultrasound image data. In step S43, the control unit 601 uses the synchronization information to synchronize the dynamic image data with the moving image ultrasound image data and displays them on the display unit 604.

[0118] According to this modification, the control unit 301 uses the first trained model to extract frames corresponding to the time of R waves from the dynamic image data and generates synchronization information indicating the extracted frames. The control unit 18 uses the second trained model to extract frames corresponding to the time of R waves from ultrasound image data of a moving image and generates synchronization information indicating the extracted frames. The control unit 601 synchronizes the dynamic image data and the ultrasound image data based on the synchronization information and displays them on the display unit 604. The first trained model extracts frames of dynamic image data corresponding to the time of R waves of an electrocardiogram from the dynamic image data. The second trained model extracts frames of ultrasound image data corresponding to the time of R waves of an electrocardiogram from the ultrasound image data.

[0119] In another configuration, the control unit 601 uses the first trained model to extract frames of dynamic image data corresponding to the time of an R wave from the dynamic image data and generates synchronization information indicating the extracted frames. The control unit 601 uses the second trained model to extract ultrasound image data corresponding to the time of an R wave from the ultrasound image data and generates synchronization information indicating the extracted frames. The control unit 601 synchronizes the frames of the dynamic image data and the ultrasound image data based on the synchronization information and displays the dynamic image data and the ultrasound image data on the display unit 604.

[0120] Therefore, the time phases of the subject's condition between dynamic image data obtained by dynamic radiation imaging and ultrasound image data obtained by ultrasound imaging can be matched without the need for an electrocardiogram measurement device 500. Furthermore, the configuration of the imaging system 1000 can be simplified.

[0121] 9, when the subject's heart is ultrasonically imaged, the control unit 301 may be configured to perform image analysis of the ultrasonic image data of the moving image. The control unit 301 determines the frame in which the volume of the left ventricle of the subject's heart (area on the image) is maximized as the frame corresponding to the R wave of the electrocardiogram. Similarly, in step S42 of the second image display processing in FIG. 12, the control unit 601 may be configured to perform image analysis of the ultrasonic image data of the moving image and determine the frame in which the volume of the left ventricle of the heart is maximized as the frame corresponding to the R wave of the electrocardiogram.

[0122] In the above description, an example has been disclosed in which a ROM, HDD, or SSD is used as a computer-readable medium for the program according to the present invention, but the present invention is not limited to this example. Other computer-readable media include non-volatile memory such as flash memory and portable recording media such as CD-ROM. Furthermore, a carrier wave is also applicable to the present invention as a medium for providing data for the program according to the present invention via a communication line.

[0123] The above-described embodiments and modifications are merely examples of the information processing device, information processing system, information processing method, and program according to the present invention, and are not intended to be limiting. For example, at least two of the above-described embodiments and modifications may be appropriately combined.

[0124] In addition, in step S15 of the dynamic image generation processing in FIG. 8 of the first embodiment, synchronization information is automatically generated by information processing, but this is not limited to this. In step S15, the control unit 301 displays dynamic image data and electrocardiogram data on the display unit 304. The control unit 301 receives a designation input of a frame of dynamic image data corresponding to an R wave from the user via the operation unit 302. The control unit 301 generates synchronization information in response to the designation input. Similarly, in step S25 of the ultrasound image generation processing in FIG. 9, the control unit 18 displays ultrasound image data and electrocardiogram data on the display unit 17. The control unit 18 receives a designation input of a frame of ultrasound image data corresponding to an R wave from the user via the operation unit 11. The control unit 18 generates synchronization information in response to the designation input.

[0125] The same applies to the second embodiment. In step S42 of the second image display processing in FIG. 12, the control unit 601 displays dynamic image data and its electrocardiogram data, and moving image ultrasound image data and its electrocardiogram data on the display unit 604. The control unit 601 receives input from the user via the operation unit 602 specifying frames of dynamic image data and ultrasound image data corresponding to R waves. The control unit 601 generates synchronization information in response to the operation input. The operation input corresponds a frame of dynamic image data corresponding to the time of an R wave in electrocardiogram data measured simultaneously with dynamic imaging with a frame of ultrasound image data corresponding to the time of an R wave in electrocardiogram data measured simultaneously with ultrasound imaging. This allows the user to reliably specify frames of dynamic image data and ultrasound image data to be synchronized.

[0126] Furthermore, in the first and second embodiments and the modified example, only dynamic image data is displayed in synchronization with the ultrasound image data of the moving image. However, this is not limiting. For example, consider the dynamic image generation process of FIG. 8 executed by the console 300 of the first embodiment. In step S16, the control unit 301 transmits the dynamic image data acquired in step S14 to the dynamic analysis device 400 via the communication unit 305, performs dynamic analysis, and obtains dynamically analyzed functional information. The control unit 301 transmits the dynamic image data and functional information to the image management device 600 and stores them in the storage unit 603. In the first image display process of FIG. 10 executed by the image management device 600, the control unit 601 displays the dynamic image data synchronized with the ultrasound image data of the moving image together with the functional information on the display unit 604. Similarly, in the second embodiment, in the second image display process of FIG. 11, the control unit 601 displays the dynamic image data together with the functional information on the display unit 604. This functional information is, for example, image data corresponding to each frame of the dynamic image data, or data indicating characters or numerical values.

[0127] Furthermore, a configuration may be adopted in which synchronization information is generated to synchronize functional information corresponding to the R waves of the electrocardiogram data with frames of ultrasound image data of a moving image corresponding to the R waves of the electrocardiogram data. Based on the synchronization information, the control unit 601 synchronizes the functional information with the ultrasound image data of the moving image and displays them on the display unit 604.

[0128] In the above-described embodiment and modified example, dynamic image data and moving ultrasound image data are played back in parallel by synchronizing one R-wave frame of each electrocardiogram data during dynamic imaging and ultrasound imaging. However, this is not limited to this. Dynamic image data and moving ultrasound image data may also be played back by synchronizing all R-wave frames of each electrocardiogram data during dynamic imaging and ultrasound imaging. In particular, the interval between R waves in electrocardiogram data during dynamic imaging may differ from the interval between R waves in electrocardiogram data during ultrasound imaging. For this reason, frames of dynamic image data and ultrasound image data may be interpolated or deleted so that the interval between R waves during dynamic imaging and ultrasound imaging are the same.

[0129] For example, in step S15 of the dynamic image generation processing of FIG. 8 according to the first embodiment, the control unit 301 generates synchronization information indicating frames of dynamic image data corresponding to all R waves of the electrocardiogram data. In step S25 of the ultrasound image generation processing of FIG. 9, the control unit 18 generates synchronization information indicating frames of ultrasound image data corresponding to all R waves of the electrocardiogram data. After step S31 of the first image display processing of FIG. 10, the control unit 601 acquires synchronization information for the dynamic image data and synchronization information for the ultrasound image data. Using the acquired synchronization information, the control unit 601 interpolates or deletes frames of the dynamic image data or ultrasound image data so that all R-wave frames are displayed simultaneously. When interpolating frames, the image data to be interpolated is either the dynamic image data or the ultrasound image data, whichever has the shorter interval between R waves. When deleting frames, the image data to be deleted is either the dynamic image data or the ultrasound image data, whichever has the longer interval between R waves. In step S32, the control unit 601 reproduces the interpolated or deleted dynamic image data and ultrasound image data on the display unit 604 so that all R-wave frames are displayed simultaneously.

[0130] In the second embodiment, after step S42 of the second image display process in Fig. 12, the control unit 601 acquires synchronization information for dynamic image data and synchronization information for ultrasound image data. Using the acquired synchronization information, the control unit 601 interpolates or deletes frames of the dynamic image data or ultrasound image data so that all R-wave frames are displayed simultaneously. In step S42, the control unit 601 plays the interpolated or deleted dynamic image data and ultrasound image data on the display unit 604 so that all R-wave frames are displayed simultaneously.

[0131] While embodiments of the present invention have been described and illustrated in detail, the disclosed embodiments are made for purposes of illustration and example only, and are not intended to be limiting. The scope of the present invention should be interpreted by the terms of the appended claims. [Explanation of symbols]

[0132] 1000 Image Acquisition System 100 Ultrasound diagnostic equipment 1. Ultrasound diagnostic device 11 Control section 12 Transmitter 13 Receiving unit 14 Signal processing section 15 Image processing section 151 Image memory section 16 Display control unit 17 Display 18 Control Unit 19 Memory section 31 Communications Department 2 Ultrasonic probe 21 Ultrasonic probe body 211 Oscillator 22 Cable 23 Connector 2000 Dynamic Image Generation Device 200 Radiography equipment 201 Radiation source 202 Radiation exposure control device 203 FPD 300 Console 301 Control Unit 302 Operation section 303 Storage section 304 Display section 305 Communications Department 400 Dynamic analysis device 500 Electrocardiogram measuring device 501 Control section 502 Operation section 503 Storage section 504 Display section 505 Communications Department 506 Electrocardiogram Measurement Unit 600 Image management device 601 Control Unit 602 Operation section 603 Storage section 604 Display section 605 Communications Department

Claims

1. An information processing device having a control unit that synchronizes and displays dynamic information showing changes in a subject through dynamic radiation imaging and ultrasound information showing morphological changes in the subject through moving ultrasound imaging so that the time phases of the subject's condition are aligned.

2. The dynamic information is at least one of dynamic image data and functional information analyzed from the dynamic image data, The information processing apparatus according to claim 1 , wherein the ultrasound information is moving ultrasound image data.

3. 2. The information processing device according to claim 1, wherein the control unit displays the dynamic information and the ultrasound information in synchronization with each other based on synchronization information for synchronizing the dynamic information corresponding to the time of a predetermined wave of first electrocardiogram data measured simultaneously with the dynamic imaging and the ultrasound information corresponding to the time of the predetermined wave of second electrocardiogram data measured simultaneously with the ultrasound imaging.

4. The information processing device according to claim 3, wherein the control unit generates the synchronization information that matches the dynamic information corresponding to the time of a predetermined wave of the first electrocardiogram data with the ultrasound information corresponding to the time of a predetermined wave of the second electrocardiogram data.

5. 5. The information processing device according to claim 4, wherein the control unit generates the synchronization information in response to an operation input that matches the dynamic information corresponding to the time of a predetermined wave of the first electrocardiogram data with the ultrasound information corresponding to the time of a predetermined wave of the second electrocardiogram data.

6. 2. The information processing device according to claim 1, wherein the control unit uses a first trained model that extracts dynamic information corresponding to the time of a predetermined wave of an electrocardiogram from the dynamic information to extract dynamic information corresponding to the time of the predetermined wave from the dynamic information, and uses a second trained model that extracts ultrasound information corresponding to the time of the predetermined wave from the ultrasound information to extract ultrasound information corresponding to the time of the predetermined wave from the ultrasound information, generates synchronization information for synchronizing the extracted dynamic information and ultrasound information, and displays the dynamic information and the ultrasound information in synchronization based on the synchronization information.

7. The information processing device according to claim 3; a dynamic imaging control device that controls the dynamic imaging; an ultrasonic diagnostic device that performs the ultrasonic imaging; the synchronization information includes first synchronization information indicating the dynamic state information corresponding to a time of a predetermined wave of the first electrocardiogram data, and second synchronization information indicating the ultrasound information corresponding to a time of a predetermined wave of the second electrocardiogram data, the dynamic imaging control device generates the first synchronization information; The ultrasonic diagnostic apparatus includes an information processing system that generates the second synchronization information.

8. the dynamic imaging control device generates the first synchronization information in response to an operation input that associates a time of a predetermined wave of the first electrocardiogram data with the dynamic information; The information processing system according to claim 7 , wherein the ultrasound diagnostic device generates the second synchronization information in response to an operation input that associates a time of a predetermined wave of the second electrocardiogram data with the ultrasound information.

9. The information processing device according to claim 1 ; a dynamic imaging control device that controls the dynamic imaging; an ultrasonic diagnostic device that performs the ultrasonic imaging; the dynamic imaging control device extracts, from the dynamic information, dynamic information corresponding to the time of a predetermined wave using a first trained model that extracts, from the dynamic information, dynamic information corresponding to the time of the predetermined wave, and generates first synchronization information indicating the extracted dynamic information; the ultrasound diagnostic device extracts ultrasound information corresponding to the time of a predetermined wave from the ultrasound information using a second trained model that extracts ultrasound information corresponding to the time of the predetermined wave from the ultrasound information, and generates second synchronization information indicating the extracted ultrasound information; The control unit is an information processing system that synchronizes and displays the dynamic information and the ultrasound information based on synchronization information that includes the first synchronization information and the second synchronization information and is used to synchronize the extracted dynamic information and ultrasound information.

10. An information processing method including a control step of synchronizing and displaying dynamic information showing changes in a subject obtained by dynamic radiation imaging and ultrasound information showing morphological changes in the subject obtained by moving ultrasound imaging so that the time phases of the subject's condition are matched.

11. Computer, a control unit that synchronizes and displays dynamic information showing changes in the subject obtained by dynamic radiography and ultrasound information showing morphological changes in the subject obtained by ultrasonic imaging of a moving image, so that the time phases of the state of the subject are matched; A program to function as a

Images