Information processing apparatus and method of operating information processing apparatus

Abstract

An aspect of the present invention provides an information processing apparatus capable of appropriately storing medical images and a method for operating the information processing apparatus.SOLUTION: According to an aspect of the present invention, there is provided an information processing apparatus including a processor, wherein the processor is configured to acquire medical data including at least one of a medical image and a medical finding sentence concerning the medical image, detect predetermined medical information from the medical data by applying a machine learning method to the medical data, and move the medical image stored in a temporary storage area of a storage apparatus to a long-term storage area of the storage apparatus if the medical information is detected.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an operating method of the information processing device, and more particularly to a technique for storing medical information. [Background technology]

[0002] Regarding technology for storing medical information, for example, Patent Document 1 describes a technique for storing temporarily stored data in a permanent storage device in accordance with a user instruction. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-228921 Summary of the Invention [Problem to be solved by the invention]

[0004] Systems such as PACS (Picture Archiving and Communication System) are known that include a long-term storage area in which data such as medical images are stored for a long period of time, and a temporary storage area from which data is automatically deleted after a predetermined period of time has passed, depending on the remaining storage capacity, etc. Medical images can have large data sizes depending on imaging conditions such as modality and slice thickness, and if such medical images were always stored in the long-term storage area, it would consume storage capacity, so they are stored in the temporary storage area.

[0005] Furthermore, in recent years, the development of CT systems equipped with photon-counting detectors (PCCT: Photon Counting CT, CT stands for Computer Tomography) has progressed, and PCCT systems can not only obtain images with higher resolution than conventional CT systems, but also images that distinguish and visualize materials with different radiation attenuation coefficients (material-decomposed images).These high-resolution images and material-decomposed images have extremely large data sizes that put strain on storage, so they are expected to be stored in temporary memory areas.

[0006] When a user uses an image in the temporary storage area for diagnosis, the user may want to store the image in the long-term storage area for future reference. Also, when a user reads an image in the viewer, if the user registers the image in the temporary storage area as a key image in the report system or saves the image as a bookmark, the user copies or moves the image from the temporary storage area to the long-term storage area.

[0007] However, since users do not necessarily register key images or save them as bookmarks, if they do not perform such operations, a situation may arise in which "the image was stored in a temporary storage area, but has already been deleted, and so cannot be referenced even though one wishes to do so." However, conventional technologies such as those disclosed in Patent Document 1 above do not take such issues into consideration.

[0008] The present invention has been made in view of the above circumstances, and provides an information processing device and an operating method for the information processing device that can appropriately store medical images. [Means for solving the problem]

[0009] An information processing device according to a first aspect of the present invention is an information processing device including a processor, which acquires medical data including at least one of a medical image and a medical finding statement related to the medical image, applies a machine learning technique to the medical data to detect predetermined medical information from the medical data, and, if the medical information is detected, moves the medical image stored in a temporary storage area of ​​the storage device to a long-term storage area of ​​the storage device.

[0010] According to the first aspect, when predetermined medical information is detected, the medical images stored in the temporary storage area of ​​the storage device can be automatically moved to the long-term storage area of ​​the storage device, thereby reducing the possibility of a situation where "the medical images stored in the temporary storage area have been deleted and cannot be referenced even though you want to," and allowing the medical images to be stored appropriately. The "predetermined medical information" may be, for example, information that is important for diagnosis or report creation.

[0011] In the first aspect and each of the following aspects, "automatically" means that when a predetermined condition is met, a predetermined process or operation (moving medical information in the first aspect) is executed without user operation. This allows medical images for which the user has not registered a key image or saved a bookmark to be moved to a long-term storage area when predetermined medical information is detected, thereby reducing the burden on the user regarding the storage of medical images.

[0012] In the first aspect and each of the following aspects, the medical data includes at least one of a medical image and a medical finding statement related to the medical image. Therefore, the processor may detect the "predetermined medical information" from either the medical image or the medical finding statement, or from both. It is preferable that the medical image and the medical finding statement are linked (associated).

[0013] Furthermore, in the first aspect and each of the following aspects, the machine learning method used to detect the "predetermined medical information" is not particularly limited. In addition to deep learning such as CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), and Attention, algorithms such as decision trees, random forests, support vector machines, and naive Bayes may also be used.

[0014] In the first embodiment and each of the following embodiments, the storage periods of the long-term storage area and the temporary storage area are not particularly limited, but the long-term storage area is an area where information is stored for a longer period than the temporary storage area. The long-term storage area may be a permanent storage area (a storage area where information is not erased unless a user operates it).

[0015] In the information processing device according to the second aspect of the present invention, in the first aspect, if the processor acquires information indicating that the detection result is a false detection, the processor returns the medical image that was moved to the long-term storage area to the temporary storage area. The second aspect takes into consideration that there is little need to store the image in the long-term storage area in the case of a false detection. Note that in the second aspect and each of the following aspects, the processor may acquire "information indicating a false detection" based on the processing results of the processor or another information processing device, or may acquire information input by the user who performed the image interpretation or other operation as "information indicating a false detection."

[0016] In the information processing device according to the third aspect, in the first or second aspect, the processor moves the medical image from the temporary storage area to the long-term storage area when it acquires information indicating that the medical image has been displayed on the display device together with the detection result (the detection result of the "predetermined medical information" defined in the first aspect). When it acquires the information defined in the third aspect, the processor can determine that the "predetermined medical information" in the first aspect has been detected.

[0017] In an information processing device according to a fourth aspect, in any one of the first to third aspects, when specific information is detected from a medical finding text, the processor moves a medical image associated with the specific information from a temporary storage area to a long-term storage area.

[0018] In the information processing device according to the fifth aspect, in the fourth aspect, the processor detects, as specific information, information indicating an organ that is the subject of the medical finding text from the medical finding text, and moves a medical image showing the organ from the temporary storage area to the long-term storage area. The fifth aspect specifies a specific aspect of the "specific information" in the fourth aspect.

[0019] An information processing device according to a sixth aspect is the fourth or fifth aspect, wherein the processor detects a specific phrase as the specific information from the medical finding text, and moves a medical image related to the specific phrase.

[0020] An information processing device according to a seventh aspect is the sixth aspect, and the processor detects a term related to a region of interest as the specific term. The "term related to the region of interest" may be, for example, one or more of the existence, type, size, shape, and color of the region of interest. Note that the "region of interest" (ROI) is also called a "region of interest."

[0021] An information processing device according to an eighth aspect is any one of the fourth to seventh aspects, wherein the processor detects specific information from the medical finding text using a natural language processing technique. The processor may perform the natural language processing using a machine learning technique. As described above for the first aspect, the machine learning technique used in the eighth aspect is not particularly limited and may be deep learning such as CNN, RNN, LSTM, or Attention, or may be a decision tree, random forest, support vector machine, naive Bayes, or the like.

[0022] An information processing device according to a ninth aspect is any one of the first to eighth aspects, wherein when a region of interest is detected in a first medical image, the processor moves the first medical image and a second medical image related to the first medical image and of a different image type from the first medical image from a temporary storage area to a long-term storage area.

[0023] An information processing device according to a tenth aspect is the ninth aspect, wherein the second medical image is an image that is different from the first medical image in at least one of modality, imaging parameters, and image processing content.

[0024] An information processing device according to an eleventh aspect is the ninth or tenth aspect, wherein the processor moves a medical image of the second medical image that shows a specified organ and / or region of interest.

[0025] An information processing device according to a twelfth aspect is any one of the first to eleventh aspects, wherein the processor acquires a CT image and / or a material decomposition image as the medical image.

[0026] An information processing device according to a thirteenth aspect is the twelfth aspect, wherein the CT image and the material decomposition image are images captured by a photon-counting CT device.

[0027] An information processing device according to a fourteenth aspect is any one of the seventh, ninth and eleventh aspects, wherein the region of interest is at least one of a lesion, a lesion candidate region and a post-treatment region.

[0028] In an information processing device according to a 15th aspect, in any one of the first to fourteenth aspects, the processor causes an output device to output information indicating medical data stored in a temporary memory area and / or information indicating medical data stored in a long-term memory area.

[0029] In an information processing device according to a 16th aspect, in any one of the first to fifteenth aspects, the processor causes an output device to output information indicating the history of movement of medical images between a temporary storage area and a long-term storage area.

[0030] In an information processing device according to a seventeenth aspect, in any one of the first to sixteenth aspects, the processor moves the medical images to a long-term storage area, which is an area that stores the medical images for a longer period of time than the temporary storage area.

[0031] An eighteenth aspect of the present invention relates to a method for operating an information processing device, which includes a processor. The processor acquires medical data including at least one of medical images and medical findings associated with the medical images, applies machine learning techniques to the medical data, and detects predetermined medical information from the medical data. If the medical information is detected, the processor moves the medical image stored in a temporary storage area of ​​a storage device to a long-term storage area of ​​the storage device. The eighteenth aspect of the present invention allows medical images to be appropriately stored, as in the first aspect, and also reduces the burden on users associated with storing medical images. Note that a program for causing a computer to execute the method according to the eighteenth aspect, a program product, and a non-transitory tangible recording medium storing computer-readable code for such a program can also be cited as aspects of the present invention. The term "non-transitory tangible recording medium" does not include non-tangible recording media such as carrier signals or propagation signals themselves. [Effects of the Invention]

[0032] As described above, the information processing device and the method for operating the information processing device of the present invention make it possible to appropriately store medical images and also to reduce the burden on the user regarding the storage of medical images. [Brief explanation of the drawings]

[0033] [Figure 1] FIG. 1 is a diagram showing the configuration of a medical information processing system. [Figure 2] FIG. 2 is a diagram showing the main configuration of the information processing device. [Figure 3] FIG. 3 is a schematic diagram showing an example of the layer structure of a recognizer. [Figure 4]FIG. 4 is a schematic diagram showing an example of the layer structure of the intermediate layer. [Figure 5] FIG. 5 is a schematic diagram showing an example of the configuration of a detector using an RNN. [Figure 6] FIG. 6 is a flowchart showing the processing in the medical information processing system. [Figure 7] FIG. 7 is a diagram showing an example of a movement history of a medical image. [Figure 8] FIG. 8 is a flowchart showing the process of returning medical images that have been moved to the long-term storage area to the temporary storage area. [Figure 9] FIG. 9 is another flowchart showing the processing in the medical information processing system. DETAILED DESCRIPTION OF THE INVENTION

[0034] [First embodiment] [Configuration of medical information processing system] 1 is a diagram showing the configuration of a medical information processing system 10 (medical information processing system) according to the first embodiment. As shown in the figure, the medical information processing system 10 includes a PCCT apparatus 100 (PCCT: Photon Counting CT, CT stands for Computer Tomography), an MRI apparatus 200 (MRI: Magnetic Resonance Imaging), an information processing apparatus 300 (information processing apparatus), a PACS 400 (Picture Archiving and Communication System: PACS), a viewer server 500, a viewer terminal 600, and a report server 700, and these devices are connected via a network 20 so as to be able to communicate with each other as necessary.

[0035] The medical information processing system 10 may include other modalities (medical imaging devices), such as a general CT device. The medical information processing system 10 may also include a hospital information system (HIS), a radiology information system (RIS), etc.

[0036] [PCCT device] Conventional CT systems are equipped with solid-state scintillation detectors. These solid-state scintillation detectors convert X-rays into visible light, which is then converted into electrical signals by photodiodes. In contrast, the photon-counting detectors installed in the PCCT system 100 can directly convert X-ray photons into electrical signals. Photon-counting detectors improve dose utilization efficiency compared to conventional detectors, and because their pixels are extremely small, they can significantly improve spatial resolution. As a result, the PCCT system 100 (photon-counting CT system) can obtain images with higher contrast and resolution than conventional CT systems, while also reducing the amount of radiation and contrast agent used.

[0037] The PCCT device 100 can capture and generate CT images with different imaging parameters and image processing contents (for example, slice position, slice thickness, slice direction (axial, sagittal, coronal, etc.), highlighting of specific organs or regions of interest, etc.). Furthermore, the PCCT device 100 can generate not only normal CT images but also images in which materials with different radiation attenuation coefficients are differentiated and visualized (material-decomposed images). It is preferable that the PCCT device 100 is a device that can generate multiple material-decomposed images with different discrimination contents and / or degrees.

[0038] [MRI device] The MRI device 200 is a device that uses the nuclear magnetic resonance (NMR) phenomenon to create an image of information inside a living body.

[0039] [PACS] PACS400 (Picture Archiving and Communication Systems) is a system that receives, stores, and manages medical images taken with modalities such as PCCT device 100 and MRI device 200 via a network 20. PACS is introduced to manage and share large volumes of images, and to meet the need to "read necessary images immediately," and by linking with other systems at medical institutions (including the elements in Figure 1), it can improve the efficiency of work involving the handling of medical images.

[0040] The PACS 400 includes a storage device 410 (storage device) for storing medical images and the like. The storage device 410 is composed of a storage device such as a magneto-optical storage device or a semiconductor memory, and its control device. The storage device 410 has a temporary storage area 412 (temporary storage area) and a long-term storage area 414 (long-term storage area). While the storage period of information in the temporary storage area 412 and the long-term storage area 414 is not particularly limited, the long-term storage area 414 is assumed to be an area where information is stored for a longer period than the temporary storage area 412. The long-term storage area 414 may be a permanent storage area (a storage area where information is not erased unless a user operates it). It is preferable that the temporary storage area 412 and the long-term storage area 414 are storage areas on a non-volatile storage device. The temporary storage area 412 and the long-term storage area 414 may be different areas on a common storage device, or may be storage areas on separate storage devices. In addition to the storage device 410, the PACS 400 also includes a processor and memory (not shown).

[0041] In the first embodiment, the PACS 400 is described as having a temporary storage area 412 and a long-term storage area 414, but instead of or in addition to this, other devices such as the information processing device 300 may have a temporary storage area and a long-term storage area.

[0042] [Information processing device] 2 is a diagram showing the main configuration of an information processing device 300. As shown in FIG. 2, the information processing device 300 includes a processor 310, a ROM (Read Only Memory) 320, a RAM (Random Access Memory) 330, a storage device 340, and a communication interface 350.

[0043] [Processor] The processor 310 is configured with various processors and electrical circuits, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a PLD (Programmable Logic Device), etc. When these processors and electrical circuits execute software (programs), computer-readable code of the software to be executed (for example, various processors and electrical circuits constituting the processor, and / or a combination thereof) is stored in a non-transitory and tangible recording medium such as the ROM 320, and the computer accesses the software.

[0044] The software stored in the non-transitory, tangible recording medium includes the program of the present invention (the method of operating the information processing device of the present invention and the program for operating the information processing device) and data used in executing the program. Instead of the ROM 320, the code may be recorded in a non-transitory, tangible recording medium such as a flash ROM or an EEPROM (Electronically Erasable and Programmable Read Only Memory). Note that this "non-transitory, tangible recording medium" does not include non-tangible recording media such as carrier signals or propagation signals themselves. During processing using the software, the RAM 330 is used as a temporary storage area or working area.

[0045] The processor 310 mainly comprises an input / output control unit 312, a medical information detection unit 314, and a movement control unit 316. The input / output control unit 312 acquires medical data including at least one of medical images and medical findings related to the medical images from the storage device 410 of the PACS 400, the viewer server 500, or the report server 700, and also acquires various processing conditions and outputs (stores and displays) the processing results. The medical information detection unit 314 detects predetermined medical information from the acquired medical data. The movement control unit 316 controls the movement of medical images between the temporary storage area 412 and the long-term storage area 414. The processor 310 may also have a configuration that performs processing other than that of each unit described above.

[0046] [Functional implementation using various processors] The functions of each part of the processor 310 described above can be realized using various processors and recording media. The various processors include, for example, a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to realize various functions. The various processors also include a GPU (Graphics Processing Unit), which is a processor specialized for image processing, and a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacturing. A configuration using a GPU is effective when performing image learning and recognition, as in the present invention. Furthermore, the various processors described above also include dedicated electrical circuits, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing specific processing.

[0047] The functions of each unit may be realized by a single processor, or by multiple processors of the same or different types (e.g., multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). Also, multiple functions may be realized by a single processor. Examples of multiple functions configured by a single processor include: a first configuration, as typified by a computer, in which a single processor is configured by combining one or more CPUs and software, and this processor realizes multiple functions; a second configuration, as typified by a system-on-chip (SoC), in which a processor is used to realize the functions of the entire system on a single IC (Integrated Circuit) chip; and various functions are thus configured as hardware structures using one or more of the various processors described above. Furthermore, the hardware structures of these various processors are, more specifically, electrical circuits combining circuit elements such as semiconductor devices. These electrical circuits may be electrical circuits that realize the above-mentioned functions using logical operations such as logical sum, logical product, logical negation, exclusive OR, and combinations of these.

[0048] When the processor or electrical circuit executes software (programs), computer-readable code for the software to be executed (e.g., various processors and electrical circuits constituting processor 310, and / or a combination thereof) is stored in a non-transitory recording medium such as ROM 320 (Read Only Memory), and the computer references the software. The software stored in the non-transitory recording medium includes a program for executing the method for operating an information processing device according to the present invention and data used during execution (e.g., data related to the acquisition of medical data, data used to detect medical information, parameters used in the recognizer and detector described below, and a list of specific words to be detected). Instead of ROM 320, the code and data may be recorded in various non-transitory recording media such as optical magnetic recording devices and semiconductor memories. During processing using the software, for example, RAM 330 (Random Access Memory) is used as a temporary storage area, and data stored in, for example, an EEPROM (Electronically Erasable and Programmable Read Only Memory), not shown, may also be referenced. A storage device 340 may be used as a non-transitory recording medium for recording the code and data.

[0049] The processing performed by each of these units will be described in detail below.

[0050] [Construction of a medical information detection unit using machine learning techniques] In the first embodiment, the medical information detection unit 314 includes a recognizer 314A (processor) that recognizes a region of interest in a medical image (CT image, material decomposition image, MRI image, etc.), and a detector 314B (processor) that detects specific information from a medical finding text. The recognizer 314A and the detector 314B can be constructed using a machine learning technique.

[0051] [Recognizer configuration] The recognizer 314A can be constructed using, for example, a neural network such as a convolutional neural network (CNN) or a deep neural network (DNN), or a support vector machine (SVM), and can detect, differentiate, and measure a region of interest (ROI). The region of interest may be at least one of a lesion, a candidate lesion region, and a post-treatment region. The region of interest is sometimes called a region of interest. Below, the layer structure when the recognizer is constructed using a CNN is described. Note that the detector (the recognizer that detects the region of interest) will be mainly described, but a similar layer structure can also be used for classification (differentiation) and measurement.

[0052] FIG. 3 is a schematic diagram illustrating an example of the layer configuration of the recognizer 314A. In the example shown in part (a) of FIG. 3, the recognizer 314A includes an input layer 362A, an intermediate layer 362B, and an output layer 362C. The input layer 362A receives a medical image acquired by the input / output control unit 312 and outputs features. The intermediate layer 362B includes a convolutional layer 364 and a pooling layer 365, and receives the features output by the input layer 362A and calculates other features. These layers have a structure in which multiple "nodes" are connected by "edges," and hold multiple weight parameters. The values ​​of the weight parameters change as learning progresses. The recognizer 314A may also include a fully connected layer 366, as shown in part (b) of FIG. 3. The layer configuration of the recognizer 314A is not limited to a case in which the convolutional layer 364 and the pooling layer 365 are repeated one by one, but may also include multiple consecutive layers (e.g., the convolutional layer 364). Additionally, multiple fully connected layers 366 may be included in series.

[0053] [Processing in the middle layer] The intermediate layer 362B calculates features through convolution and pooling. The convolution performed in the convolution layer 364 is a process for obtaining a feature map through convolution using a filter, and is responsible for extracting features such as edge extraction from an image. The convolution using this filter generates a "feature map" of one channel (one image) per filter. The size of the "feature map" is downscaled by the convolution, becoming smaller as convolution is performed in each layer. The pooling process performed in the pooling layer 365 reduces (or enlarges) the feature map output by the convolution to create a new feature map, and is responsible for providing robustness to the extracted features so that they are not affected by translation, etc. The intermediate layer 362B can be composed of one or more layers that perform these processes.

[0054] FIG. 4 is a schematic diagram illustrating an example configuration of the intermediate layer 362B of the recognizer 314A shown in FIG. 3. The first convolutional layer of the intermediate layer 362B performs a convolution operation between an image set (a training image set during training, and a recognition image set during recognition) consisting of multiple medical images and a filter F1. The image set consists of N images (N channels), each with an image size of H vertically and W horizontally. When inputting color images, the images constituting the image set are three-channel images of R (red), G (green), and B (blue). The number of channels varies depending on the modality, such as CT images or MRI images. The filter F1 convolved with this image set has N channels (N images). For example, for a filter of size 5 (5 × 5), the filter size is 5 × 5 × N. The convolution operation using this filter F1 generates a "feature map" of one channel (one image) for each filter F1. For example, if the filter F2 used in the second convolutional layer is a filter of size 3 (3 × 3), the filter size will be 3 × 3 × M.

[0055] Similar to the first convolutional layer, the second to nth convolutional layers use filters F2 to F nThe size of the "feature map" in the nth convolutional layer is smaller than the size of the "feature map" in the second convolutional layer because it has been downscaled by the previous convolutional layers or pooling layers.

[0056] Of the layers in intermediate layer 362B, low-level feature extraction (edge ​​extraction, etc.) is performed in the convolutional layers closer to the input side, while higher-level feature extraction (extraction of features related to the shape, structure, etc. of the object) is performed closer to the output side. When segmentation is performed for measurement purposes, upscaling is performed in the latter convolutional layers, and the final convolutional layer obtains a "feature map" of the same size as the input image set. On the other hand, when performing object detection, position information only needs to be output, so upscaling is not required.

[0057] The intermediate layer 362B may include a layer that performs batch normalization in addition to the convolution layer 364 and the pooling layer 365. The batch normalization process is a process that normalizes the distribution of data in units of mini-batches when performing learning, and plays a role in accelerating learning, reducing dependency on initial values, suppressing overlearning, etc.

[0058] [Processing in the output layer] The output layer 362C is a layer that detects the position of a region of interest in an input medical image based on the feature values ​​output from the intermediate layer 362B and outputs the results. When performing segmentation, the output layer 362C grasps the position of the region of interest in the image at the pixel level using the "feature map" obtained from the intermediate layer 362B. In other words, it can detect whether each pixel in the medical image belongs to the region of interest and output the detection result. On the other hand, when performing object detection, no judgment at the pixel level is required, and the output layer 362C outputs the position information of the target object.

[0059] The output layer 362C may perform discrimination (classification) of the region of interest and output the discrimination result. For example, the output layer 362C may classify medical images into three categories, "neoplastic," "non-neoplastic," and "other," and output the discrimination result as three scores corresponding to "neoplastic," "non-neoplastic," and "other" (the sum of the three scores is 100%). Alternatively, if a clear classification can be achieved from the three scores, the output layer 362C may output the classification result. Note that when the discrimination result is output, the output layer 362C may or may not include a fully connected layer as the last layer or layers (see part (b) of Figure 3).

[0060] The output layer 362C may output the measurement results of the region of interest. When performing measurement using CNN, the target region of interest can be segmented, for example, as described above, and then measured by the processor 310 based on the segmentation results. Also, the measurement values ​​of the target region of interest can be output directly from the recognizer 314A. When the measurement values ​​are output directly, the measurement values ​​themselves are learned from the image, which results in a regression problem of the measurement values.

[0061] When using a CNN with the above-described configuration, it is preferable to perform a process (error backpropagation) in the learning process, in which the result output by the output layer 362C is compared with the correct recognition answer for the image set to calculate the loss (error), and to update the weight parameters in the intermediate layer 362B from the output layer to the input layer so as to reduce the loss.

[0062] The recognizer applied to the medical images (e.g., thin slice CT images, high-resolution images obtained by the PCCT device 100, and material decomposition images) stored in the temporary storage area 412 may not be a recognizer using a neural network like the recognizer 314A, but may be a CAD (Computer Aided Diagnosis / Detection) system that uses other machine learning (artificial intelligence) techniques.

[0063] The processor 310 may include multiple recognizers tailored to the characteristics of organs, regions, and regions of interest. These recognizers may be, for example, recognizers for adrenal tumors, urinary tract stones, gallstones, liver fat content, etc. Furthermore, the recognizers may have different algorithms (machine learning methods) and parameters depending on the characteristics of the organs, regions, and regions of interest.

[0064] [Detector configuration] FIG. 5 is a schematic diagram showing an example configuration of a detector 314B using an RNN. Detector 314B has an input layer 370, a hidden layer 380, and an output layer 390. The hidden layer 380 differs from a typical neural network in that it includes a hidden layer 382 representing the state at the current time (time t) and a hidden layer 384 representing the state at a past time (time t-1). By retaining the state of the hidden layer at time t-1 and using it for the input at the next time t, detector 314B can perform estimation using the past history of information input in a chronological order, such as natural language (in this embodiment, the context of characters, words, and phrases in a medical finding document). Note that detector 314B may include multiple components of the configuration shown in FIG. 5, or they may be configured hierarchically. Detector 314B may also be configured using LSTM (Long Short-Term Memory), a type of RNN.

[0065] The detector 314B detects specific information from the medical finding text using natural language processing techniques. The detector 314B can detect specific words or phrases as "specific information," but these "specific words" may be words or phrases related to the region of interest. The words or phrases related to the region of interest may be, for example, one or more of the presence or absence, position, type, condition or degree, number, shape, and size of the region of interest. The "specific words" may also be information indicating the organ that is the subject of the medical finding text, such as the name of the organ in which the region of interest is detected. The user of the medical information processing system 10 can set the type of information or words or phrases to be detected.

[0066] [Storage device] The storage device 340 (output device) is composed of a non-temporary storage medium such as an optical magnetic storage medium or various semiconductor memories, and its control unit, and can store medical data including at least one of medical images and medical findings related to the medical images, detection conditions for medical information (what information or phrases to detect, detection method, detection timing, etc.), detection results of medical information, information indicating movement history, etc.

[0067] [Display device and operating device] The information processing device 300 includes a display device (one aspect of an output device) and operation devices (keyboard, mouse, etc.), not shown. A user of the information processing device 300 can input instructions necessary to operate the information processing device 300 via these devices. Information stored in the storage device 340 and the results of execution of the method for operating the information processing device according to the present invention can be displayed on the display device in response to a user instruction via these display devices or operation devices, or automatically without a user instruction.

[0068] In addition to the above components, the information processing device 300 includes a communication interface 350 for communicating with other devices in the medical information processing system 10 via the network 20.

[0069] [Report server, viewer server, and viewer terminal] The viewer terminal 600 accepts user operations related to viewing medical images, inputting medical findings, and viewing reports via an operating device (keyboard, mouse, microphone, etc.) not shown, and communicates with the viewer server 500 or the like as necessary to obtain necessary information and display it on a display device (liquid crystal display, etc.) not shown. The viewer server 500 transmits and receives necessary information between the PACS 400, the information processing device 300, and the report server 700. The report server 700 manages the storage and viewing of reports created by the viewer terminal 600. The report server 700 may store the created reports in its own storage device or in the storage device 410 of the PACS 400. The report server 700 may be omitted, in which case the functions of the report server 700 can be realized by another device such as the PACS 400.

[0070] [Processing in medical information processing system (first aspect)] Next, the processing in the medical information processing system 10 having the above configuration will be described. Figure 6 is a flowchart showing the processing in the medical information processing system 10. In Figures 6, 8, and 9, the steps to the right of the dashed dotted lines in the figures indicate processing in the information processing device 300, and the steps to the left of the dashed dotted lines indicate processing in a device other than the information processing device 300. The order of processing in these figures is an example, and may be changed as necessary.

[0071] [Medical image generation] A medical imaging device such as the PCCT device 100 generates a medical image of a subject in response to a user's instruction (step S100). For example, the PCCT device 100 can generate a normal CT image, but it may generate one type of CT image (one aspect of a first medical image), or multiple CT images (one aspect of a first and second medical images) with different imaging parameters and / or image processing contents. The PCCT device 100 may also generate a normal CT image and a material-decomposed image (one aspect of a first and second medical image), or multiple types of material-decomposed images.

[0072] Furthermore, medical images may be generated by another modality (MRI apparatus 200 in the embodiment of FIG. 1) instead of or in addition to the PCCT apparatus 100. If the medical information processing system 10 includes other medical imaging apparatuses such as a normal CT apparatus or an ultrasound imaging apparatus, medical images may be generated by these apparatuses. Medical images generated by these modalities are also one embodiment of the first and second medical images of the present invention.

[0073] In this way, the medical information processing system 10 according to the first embodiment can generate multiple medical images. These medical images are one aspect of "a first medical image and a second medical image that is related to the first medical image but is a different type of image from the first medical image." In the first embodiment, examples of "the first medical image and the second medical image being related" include the same subject, the same or nearby imaging dates and times, and the same imaging site or slice position. Regarding "different types of images," the first medical image and the second medical image may differ in at least one of modality, imaging parameters, and image processing content.

[0074] [Storing medical images in temporary storage] The PACS 400 stores the medical image acquired in step S100 in the temporary storage area 412 (step S110), and transmits medical data including the medical image to the information processing device 300, instructing it to recognize a region of interest (step S120). The region of interest may be at least one of a lesion, a candidate lesion region, and a post-treatment region.

[0075] In the present invention, "medical data" includes at least one of medical images and medical findings related to the medical images. In the following first aspect, a case where medical data includes only medical images out of medical images and medical findings will be described, and a case where medical data includes medical images and medical findings (second aspect) will be described later.

[0076] [Medical image acquisition and region of interest recognition] The processor 310 (processor) of the information processing device 300 acquires a medical image (medical data) (step S300) and recognizes a region of interest (ROI) by applying a machine learning technique to the acquired medical image (step S310). Specifically, the recognizer 314A, which is configured using a neural network as described above, detects a ROI from the medical image. Note that while "recognizing" a ROI here refers to detection, the recognizer 314A may also perform differentiation or measurement. Once the recognition is complete, the processor 310 transmits the recognition result to the viewer server 500 (and / or the report server 700) and instructs the display device of the viewer terminal 600 to display the medical image and the recognition result (step S320). In response to this instruction, the viewer server 500 displays the medical image and the recognition result on the display device of the viewer terminal 600 (step S130).

[0077] [Identification / highlighting of regions of interest and notification of detection] When viewing medical images and recognition results, the viewer server 500 may distinguish and / or highlight the detected region of interest, or may notify the user that a region of interest has been detected, in response to instructions from the processor 310 in response to a user operation. Examples of the distinguishing and / or highlighting include displaying characters, numbers, figures, symbols, etc., and displaying the region of interest in color. Similarly, examples of the notifying mode include displaying characters, numbers, figures, symbols, etc., and outputting audio. The processor 310 can determine the distinguishing, highlighting, and notifying modes in response to a user instruction via the operating unit of the information processing device 300 or the operating unit of the viewer terminal 600.

[0078] The viewer terminal 600 accepts input of a medical finding statement for a medical image in response to a user operation (step S140), and associates the medical image with the medical finding statement in response to the input. The association can be performed, for example, using tag information of image data in DICOM (Digital Imaging and Communications in Medicine) format, but other formats are also possible. The report server 700 stores the associated medical image and medical finding statement in a storage device (not shown). If the medical information processing system 10 does not include the report server 700, the PACS 400 may store the associated data in the storage device 410.

[0079] [Transfer of medical images to long-term storage] Based on the result of step S310, the recognizer 314A (processor 310) determines whether a predetermined region of interest has been detected from the medical image (step S330). The "predetermined region of interest" is one aspect of "predetermined medical information," and may be, for example, a region of interest having specific characteristics (type, shape, size, quantity, degree, etc.). Note that "what kind of region of interest is to be detected" can be recorded in the storage device 340 or the like, and the processor 310 can refer to it to make the above determination.

[0080] If a predetermined region of interest is detected (YES in step S330), the movement control unit 316 (processor) instructs the PACS 400 to move the medical image to the long-term storage area 414 (step S340). The movement control unit 316 (processor) also outputs information indicating the movement history to an output device (step S350). This "output device" may be a display device or storage device 340 (not shown) of the information processing device 300, a display device of the viewer terminal 600, the storage device 410 of the PACS 400, etc. When the medical image has been moved, the movement control unit 316 may notify the user of this (screen display, audio output, etc.).

[0081] Furthermore, in response to an instruction (step S340) from the processor 310 (information processing device 300), the PACS 400 moves the medical images stored in the temporary storage area 412 to the long-term storage area 414. Therefore, the PACS 400 may output information indicating the history of the movement to an output device (the storage device 410 or a display device not shown).

[0082] [Movement of first and second medical images] As described above, in the medical information processing system 10, multiple types of medical images can be generated by the PCCT apparatus 100 and the MRI apparatus 200. Therefore, when multiple types of medical images are generated in step S100, the movement control unit 316 (processor) and the PACS 400 may move these medical images together to the long-term storage area 414.

[0083] Specifically, when a first medical image and a second medical image related to the first medical image but of a different image type from the first medical image are generated (for example, a normal CT image and a material decomposition image, a CT image and an MRI image, etc.), if a predetermined region of interest is detected in the first medical image, the first medical image and the second medical image may be moved to the long-term storage area 414. At this time, a medical image of the second medical image that shows a specified organ and / or region of interest may be moved to the long-term storage area. This allows the data size to be reduced.

[0084] In step S130, the medical image is displayed on the display device of the viewer terminal 600 together with the recognition result (detection result), but this display may or may not be a requirement for moving the medical image.

[0085] If the predetermined region of interest is not detected in step S330, the medical information processing system 10 continues the process without moving the medical image (step S360).

[0086] [Travel History Information] FIG. 7 is a diagram showing an example of output of movement history information. As described above, the output may be a storage device or a display device. In the example of FIG. 7, the IDs of medical images and medical findings, the movement date and time, the movement content, and the reason for the movement are summarized in a table format, and the user can easily understand the movement history by viewing this table. Note that links are provided to the medical images and medical findings, and the user can view the medical images and medical findings by specifying the ID. Note that the movement control unit 316 (processor) and / or the PACS 400 may output (display, record) to an output device information indicating the medical images stored in the temporary storage area 412 and / or information indicating the medical images stored in the long-term storage area 414, rather than information indicating the movement history.

[0087] [Handling false positives] As described above, in this embodiment, when a predetermined region of interest is detected from a medical image, the medical image is moved to the long-term storage area 414. However, there are cases where this detection is incorrect. In such cases, there is no need to move the medical image to the long-term storage area 414, and it can be returned to the temporary storage area 412 as follows.

[0088] 8 is a flowchart showing the process in the event of a false detection (the process of returning a medical image moved to the long-term storage area to the temporary storage area). The viewer server 500 determines whether the viewer terminal 600 has received input of information indicating a false detection (step S155). When a user realizes a false detection while interpreting a medical image (for example, when the user realizes that the detection result is an FP (False Positive)), the user can operate the viewer terminal 600 to input information indicating the false detection. The user can input this information by performing a predetermined operation, such as clicking a predetermined button on the display device screen or selecting a menu for false detection notification.

[0089] The user may also input "information indicating a false positive" as part of the medical finding text. For example, phrases such as "false positive" or "no region of interest is found" may be input into the medical finding text. In this case, the viewer server 500, the report server 700, or the like may detect such phrases from the medical finding text and make the determination in step S155 based on the detection result. The viewer server 500, or the like may perform such detection based on a machine learning technique such as natural language processing. Alternatively, the viewer server 500, or the like may transmit the medical finding text to the information processing device 300, and the information processing device 300 may make the determination in step S155.

[0090] When the viewer server 500 receives input of information indicating a false detection (YES in step S155), it transmits the information to the information processing device 300 (step S165), and the information processing device 300 receives the information (step S335). In response to receiving (acquiring) the information, the movement control unit 316 (processor) instructs the PACS 400 to return the medical images moved to the long-term storage area 414 to the temporary storage area 412 (step S340), and the PACS 400 returns the medical images to the temporary storage area 412 in response to this instruction (step S170). In this case, the movement control unit 316 also outputs information indicating the movement history to the output device (step S350). When the medical images are moved, the movement control unit 316 may notify the user of this (by displaying on the screen, outputting audio, etc.).

[0091] In the process of false detection, if all detection results of a series of medical images (for example, medical images acquired in a single examination) are false detections (such as FP), those medical images may be returned to the temporary storage area 412. Furthermore, in determining whether a false detection has occurred, instead of the user inputting information as described above, the information processing device 300 or the like may perform image processing on the medical images or apply a recognizer to the medical images, and compare the images with the recognition results of step S310 to determine whether a false detection has occurred. This recognizer (the recognizer for false detection determination) can also be constructed using a machine learning technique, and may have a different algorithm, layer structure, parameters, etc. from the above-described recognizer 314A.

[0092] As described above, according to the medical information processing system 10 of the first embodiment, when predetermined medical information is detected from medical data, the medical image stored in the temporary storage area 412 is automatically moved to the long-term storage area 414, thereby reducing the possibility of a situation where "a medical image stored in the temporary storage area has been deleted and cannot be referenced even though one wishes to do so," and allowing medical images to be stored appropriately. Furthermore, even for medical images for which the user has not registered a key image or saved a bookmark, when predetermined medical information is detected, the medical image can be moved to the long-term storage area 414, thereby reducing the burden on the user regarding the storage of medical images.

[0093] [Processing in medical information processing system (second aspect)] Next, we will explain the second mode of processing in the medical information processing system 10. In the above-mentioned first mode, we have explained the case where the medical data includes only medical images out of the medical images and medical findings, but in the second mode, the medical data includes medical images and medical findings.

[0094] 9 is a flowchart showing the processing in the second aspect. The same steps as those in the first aspect (FIG. 6) are given the same step numbers, and detailed explanations will be omitted.

[0095] The viewer server 500 displays the medical image on the display device of the viewer terminal 600 in response to an instruction from the processor 310 (information processing device 300) (step S125). The user interprets the displayed medical image and operates the viewer terminal 600 to input a medical finding statement, which is then accepted by the viewer terminal 600 (step S135). The viewer server 500 transmits medical data including at least one of the medical image and the medical finding statement to the information processing device 300 (step S145), and the information processing device 300 analyzes the received (acquired) medical data (step S315).

[0096] As this "analysis," the medical information detection unit 314 (processor) can perform at least one of detecting a predetermined region of interest from a medical image using a recognizer 314A and detecting specific information from a medical finding text using a detector 314B. In this "analysis," the detection of a predetermined region of interest or the detection of specific information is one aspect of "detection of predetermined medical information" in the present invention.

[0097] When the medical information detection unit 314 receives only one of the medical image and the medical finding text from the viewer server 500, it can perform one of the processes corresponding to the received information. When receiving both, it can perform one of the processes or both. When performing both processes, the movement control unit 316 can determine to move the medical image if the detection results match. The medical information detection unit 314 may determine which process to perform in response to a user operation, or may automatically determine the process without a user operation. As described above, these processes can be performed using machine learning techniques (including natural language processing techniques). Note that the "specific information" may be information indicating a specific phrase or a specific organ, and the "specific phrase" may be a phrase related to a predetermined region of interest.

[0098] When a specific term is detected, the movement control unit 316 can move medical images related to the term. For example, when "iodine density" or "fat" is detected as a specific term, the movement control unit 316 can move the iodine density image and the fat density image, respectively. Furthermore, the movement control unit 316 may determine an organ in which a finding is recorded based on the detection result of a specific term or information indicating a specific organ, and move only slices showing that organ to the long-term storage area 414.

[0099] The "specific information" may be one or more, and when multiple pieces of information are detected, it is preferable to consider the relevance of the information. For example, it is preferable that the movement control unit 316 determines whether or not movement is necessary by considering the relevance of multiple detected words (in this case, "tumor," "recognize," "do not recognize," etc.) such as "recognize the presence of a tumor" and "do not recognize the presence of a tumor" in addition to the word "tumor."

[0100] If the movement control unit 316 (processor) detects predetermined medical information from the medical data (YES in step S325), it instructs the PACS 400 to move the medical image stored in the temporary storage area 412 to the long-term storage area 414 (step S340). In response to this instruction, the PACS 400 moves the medical image to the long-term storage area 414 (step S150) and displays the medical image and its analysis results (the analysis results in step S315) on the display device of the viewer terminal 600 (step S160). The movement control unit 316 (processor) also outputs information indicating the movement history (see the example in FIG. 7) to the output device (step S350). If the predetermined medical information is not detected from the medical data (NO in step S325), the processor 310 (information processing device 300) continues processing without instructing the movement of the medical image (proceeding to step S360).

[0101] The movement control unit 316 (processor) may make the determination in step S325 based on either the detection result from the medical image or the detection result from the medical finding text, or may make the determination based on both.

[0102] In the second embodiment, it is possible to deal with erroneous detection of medical information in the same way as in the first embodiment described above. A user can input information indicating erroneous detection, or a recognizer can be applied to a medical image to detect a predetermined region of interest, and the processor 310 can obtain information indicating erroneous detection from the results.

[0103] As described above, in the second aspect, medical images can be appropriately stored in the same manner as in the first aspect, and the burden on the user regarding the storage of medical images can be reduced.

[0104] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described aspects and various modifications are possible. [Explanation of symbols]

[0105] 10 Medical Information Processing Systems 20 Network 100 PCCT equipment 200 MRI machine 300 Information processing device 310 processor 312 Input / Output Control Unit 314 Medical Information Detection Unit 314A Recognizer 314B detector 316 Movement control unit 340 Storage device 350 Communication Interface 362A Input Layer 362B Middle Tier 362C output layer 364 convolutional layers 365 Pooling Layer 366 fully connected layer 370 Input Layer 380 Hidden Layer 382 Hidden Layer 384 Hidden Layer 390 Output Layer 410 Storage device 412 Temporary storage area 414 Long-term storage 500 Viewer Server 600 viewer devices 700 Report Server

Claims

1. An information processing device including a processor, The processor: acquiring medical data including at least one of a medical image and a medical finding statement related to the medical image; Applying a machine learning technique to the medical data to detect predetermined medical information from the medical data; When the medical information is detected, the information processing device moves the medical image stored in a temporary storage area of ​​a storage device to a long-term storage area of ​​the storage device.

2. The processor: The information processing apparatus according to claim 1 , wherein, when information indicating that the detection result is a false detection is acquired, the medical image that has been moved to the long-term storage area is returned to the temporary storage area.

3. The processor:

3. The information processing device according to claim 1, wherein the medical image is moved from the temporary storage area to the long-term storage area when information indicating that the medical image has been displayed on a display device together with the detection result is acquired.

4. The processor:

3. The information processing apparatus according to claim 1, wherein, when specific information is detected from the medical finding text, a medical image related to the specific information is moved from the temporary storage area to the long-term storage area.

5. 5. The information processing device according to claim 4, wherein the processor detects information indicating an organ that is the subject of the medical finding text as the specific information from the medical finding text, and moves a medical image showing the organ from the temporary storage area to the long-term storage area.

6. The information processing apparatus according to claim 4 , wherein the processor detects a specific word or phrase from the medical finding text as the specific information, and moves a medical image related to the specific word or phrase.

7. The information processing device according to claim 6 , wherein the processor detects a term relating to a region of interest as the specific term.

8. The information processing device according to claim 4 , wherein the processor detects the specific information from the medical finding text using a natural language processing technique.

9. The processor:

3. The information processing device according to claim 1, wherein, when a region of interest is detected in a first medical image, the first medical image and a second medical image related to the first medical image and of a different image type from the first medical image are moved from the temporary storage area to the long-term storage area.

10. The information processing apparatus according to claim 9 , wherein the second medical image is an image that is different from the first medical image in at least one of modality, imaging parameters, and image processing content.

11. The information processing apparatus according to claim 9 , wherein the processor moves a medical image showing a designated organ and / or region of interest among the second medical images.

12. The information processing apparatus according to claim 1 , wherein the processor acquires a CT image and / or a material decomposition image as the medical image.

13. The information processing apparatus according to claim 12 , wherein the CT image and the material decomposition image are images captured by a photon-counting CT apparatus.

14. The information processing device according to claim 7 , wherein the region of interest is at least one of a lesion, a suspected lesion region, and a post-treatment region.

15. 3. The information processing device according to claim 1, wherein the processor causes an output device to output information indicating the medical data stored in the temporary storage area and / or information indicating the medical data stored in the long-term storage area.

16. The processor:

3. The information processing apparatus according to claim 1, further comprising an output device that outputs information indicating a history of transfer of the medical images between the temporary storage area and the long-term storage area.

17. The information processing apparatus according to claim 1 , wherein the processor transfers the medical images to an area that stores the medical images for a longer period than the temporary storage area, the long-term storage area being the long-term storage area.

18. A method for operating an information processing device including a processor, comprising: The processor: acquiring medical data including at least one of a medical image and a medical finding statement related to the medical image; Applying a machine learning technique to the medical data to detect predetermined medical information from the medical data; If the medical information is detected, the medical image stored in a temporary storage area of ​​a storage device is moved to a long-term storage area of ​​the storage device.

Images